Self-hosted Ollama tests expose limits of large AI prompts

A blog post linked through Hacker News has detailed initial experiments moving large agent prompts from OpenAI and Anthropic to self-hosted Ollama and open-weight models, highlighting the trade-offs involved in running AI systems on personally controlled hardware.

The author said a 35 KB prompt and its session history rapidly consumed a 65,000-token context window. Reported failure signals included repeated file reads, unnecessary tool calls, thrashing and rewriting work that had already been completed.

The tests used a system with 128 GB of memory, including 32 GB allocated to the host operating system, and involved abliterated open-weight models with 27 billion parameters. The author said Ollama began to lose effectiveness within minutes when handling larger prompts that had run cleanly through frontier-provider APIs.

The post argues that the problem was not simply the smaller model size. Limited context windows meant the prompt and accumulated session history saturated the system quickly, causing the agent to lose awareness of earlier instructions and repeatedly revisit tasks.

The author presents self-hosting as a potential way to retain greater control over agent sessions and reduce dependence on external providers. The post also argues that locally hosted models could be useful for defensive cybersecurity testing where commercial safety filters may refuse vulnerability-related requests.

However, the observations are based on initial experiments and one hardware setup, rather than a controlled comparison. The specific models, configurations and benchmarks were not fully detailed, leaving the relative effects of context size, model capability, prompt design and Ollama configuration unclear.

Suggested responses include shortening prompts, monitoring logs for signs of context exhaustion and adapting agents to operate within smaller context windows.