A local LLM without a GPU: what a year of Ollama on an old i5 taught me

Why bother with a slow model when the cloud is fast

The short answer: because prompts never leave the house. My home agent reads server alerts, the family's notes and the state of our devices — exactly the kind of data I don't want to ship to an external API, however cheap it may be. Then there's predictability: a local model won't change its pricing, won't get pulled from an offering, and won't suddenly start answering differently because someone on the other end swapped the version.

The long answer takes up the rest of this post, because a local LLM without a GPU is a series of trade-offs — and the tutorials that end at ollama run rarely talk about those.

Hardware: disappointment factored into the plan

My server is an Intel i5-8500 — six cores from 2018 — and 32 GB of RAM. No GPU whatsoever. On this hardware, Ollama currently runs a handful of models, each with its own job:

These numbers set expectations better than any benchmark from the internet. Eight tokens per second means an alert summary takes somewhere between ten and a few dozen seconds. One curiosity in that table: gemma4:e4b takes up nearly 10 GB on disk, yet generates faster than the noticeably smaller, dense 4B model I used before — because it's a MoE architecture and only a fraction of the parameters is active at any time. "Bigger file" stopped meaning "slower".

The agent with tools used to answer in ~250 seconds for a single-tool query. Today the same conversation takes ~30 seconds — not thanks to a new model, but thanks to rebuilding the pipeline and the prompt cache, more on which in a moment. A cold start, when the model is still loading into memory, adds its own toll.

The most painful lesson, though, wasn't about generating — it was about reading. My personal agent once got an elaborate system prompt — around two thousand tokens of instructions. On CPU, just processing the prompt took over four minutes before the model wrote its first word. On a GPU nobody notices this; on CPU, prompt length is a real cost, paid on every single conversation.

Paid — unless you learn to use the prompt cache. Ollama can skip re-processing the beginning of a conversation, as long as it's byte-for-byte identical to the previous request. For a long time this didn't work for me, because I was gluing a fresh context block onto every user message — the prompt diverged right after the system instructions and the cache had nothing to hit. After moving the context into the stable part of the prompt, the second and every following turn of a conversation processes in nine seconds instead of a hundred and four. Same machine, same model, a twelvefold difference.

Three small models instead of one big one

The first instinct is always the same: grab the biggest model that fits in RAM. My biggest early mistake — because "fits" and "is usable" are two different things. An 11B model technically works on CPU. In practice: you'll have time to brew tea (after starting the campfire first) before it finishes a sentence.

What works for me instead is division of labour. The fast phi4-mini classifies monitoring alerts and Bielik turns them into short Polish notifications — both narrow, repetitive tasks where small models are plenty. Gemma 4 E4B runs the home agent, because in my tests it calls tools most reliably: it doesn't drop required arguments and doesn't invent functions that don't exist. And qwen3-embedding turns notes and the agent's memory into vectors — so search works by meaning rather than keywords, and does it in a fraction of a second.

With embeddings, a trap was waiting that no model card spells out: qwen3-embedding is an asymmetric model. Documents are encoded as-is, but queries need a special instruction prefix — without it, result rankings can get absurd: for the question "what does he like to drink", a note about a Nutella jar ranked above the actual fact about coffee. One prefix in the code and the ordering returns to sanity. If your search suddenly "goes dumb" after switching embedding models — start there.

Bielik — the Polish model from SpeakLeash — I tested in its 4.5B version with an eye to replacing phi4-mini for alerts. The result was instructive: it summarizes in Polish noticeably more elegantly, but it flunked alert classification with a strict JSON format — it flattened every priority to medium, while phi4-mini separated them flawlessly and three times faster. The most beautiful language skills don't win a task that demands hard classification rather than nice style. Bielik stayed on disk and eventually got one: in today's alert path, phi4-mini classifies and Bielik writes the final message in Polish — each model doing what it's good at. One deployment trap along the way: through the raw /api/generate endpoint Bielik would echo the prompt back instead of answering; what fixed it was chat-style calls via /api/chat plus an example answer in the system prompt.

Temperatures, or physics collecting its dues

Tutorials are completely silent on this: an LLM on CPU is the hardest work this processor has ever done. During inference, all cores run flat out, uninterrupted, for tens of seconds. My i5 was hitting 88-95°C — against a warning threshold of 82°C. For a while I considered that "acceptable", which is a euphemism for "I was ignoring the problem".

The fix turned out to be banal and had nothing to do with AI:

# docker-compose.yml — Ollama configuration fragment
environment:
  - OLLAMA_NUM_THREAD=4        # 4 of 6 cores → ~65% CPU, temps below 80°C
  - OLLAMA_KEEP_ALIVE=2m       # default: quick model unload after a conversation
  - OLLAMA_MAX_LOADED_MODELS=3 # agent + alerts + embeddings at the same time
  - OLLAMA_NUM_PARALLEL=2      # max 2 parallel requests
deploy:
  resources:
    limits:
      memory: 24g              # hard RAM limit for the container — more on that shortly

Giving up two cores costs some speed, but the processor stopped flirting with the critical threshold — and a server that also does everything else has room to breathe. The second parameter is a compromise in the other direction: a short global KEEP_ALIVE frees RAM and lowers idle temperatures, but a conversation after a pause may start from a cold start.

You can have both, though, because keepAlive can be overridden per request. The global 2m is my default hygiene for ad-hoc tasks, but the agent's requests set their own keepAlive: 24h and numCtx: 8192 — the agent's model sits in RAM permanently, so a conversation doesn't begin with a cold start and a re-processing of the long prompt. The other models come and go.

Here's the thing I underestimated for half a day: keepAlive is a request, not a guarantee. Whether a model actually stays in memory is decided by MAX_LOADED_MODELS — with a limit of 2, every alert (loading the summarization model) and every search (loading the embedding model) was quietly evicting my "permanent" agent from RAM. The effect: forty seconds of loading tacked onto seemingly random conversations. A limit of 3 closed the case — all three staff models fit in 24 GB with room to spare.

The trap that looks like running out of RAM

The strangest bug of the year: Ollama refuses to load a model, claiming it doesn't fit in memory — on a machine with 32 GB, of which eight are actually in use. The reason: when checking free memory, Ollama looks at MemFree, not MemAvailable. And MemFree on a server that reads a lot from disk can drop below a gigabyte, because Linux rightly keeps the rest in cache — which it will hand back the moment anyone needs it.

In numbers it looks absurd: 32 GB of RAM, ~8 actually used, some 20 GB in cache — and MemFree shows less than a gigabyte, so Ollama refuses to load a model that needs eight. The model "doesn't fit" into memory that exists in abundance; the system just hasn't tidied it up yet. What helped in my case was setting an explicit memory limit for Ollama and moving heavier AI jobs to the night, outside the hours of heavy I/O. But the diagnosis took far longer than the fix — because the error message lies about the cause.

Timeouts: all the clocks have to tick in step

A slow model exposes a problem you never think about with a fast API: timeouts. For a long time my request travelled through the application, then through n8n, and finally to Ollama — and every layer had its own clock. If the application waits less than n8n, the user gets an error while the answer is still being written. If n8n cuts off before Ollama does, the model writes into the void.

The rule I arrived at after a few "Error in workflow" messages with no clear culprit: you set the clocks from the end. First, the realistic time of the model's slowest answer — with a cold start and a long prompt, not the average — and then every layer above gets at least as much. Plus documentation of why that much, because six months later those values look arbitrary.

These clocks can reach tens of minutes, and that sounds like an eternity — it usually is one. But the headroom is budgeted for the worst case: cold start plus long prompt plus two tools along the way. A clock that cuts off ten seconds too early wastes five minutes of processor work. Today I have fewer layers, and therefore fewer clocks — why, in a moment.

The bug you can't fix: why the agent moved out of n8n

This story began with a message every user of n8n's AI Agent node knows: Received tool input did not match expected schema. An agent that had been answering questions about the server's state for months suddenly failed on every tool call. First suspect: a model that's too weak. I swapped it for a better one — no change. Second suspect: the model getting lost with a dozen-plus tools. The deciding test: a tool with no parameters at all, to which the model sent correct, empty arguments — failed too.

The real culprit sat inside n8n — in version 2.25, which I was running at the time: after one of the updates, tool call arguments were getting lost between the LLM library and execution, and the validator received emptiness regardless of what the model answered. There are several reports with this message on n8n's GitHub (#14399, #17241, #23588) — as I write this, they're being closed as stale or as "support issues", without a fix. Rolling back to an older version? Impossible — n8n migrates its database one way only. This is the moment self-hosting bares its teeth: a bug in someone else's middle layer, which you can't influence and can't route around.

The way out turned out simpler than it sounded: remove the middle layer. An agent loop is, at its core, a small program — send the conversation with a list of tools to Ollama; if the model wants to use a tool, run it and append the result; repeat until an answer emerges. I wrote it directly into the application backend: no LLM framework, no visual editor, but full control over every timeout, prompt and log. n8n still does excellent work in my automations — the alert pipeline, notifications, schedules — but the agent's conversation with the model no longer passes through it.

The side effect turned out to matter more than the fix itself: full control over the pipeline made it possible to lay out the prompt for the cache (those nine seconds instead of a hundred and four from the beginning of this post), keep the right model in memory, and stream the answer live. A conversation with a tool that took ~250 seconds in the old pipeline now wraps up in ~30. Same processor, same 4B-class model.

Where a local LLM makes sense — and where it doesn't

After a year, the split is clear. Local AI works for tasks that run in the background and don't make anyone wait: monitoring alert summaries, embeddings and semantic search, content tagging, a home agent that answers within tens of seconds and drives the vacuum. The common denominator: nobody is sitting there staring at a cursor — and since the agent's answer started streaming live, even that wait looks like work in progress rather than a hang.

It doesn't work where the waiting is supposed to be a conversation: interactive chat with a two-second response, long contexts processed live, anything that competes with the fluidity of a cloud chat. You can resent that, or accept it and design around the strengths — my alert list doesn't need an answer in two seconds; it needs an answer that didn't send the alert's contents outside my home.

Where to start

Not with buying a GPU. With one small model — 3-4B parameters — and one real task: summarize this for me, tag that, find similar notes. If after a week the task is genuinely happening on its own, it's worth investing further. If not — you've saved on a graphics card, and docker compose down costs less than regret over money spent.

In my case, it stuck. Alerts arrive summarized in Polish, the agent turns off the lights and keeps track of family arrangements, and note search understands that "dad's birthday" and "gift for father" are the same topic. All of it on six cores from 2018 — slowly, locally, and exactly the way I wanted. And a GPU? It can always be added — but that's a whole different story.