What a local AI model can do — and how you talk to it

A local model runs as a small server on your machine and speaks HTTP. You send it a list of messages — each with a role (system for standing instructions, user for what you ask, assistant for what it said before) — and it returns one new assistant message. That is the whole interface; everything else on this page is a field you add to the request. Nothing leaves your machine.

{"model": "qwen3.8:27b",
 "messages": [{"role": "user", "content": "In two sentences, what is free cash flow yield and why do value investors care?"}],
 "think": false, "options": {"temperature": 0}}

{"message": {"role": "assistant",
   "content": "Free cash flow yield is a valuation metric calculated by dividing a company's free cash flow per share by its current stock price, representing the cash return an investor receives relative to their investment. Value investors care because it measures actual cash generation rather than accounting earnings, revealing whether a stock is genuinely cheap on the money the business produces."},
 "done": true, "eval_count": 76}

The reply carries the text plus bookkeeping — how many tokens it read and wrote, how long each phase took — which is how a program measures cost and speed. Streaming ("stream": true) returns the same message a few words at a time, which is what makes chat interfaces feel live.

By default it returns prose. Asking politely for JSON works most of the time, which is not good enough for a program. The reliable way is structured output: you attach a JSON Schema to the request and the server constrains generation so the reply cannot be anything but a document matching it — the right keys, the right types, values from your list. This is the bridge between "a chat model" and "a data source".

{"model": "qwen3.8:27b",
 "messages": [{"role": "user", "content": "Company ZZZ trades at 6x earnings with net cash equal to 30% of its market cap and revenue growing 8%. Give a stance."}],
 "format": {"type": "object",
            "properties": {"ticker": {"type": "string"},
                           "direction": {"type": "string", "enum": ["undervalued", "fairly_valued", "overvalued"]},
                           "bullish_0_10": {"type": "integer"},
                           "one_line": {"type": "string"}},
            "required": ["ticker", "direction", "bullish_0_10", "one_line"]},
 "options": {"temperature": 0}}

{"ticker": "ZZZ",
 "direction": "overvalued",
 "bullish_0_10": 3,
 "one_line": "The 6x P/E is misleadingly low because 30% of the market cap is net cash, implying an operating multiple of ~8.6x …"}

Two things to notice. The schema did its job — a valid object, the direction drawn from the allowed list, an integer score. And the judgment is still the model's own: handed a 6× earnings, net-cash, growing company, this one called it overvalued. Structured output fixes the container, not the contents.

The model has no internet, no clock and no access to your database. What it has is tool calling: you describe functions your program can run — name, purpose, parameters — and when the model decides one is needed it replies not with text but with a structured request to call it. Your code runs the function, sends the result back as a tool message, and the model continues with real data. That loop is what "an agent" is.

{"model": "qwen3.8:27b",
 "messages": [{"role": "user", "content": "What is NVDA trading at right now?"}],
 "tools": [{"type": "function",
            "function": {"name": "get_quote", "description": "Latest price for a ticker",
                         "parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}}}]}

{"message": {"role": "assistant", "content": "",
   "tool_calls": [{"function": {"name": "get_quote", "arguments": {"ticker": "NVDA"}}}]}}

The tool description format is the same one the hosted APIs use, so a tool written once works against any model. The newer MCP standard (Model Context Protocol) packages whole tool servers — a file system, a database, a browser — so they can be plugged into any client without writing the glue each time.

Qwen3.8 is a hybrid reasoner. With "think": true it works the problem through in a separate thinking field before writing the answer; with false it answers directly. Thinking costs time and tokens — the example below spent 2,581 characters of working to produce one sentence — and on judgment-heavy tasks it is usually where the quality comes from. Effort can be dialled ("low" / "high") on this model.

{"message": {"role": "assistant",
   "thinking": "The user is asking about a stock with P/E 12, EV/EBITDA 5, but negative FCF for two years. The disconnect between earnings/EBITDA and cash flow is the key issue … (2,581 characters of working, not shown to the user by default)",
   "content": "Pull the cash-flow statement and see whether the FCF gap is driven by heavy capex (a capital-intensive investment cycle that EBITDA conveniently adds back) or by operating cash flow that's weak or negative — the first is a timing question, the second means the earnings are not converting to cash."}}

The working is returned to the program, not shown to the end user by default — so you can log it, audit it, or throw it away. Two practical notes from our own runs: on a full analyst briefing the model wrote roughly 7,000 characters of working and took about three times as long (~50 s vs ~17 s); and always verify the flag was applied by checking the length of the returned thinking field — a setting you sent is not a setting that took. We lost a whole experiment to a request where a later option silently overrode the flag, and the only tell was an empty thinking field.

Vision is built in. The model card lists four capabilities — completion, vision, tools, thinking — and images go into a message as an images field alongside the text: a chart, a screenshot, a scanned filing. It reads them the way it reads text; there is no separate image model to install.

Coding is one of its strongest suits. On the published benchmarks it solves 61.7% of SWE-bench Pro tasks (fixing real repository issues end to end) and scores 90.3 on LiveCodeBench — the numbers behind the "runs as a local coding agent" headlines. Asked for a function, it returns clean code with no preamble:

function fcfYield(float $fcf, float $marketCap): ?float {
    if ($marketCap <= 0) {
        return null;
    }
    return $fcf / $marketCap;
}

The honest limit: it writes and fixes code well inside a bounded task; on the hardest multi-file repository work it trails the closed frontier by a wide margin (42 vs 69–73 on the hardest agentic-coding set). The practical way to use it for code is not a chat box but an editor integration — see the ecosystem table below.

These are the pre-configured options — sent per request, or baked into a named variant (next section).

One lesson from our own testing: at temperature 0.7 the same document read twice changed the model's verdict 6 times out of 10. At 0 it was identical 10 times out of 10. If you intend to measure anything, start at 0.

"Upgrading" a local model rarely means new weights. Most of the improvement available to you is in how you talk to it, and each rung below is a real, distinct lever. A Modelfile is the small text recipe that turns any rung into a named model: FROM a base, SYSTEM a prompt, PARAMETER the settings, ADAPTER a trained add-on — then one command creates your-model and everything calls it by name.

Rungs 1–5 change what the model is told; rung 6 changes what it has learned; rung 7 changes what it is. Work down the ladder in order — and measure each rung on the same test set before taking the next, or you will not know which one helped.

You do not go looking for a bespoke API for each model. Everything has converged on a handful of conventions, which is why a local model and a hosted one can be swapped behind the same code.

No memory between calls. Each request starts blank; "memory" is you sending the history back. No knowledge after its training cut-off and none of your data — that is what tools and retrieval are for. Not an embedding model: this build refuses the embeddings endpoint; vector search needs a separate, tiny embedding model beside it. It will state wrong numbers confidently — in our testing the fix was to name the fields it should distrust, not to hope. One request at a time per model on a single GPU — throughput comes from batching servers, not from opening more connections.

Ollama structured outputs — docs, announcement; Modelfiles, adapters and imports — Ollama import docs, Modelfile guide; Qwen3.8-27B on Ollama — model library, KDnuggets coding-agent setup; ecosystem layers — iunera 2026 tools comparison, inference tools guide, July 2026; benchmarks — the model page (vendor-reported and third-party figures with dates). Example replies: local run of qwen3.8:27b (Q4_K_M), temperature 0, August 28, 2026 · 15:10.