Hardware for Qwen — and for the models that come after it

The baseline is a 24 GB-class consumer card. It runs the 4-bit Qwen3.8-27B at roughly 45–70 tokens per second with the model taking ~18 of the 24 GB — which is exactly the problem. A purchase made to run a 27 B model is already full on day one. The real target is three things at once:

The workload behind the question: many-hours, unattended, sequential analytical reads over structured financial data — batch throughput and headroom matter more than latency. Nothing here needs a model to answer in under a second.

Marketing sells cores and TFLOPS. For running language models locally, those are a distant third. The order is:

1. Memory capacity decides what you can run. A model's weights have to sit in fast memory in their entirety, every token. If it doesn't fit, it either doesn't run or it spills to system RAM and slows 5–10×. At 4-bit quantisation the rule of thumb is 0.5–0.6 GB per billion parameters, plus room for context.

2. Memory bandwidth decides how fast. Generating each token means reading every weight from memory once. So the ceiling on tokens per second is simply bandwidth ÷ model size in memory. A 17 GB model on a 1,792 GB/s card cannot exceed ~105 tok/s single-stream no matter what; the same model on a 256 GB/s box cannot exceed ~15. Software (speculative decoding, multi-token prediction) beats that ceiling by guessing several tokens per read — that is where the "tuned" figures below come from.

3. Compute decides prompt processing and batching. Reading a long document in (prefill) and serving many requests at once are compute-bound. This is where a discrete GPU is 3–5× a Mac at the same bandwidth, and it is the number that matters most for fine-tuning.

Ceiling = bandwidth ÷ 17 GB (the 4-bit Qwen3.8-27B in memory). Both numbers appear per machine in the candidates table below, next to what was actually measured.

2026 is a memory-shortage year. Hyperscalers are spending close to $700 billion on AI infrastructure and have bought the entire 2026 output of high-bandwidth memory; every gigabyte of HBM consumes 3–4× the wafer capacity of ordinary DRAM, so ordinary DRAM, GDDR7 and NAND are starved. Memory is more than 80% of the bill of materials on a high-end graphics card. When memory doubled in early 2026, cards followed — and NVIDIA is on its third GeForce price hike of the year. Industry consensus is that the squeeze lasts through 2026 and into 2027; the HBM gap does not close until 2028–29.

Two consequences for a buyer. First, waiting is not obviously rewarded — the usual "prices fall after launch" curve has inverted, and the one refresh that would help (a 24 GB RTX 5080 SUPER) is finished but held back because its memory got too expensive to ship at the planned price. Second, the answer to "isn't a workstation only a few thousand more?" has changed underneath the question, which is the next section.

The intuition comes from 2025, when it was true. A 96 GB RTX PRO 6000 Blackwell pre-ordered below $8,000 while a 32 GB RTX 5090 listed at $2,000 — the pro card cost about $6,000 more and bought three times the memory. That gap is now $9,000–12,000 on the card alone ($13,250–16,000 vs $4,200–5,100), and a workstation is the card plus a platform built to feed it.

What the extra money actually buys, in order of how much it matters for this use:

So the honest answer: a workstation is the only NVIDIA path to the 100 B+ class on one card, and it costs about three times a consumer box, not "a few thousand more." What the price shock has done is open a lane that did not exist in 2025 — Apple's 512 GB Mac Studio at roughly $11,000 now holds more model memory than a $35,000 dual-PRO-6000 workstation, at two-thirds of the bandwidth and with far less compute. That trade is the crux of the candidate table.

The pattern of 2026 is unambiguous: every lab ships one small dense model that fits a desk card (Qwen's 27 B, Gemma's 27 B) and one or more large mixture-of-experts models where only a fraction of the parameters fire per token. MoE is what makes big models viable on unified-memory boxes — a 120 B model with 5 B active reads only ~3 GB per token, so even 256 GB/s feels fine — but all of the weights still have to sit in memory. Capacity, not bandwidth, is what gates the next generation.

Read as tiers: 24–32 GB runs the dense 27–32 B class and nothing above it. 96–128 GB is the 100 B–120 B MoE class — the models that are, right now, the strongest thing you can run on one machine. 256–512 GB reaches 235 B–400 B. The trillion-parameter flagships (Qwen3.8-Max, DeepSeek V4 Pro, Kimi K3) are out of reach of any single machine anyone can buy; those are API models regardless of the licence.

Memory figures use 0.55 GB per billion parameters at 4-bit plus context headroom; vendor model sizes from the Wavect open-weight comparison (reviewed Aug 7, 2026), Unsloth docs and the Qwen3.8 release notes. "Runs it" assumes a single machine and 4-bit weights.

The question that decides how high to aim: as models get bigger, how much smarter do they get? Capability here is the Artificial Analysis Intelligence Index — one number from nine independent evaluations, on the same scale for every model — plotted against the memory each model needs at 4-bit. The best closed model (not purchasable at any price) is the dashed reference.

Where this leaves the "how high to aim" question: the evidence says aim for the shelf the next generation of small-and-mid models will land on, not for today's giants. The 100–180 GB tier already holds two models at 56–57.5 (Qwen3.8-Flash-Next 180B-A6B, GLM-5.3-Flash 320B) — within five points of the closed frontier — and both fit a 128 GB box or a 512 GB Mac, neither of which needs a workstation. The 96 GB card is the fast way to run that tier; the 512 GB Mac is the roomy way. Nothing above 512 GB buys measurable capability today. Index: Artificial Analysis v4.1.1 via artificialanalysis.ai and BenchLM, Snapshot August 28, 2026; memory at 0.55 GB per billion parameters plus headroom. Vendor-reported scores are excluded.

One table, one set of machines. Everything is buyable today or on announced pre-order, US pricing, quoted as card / complete system where both exist — "graphics card only" rows need a PC around them; "complete machine" rows are the whole box. "Ceiling" is bandwidth ÷ 17 GB, the hard single-stream limit for the 4-bit Qwen3.8-27B; the figure under it is what people actually measured. "Class" is the largest model family the machine runs at a usable speed.

Price sources on the verification date: videocardprices.com and Newegg trackers (RTX 5090, PRO 6000), NVIDIA US marketplace (PRO 6000 MSRP), Apple pre-order pricing via TechRepublic (M5 Ultra), IntuitionLabs (DGX Spark), Layer3Labs / vendor listings (Strix Halo mini PCs), Compute Exchange and JarvisLabs (used A100 / H100), ServeTheHome and Tom's Hardware (DGX Station), RunPod / Spheron (rental). Benchmarks: Context Studios (Aug 19), Hardware Corner (Mar 11), ComputingForGeeks (Aug 6), IntuitionLabs (Aug 12), RunAIHome (Jun 3), all 2026.

Strip away the also-rans and two machines survive goal #2. They are opposites.

The way to choose is to decide which sentence is true for you:

"I will spend most of the machine's life running one model as hard as possible, and I want to train it." → the PRO 6000. Speed, CUDA and batch throughput compound over a year of unattended runs, and fine-tuning is a first-class citizen. It caps at the 120 B class, which is the current frontier and likely stays a serious tier through 2027.

"I want to be able to try whatever gets released, including the 235 B–400 B models, and I mostly run, not train." → the M5 Ultra 512 GB. Nothing else under $30,000 holds those models. Accept slower tokens and a thinner training story.

Apple's announcement landed August 25: the M5 Ultra Mac Studio is real, starts at $5,499 with 96 GB, and the 512 GB configuration ships late October at a price Apple has not yet published — the 256 GB step costs +$4,000, so the 512 GB build plausibly lands in the $13,000–14,000 range (a fully maxed 256 GB / 16 TB unit already rings up at $18,299). That makes this section a decision with a date on it.

First, the number itself, because "512 GB" invites a false comparison. It is not the same kind of thing as a PC's system RAM. A conventional PC splits memory into two pools: big, slow system RAM for the CPU, and a small, very fast pool on the graphics card that the model must entirely fit inside — overflow it by a gigabyte and throughput falls off a cliff. Apple's unified memory merges the pools: one block, all of it addressable by the GPU, all of it at GPU-class speed (1.2 TB/s on the M5 Ultra — slightly faster than a 24 GB-class card's own memory). The honest comparison is therefore 24 GB against 512 GB of the same kind of memory — a 21× larger working set at equal speed, not a bigger hard-drive number.

Second, the paradox that makes the big-memory box more than a capacity play: the 100 B–250 B open models are mixtures-of-experts that activate only 10–22 B parameters per token, and generation speed is governed by active parameters × bandwidth. A 122 B-A10B model on the M5 Ultra streams ~10 B per token at 1.2 TB/s — it would plausibly generate faster than a dense 27 B does on a 24 GB card today, while judging like a model four times its size. Bigger and faster at once is rare enough in hardware decisions to be worth underlining.

Third, the CUDA wall — and why it matters less than it sounds. CUDA is NVIDIA's proprietary interface to its own silicon; nothing wraps or emulates it on Apple hardware at usable speed, and the mainstream fine-tuning stack (Unsloth/QLoRA) is CUDA-only. But inference never touches CUDA here: the daemon serving the model (Ollama and its peers) runs natively on Apple's Metal, exposes the same API on both platforms, and a pipeline that talks to the daemon over HTTP cannot tell what chip is underneath. The practical pattern is a two-box split: the big-memory box runs models all day; an ordinary 24 GB-class CUDA card — likely already owned — trains the adapters, which convert to the daemon's format and run on either machine. Neither box needs to be good at the other's job.

Fourth, the discipline before the invoice: every model this purchase would unlock is rentable by the hour in the cloud for a few dollars. Running the actual workload against a 122 B and a 235 B first answers the only question that justifies the spend — does the bigger model's judgment measurably beat the 27 B on this work — before $9,500–14,000 is committed to running it at home. If the delta is small, the 24 GB card was enough all along; if it is large, the invoice buys a number, not a hope.

Net framing for the decision: the M5 Ultra's 512 GB is the only sub-$30,000 seat at the 235 B+ table and the strongest run-everything box; the 96 GB CUDA card remains the training-and-throughput champion; and the cheapest correct next step is a cloud rental that turns "would a bigger model help?" from a belief into a measurement — ideally finished before the October ship date turns the answer into an order.

Fine-tuning does not mean retraining. The practical methods freeze the model and train small adapter layers (LoRA), optionally on a 4-bit base (QLoRA). Memory need is driven by whether the base is held at 4-bit or 16-bit, and the quality gap between them is smaller than the memory gap.

Two things worth knowing before the purchase is justified by "we'll fine-tune it". First, most of the improvement people want from a local model — domain vocabulary, output format, house style, using a private corpus — comes cheaper from retrieval and prompting than from training, and needs no extra hardware. Second, when training genuinely is the answer, renting eight H100s for an afternoon is often cheaper than owning the capacity for it; the local machine's job is then to run the result. A 96 GB card is the sweet spot precisely because it does both for the 27 B–70 B range without a rental.

A single RTX 5090 is not an upgrade for this purpose. It is the same 32 B ceiling as the 24 GB class, faster. Buying it in 2026 at 2.5× MSRP to gain speed on a model that already runs is the one clearly wrong move.

The 100 B-class MoE tier is the minimum "next year" target, and 96 GB is the smallest memory that serves it fast. Every serious open release of 2026 has landed there or above; a machine that cannot hold ~70 GB of weights will be running last year's models by next summer.

If the answer must be one machine: the RTX PRO 6000 workstation for a training-and-throughput operation, the M5 Ultra 512 GB for a run-everything operation. The price gap between them (~$9,000) is the price of CUDA and speed; the memory gap (416 GB) is the price of reach. Neither is a bad buy; they are different bets on what the next year's models look like — and the evidence of 2026 is that they get bigger faster than cards get cheaper.

A defensible third path: a $1,500–2,000 Strix Halo box today to learn what the 120 B MoE class is actually worth on real work, before committing $11,000–20,000. It is slow, but it answers the only question that matters — does a bigger model change the output enough to pay for — for the price of a graphics card's sales tax.

Not investment advice, and not a hardware endorsement: prices in this market change weekly and every figure above carries its verification date. Re-check the trackers cited before spending money.

One box, three jobs: the always-on model runner (Qwen-class today, the 100 B+ MoE class tomorrow, fine-tuning in between), the GPU sidecar for anything that must render, and a worker that reaches up to the web server for jobs and never accepts a connection from it. It sits in a closet, is reached over the network, and can be switched onto a desk with a KVM when the work is interactive. The one non-negotiable is the 96 GB card — capacity is what gates the next model generation, and it is the only single card that reaches it.

Total, card included: roughly $21,000–28,000. The card is half to two-thirds of it; the rest is the platform built to feed it and to run unattended for months. Against the verified candidates table above this is the "RTX PRO 6000 workstation" row, priced as a self-build rather than a vendor configuration, with the memory and second-card headroom that a vendor build usually strips out.

Software, either way you set it up. Headless Linux (Ubuntu 24.04 LTS, NVIDIA driver, CUDA, vLLM for batched serving, Ollama for dev, Tailscale only if push is ever needed, the job puller as a systemd service) is the cleaner always-on worker. Windows 11 Pro + WSL2 is the desk-switch plan: CUDA runs inside WSL2, Ollama runs natively, and the box is also a normal computer when the KVM flips to it — at the cost of managing Windows Update's reboots and sleep settings so the worker never quietly stops. Both work; the choice is whether you will sit at it.

The closet itself is a component: 700–900 W of heat needs a vented door or a louvre, and the blower card is louder than a gaming card under load. Prices here are ranges written 2026-09-07 in a memory-shortage year; every one of them should be re-quoted the week the order goes in.

RTX 5090 pricing — videocardprices.com tracker, Tech Insider GPU prices 2026. RTX PRO 6000 pricing — Tom's Hardware, VideoCardz, Thunder Compute. RTX 5090 vs PRO 6000 benchmarks — ComputingForGeeks, Spheron. RTX PRO 5000 72 GB — Spheron. Qwen3.8-27B per-hardware ladder — Context Studios, Unsloth GGUF report. Mac Studio M5 Ultra — TechRepublic, ModelFit; M5 Max benchmarks — Hardware Corner. DGX Spark — IntuitionLabs. DGX Station — ServeTheHome. Strix Halo — RunAIHome, Layer3Labs. Dual-5090 workstations — Puget Systems, Petronella build guide. Used A100 / H100 — Compute Exchange, JarvisLabs. Memory shortage — Barrack AI, TrendForce, Tom's Guide (RTX 50 SUPER). Open-model sizes — Wavect comparison, the Qwen3.8-Max memory math. Fine-tuning memory — Spheron VRAM guide, VRLA Tech. Bandwidth rule of thumb — Runyard, InventiveHQ. Quarterly practitioner guide — Julien Simon, April 2026. M5 Ultra announcement & pricing (retrieved Aug 30) — Apple newsroom, MacRumors (512 GB · late October), AppleInsider ($18,299 maxed / 512 GB unpriced), 9to5Mac (1.2 TB/s).