Skip to content
Phobos

LLM inference on Phobos kernels

Quantized GGUF models run end to end on kernels written in the language, behind an OpenAI-compatible server. Built and measured on an 8 GB card.
QuickstartBenchmarks

What it is

  • GGUF models on Phobos kernels

    phobos-cli loads a GGUF file, picks the forward pass from the architecture it names, and runs every layer on kernels written in Phobos and compiled for the card. Decode fuses a chain of layer stages into one persistent kernel.

  • An OpenAI-compatible server

    --listen serves /v1/chat/completions, /v1/completions and/v1/models, renders conversations and tools with the model's own chat template, streams chat over server-sent events, and puts a dashboard on the terminal.

  • Made for little VRAM

    Written and measured on an 8 GB card, on which two 27B models at under two bits a weight fit with room for their context.

  • Experts streamed from host memory

    A mixture-of-experts model that does not fit keeps a cache of experts on the card and computes the misses on the CPU, with no manual split between the two. The cache gives memory back when another program fills the card, and takes it again once it is free.

phobos-cli --listen: the splash, then the dashboard with the model, the memory on the card, the network of layers and the log.

Decode and prefill against llama.cpp

On an RTX 2080 SUPER, Phobos decodes at 0.93x to 1.82x the rate of llama.cpp over 128 tokens, and at or above it on 5 of 7 models; the faster engine is llama.cpp on GLM-4.6V-Flash-Q4_K_M and PrismML's llama.cpp fork on Ternary-Bonsai-2-27B-PTQ1_0. A 128-token prompt runs at 0.83x to 4.09x the other engine's rate.

Phobos against llama.cpp, tokens per second, higher is better

NVIDIA GeForce RTX 2080 SUPER, driver 617.14, Phobos 3eec883, results committed 2026-10-08

  • Phobos
  • llama.cpp, or the fork its row names
Qwen3.5-0.8B-Q8_0bench.json, 5 rounds x 3 reps
MiniCPM5-1B-Q8_0bench.json, 5 rounds x 3 reps
Qwen3.5-4B-Q4_K_Mbench.json, 5 rounds x 3 reps
GLM-4.6V-Flash-Q4_K_Mbench-glm.json, 3 rounds x 3 reps
Ternary-Bonsai-2-27B-PTQ1_0bench-bonsai.json, 3 rounds x 1 rep, against PrismML's llama.cpp fork
prefill pp512not measured
Qwen3.8-27B-UD-IQ1_Mbench-qwen38.json, 3 rounds x 1 rep
prefill pp512not measured
Qwen3.6-35B-A3B-UD-Q4_K_Mbench-qwen36moe.json, 3 rounds x 1 rep, llama.cpp -ncmoe 31 -t 8

Each pair of bars has its own scale, so bars compare the two engines on one model and one test, not across models. The ratio is Phobos's rate over the other engine's. Rates are the mean over the rounds bench.py found uncontended, each round the mean of its repetitions.

All numbers, with standard error
Tokens per second for every model and test, mean and standard error over rounds
modeltestPhobosother engineratiosource
Qwen3.5-0.8B-Q8_0pp1285976.29 +/- 179.576723.91 +/- 12.97 llama.cpp0.89results/bench.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
pp5128693.19 +/- 38.5710729.32 +/- 26.750.81
tg32325.00 +/- 0.66229.87 +/- 0.221.41
tg128323.94 +/- 0.11246.98 +/- 0.031.31
tg512324.50 +/- 0.06250.23 +/- 0.051.30
tg1024323.53 +/- 0.14249.84 +/- 0.121.29
tg2048322.09 +/- 0.10248.61 +/- 0.071.30
MiniCPM5-1B-Q8_0pp1287574.73 +/- 27.898569.34 +/- 16.46 llama.cpp0.88results/bench.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
pp51210086.13 +/- 26.2616164.09 +/- 20.500.62
tg32315.18 +/- 0.25270.33 +/- 0.271.17
tg128314.19 +/- 0.15270.89 +/- 0.191.16
tg512313.14 +/- 0.11269.83 +/- 0.171.16
tg1024310.84 +/- 0.05268.98 +/- 0.281.16
tg2048304.84 +/- 1.00266.84 +/- 0.201.14
Qwen3.5-4B-Q4_K_Mpp1282305.30 +/- 3.322296.17 +/- 3.53 llama.cpp1.00results/bench.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
pp5122372.05 +/- 2.442956.21 +/- 1.310.80
tg32111.92 +/- 0.14103.60 +/- 0.081.08
tg128111.75 +/- 0.14106.54 +/- 0.111.05
tg512111.54 +/- 0.42106.77 +/- 0.291.04
tg1024111.59 +/- 0.26107.01 +/- 0.121.04
tg2048111.43 +/- 0.10106.50 +/- 0.121.05
GLM-4.6V-Flash-Q4_K_Mpp1281384.87 +/- 1.661665.23 +/- 63.55 llama.cpp0.83results/bench-glm.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
pp5121160.08 +/- 2.471974.65 +/- 8.100.59
tg12859.69 +/- 0.0263.23 +/- 0.010.94
Ternary-Bonsai-2-27B-PTQ1_0pp128583.99 +/- 10.82469.76 +/- 0.40 PrismML's llama.cpp fork1.24results/bench-bonsai.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
tg12845.25 +/- 0.2348.54 +/- 0.230.93
Qwen3.8-27B-UD-IQ1_Mpp128537.82 +/- 0.00473.17 +/- 0.85 llama.cpp1.14results/bench-qwen38.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
tg12830.98 +/- 0.0121.69 +/- 0.001.43
Qwen3.6-35B-A3B-UD-Q4_K_Mpp128311.55 +/- 4.1676.25 +/- 0.12 llama.cpp4.09results/bench-qwen36moe.json, NVIDIA GeForce RTX 2080 SUPER, driver 617.14, 3eec883, 2026-10-08
pp512577.88 +/- 1.13257.57 +/- 0.552.24
tg12854.32 +/- 0.1929.88 +/- 1.641.82

How these runs were measured, the 35B on a replayed coding-agent session, and what happens when another program takes part of the card: Benchmarks. To measure your own card: Benchmarking.

Five ideas behind the numbers

Decode waits on memory

A decode step reads every weight once and does one multiply-add with it. That is about two operations per byte, and the card needs roughly ten times that before its math units, not its memory bus, set the pace. A prompt reuses each weight for every token in it, and the bottleneck flips. Decode kernels are written to keep the bus busy, not the math units.

Why decoding is bound by memory bandwidthSchematic. Weights stream from VRAM over the memory bus to the card's 48 SMs. Decoding reads every weight once per token for about 1.9 operations per byte, so the bus is always busy while most SMs wait. A 512-token prompt reuses every weight for about 960 operations per byte, so the SMs are busy and the bus mostly idle. A log scale of operations per byte marks the card's balance point at about 22.FIG / DECODE / BYTES VS MATHschematicweights in VRAMmemory bus, 496 GB/s48 SMs, 11.2 TFLOP/sonce a tokenbusy 100%busy 8%1101001,000balance, 22memory-boundcompute-boundoperations per byte, log scaledecode, 1.9

Decode: about 1.9 operations per byte, and the card needs about 22 before its math sets the pace. The bus runs flat out, the SMs mostly wait.

  • global memory
  • registers
  • memory-bound
Schematic timing. Q8_0 stores 32 weights in 34 bytes, and a token does one multiply-add per weight. The RTX 2080 SUPER's 11.2 TFLOP/s fp32 and 496 GB/s are from its specifications. Measured on the RTX 2080 SUPER, Phobos decodes Qwen3.5-0.8B-Q8_0 at 324 tokens/s over 128 tokens (results/bench.json, driver 617.14, Phobos 3eec883, 2026-10-08).

One warp per output

Every output of a decode matrix-vector product is one row of weights dotted with the activation. Giving each thread its own row scatters a warp's reads over 32 rows. The lanes matvecs give a whole warp one row instead: its 32 lanes read one contiguous run, and sum<over ctx.lane> folds their partial sums in five shuffle stages.

Two ways to map a matrix-vector product onto a warpSchematic. With one thread per output, each of a warp's 32 threads walks its own row of weights, so one load touches 32 rows. With one warp per output, the 32 lanes read 16 bytes each, 512 contiguous bytes of one row per step, then add up their partial sums in 5 stages of xor shuffles.FIG / DECODE / MATVEC MAPPINGschematicactivation, int8weights, Q8_0: 32 int8 and a scale per blockoutputone warp, 32 laneslane 0 writes the sum

A warp per output: 32 lanes read 16 bytes each, one contiguous 512-byte run a step, then sum<over ctx.lane> adds their partial sums in 5 xor-shuffle stages.

  • global memory
  • registers
  • scattered reads
Schematic. The warp form is the Q8_0 decode matvec as phobos-gguf writes it with lanes: for k in range(ctx.lane * 16, K, ctx.lanes * 16), then sum<over ctx.lane>(acc), and lane 0 stores the output.

A model that does not fit

Qwen3.6-35B-A3B routes every token to 8 of 256 experts per layer, and its experts alone do not fit on the card. Phobos keeps the experts in host memory and a cache of them on the card, least recently used out first. On a decode step the CPU computes the misses straight from host memory while the card runs the hits; a prompt copies its misses over PCIe.

An expert cache on the cardSimulated. For every token the router picks 8 of 256 experts in one layer. Picks already in the cache on the GPU run there. In decode the CPU computes the misses from host memory and one miss is copied into the cache for later tokens; a prompt copies its misses over PCIe into the least recently used slots.FIG / MOE / EXPERT CACHEschematic256 experts, host memoryPCIe 3.0 x8, 6.4 GB/sCPUfrom host memoryGPUexpert cachethe rest of the model

Token 1: 6 of 8 picks already on the card, the CPU computes the 2 misses from host memory. Simulated hit rate so far 75%.

  • global memory
  • registers
  • misses
One layer of Qwen3.6-35B-A3B. The routing is simulated, so the hit rate shown is the simulation's, not the model's. A prompt also runs part of its misses on the CPU while the copies cross the link.

Giving the memory back

The cache is sized from the card's free memory, and Phobos reads the whole card's free memory again between passes. When another program fills the card, the cache shrinks instead of letting the driver page buffers to host memory, and more of the misses go to the CPU threads. When the memory is free again, the cache grows back. llama.cpp fixes its split at load time, so the same squeeze pages its buffers and every token reads them back over PCIe.

An elastic expert cacheMeasured on Qwen3.6-35B-A3B, replaying a coding agent's session. The card's 8 GiB hold the rest of the model, its context and an expert cache of 38 experts a block. Another process takes 2 GiB; the cache shrinks in steps to 14 experts a block and the CPU threads compute more of the misses. When the process exits the cache grows back to 39. Decode before, during and after: Phobos 45.8, 30.8, 44.6 tokens/s, llama.cpp 31.1, 9.0, 31.2.FIG / MOE / ELASTIC CACHEschematiccard memory, 8 GiBmodel, contextcache 38, 24, 14, 39 a blockanother process, 2 GiBCPU threadscompute misseswall time, one replayPhobos 52.4 sllama.cpp 131.7 sdecode, tokens/sPhobosllama.cppbefore2 GiB takenafter45.830.844.631.19.031.2

The process exits and the cache grows back to 39 experts a block. Phobos decodes at 44.6 tokens/s, llama.cpp at 31.2. The replay took 52.4 s on Phobos and 131.7 s on llama.cpp.

  • the card
  • CPU, misses
  • another process
Qwen3.6-35B-A3B on an RTX 2080 SUPER, results committed 2026-10-06, median wall time of the replay, decode rates the mean over 2 and 3 replays. Cache sizes are the ones the run logged; the flow of misses is drawn, not measured.

Rotate before rounding

Ternary weights share one scale per block, so one outlier can ruin it: the best scale fits the outlier and the small weights round to zero. A Hadamard rotation spreads the outlier over the block before rounding. The activations get the same rotation at run time, so the product does not change, and one rotation serves every projection that reads the same input.

Rotating before roundingSchematic. A row of 16 weights with one large outlier is rounded to ternary values, once as it is and once after a Hadamard rotation. As it is, the best scale fits the outlier and 15 of the 16 weights round to zero. The rotation, random sign flips followed by 4 stages of additions and subtractions, spreads the outlier over all 16 values, and the rounding error drops from 52 to 35 percent.FIG / TERNARY / ROTATE, THEN ROUNDschematica row of weightsHadamard rotationrounded to -1, 0, +1signs4 butterfly stagesadds and subtracts only, then / 4; the model's blocks are 1,024 wide, 10 stages+++++--+-++---++-10+1

Rotated first: the outlier spreads over all 16 values. Rounding error 35%, and the activations get the same rotation at run time.

  • global memory
  • registers
  • outlier, rounding error
Schematic: a seeded 16-wide toy row with one outlier, rounded with the best ternary scale. PTQ1_0, the format of Ternary-Bonsai, holds 128 weights of -1, 0 or +1 and one scale in 28 bytes; phobos-gguf rotates the activations in blocks of 1,024, 10 stages.

One token at a time

Decoding is a loop: the tokenizer turns the prompt into tokens, the layers turn the last token into logits, the sampler picks the next token, and that token streams out and goes back in. Every pass reads all the weights the token needs from the card's memory, which is why the ideas above matter.

The decode token loopSchematic. A GGUF file supplies the weights to the layers. The tokenizer turns the prompt into tokens for the layers, the layers produce logits for the sampler, and each sampled token streams out and goes back into the layers as the next input.FIG / DECODE / TOKEN LOOPschematicGGUF filetokenizerlayersfused decode kernelsampleroutTiles move through shared memory.

Each sampled token streams out and goes back into the layers as the next input.

Models and formats

The architectures are llama, qwen35, qwen35moe and glm4. These files have been checked against llama.cpp, token for token where greedy decoding is stable. Every format below decodes on the card; see Models and formats for what each architecture covers.

Tested models

20 quantized formats

Block formats
Q4_0Q4_1Q5_0Q5_1Q8_0Q8_1
K-quants
Q2_KQ3_KQ4_KQ5_KQ6_K
I-quants
IQ1_MIQ1_SIQ2_SIQ2_XSIQ2_XXSIQ3_SIQ3_XXSIQ4_XS
Ternary
PTQ1_0

One file per format in phobos-gguf/src/quant/. A file can mix formats from tensor to tensor, and activations are quantized to Q8_0 whatever the weight's format.

Quickstart

  1. Get the binary

    Download a release and keep kernel-cache/ beside the binaries, or build from a checkout:

    cargo build --release --features cuda -p phobos-cli

  2. Ask a question

    Download MiniCPM5-1B-Q8_0 and pass a prompt. Without one, phobos-cli starts a REPL.

    phobos-cli --gguf minicpm5-1b-Q8_0.gguf "Why is the sky dark at night?"

  3. Serve it

    Point any OpenAI client at http://127.0.0.1:8080/v1.

    phobos-cli --gguf minicpm5-1b-Q8_0.gguf --listen 127.0.0.1:8080