PulseOnix / open source
PRISM
A CPU LLM inference engine in C99 where everything is a plugin: kernels, model architectures, tokenizers, samplers, even the generation loop and frontend. Source ships with a prebuilt AVX2 Linux runtime.
By Alish Karki, part of the PulseOnix engineering collective.
Implementation
- Swap any plugin at runtime by config — kernels, models, tokenizers, samplers, loops, io
- llama2.c and GGUF checkpoints (F32/F16/Q8_0/Q4_0/Q6_K) with GQA attention
- OpenAI-compatible API with SSE streaming, chat templates, auth, rate limiting, optional TLS
- Quantized KV cache, response cache, and 21 test suites behind an MIT license
Usage
Use the repository build and runtime instructions to load a supported llama2.c or GGUF checkpoint and run CPU inference through the command-line or HTTP frontend. The prebuilt runtime targets AVX2 Linux; check your CPU before using it.
Limitations and evidence
This is an experimental inference engine, not a performance replacement for llama.cpp. The development article reports an unresolved large-matrix classifier bug affecting MoE inference. Historical timings lack a complete reproducibility record.
Source and technical writing
Source code and current build instructions