Forward Deployed Engineer - Project Case Study · 2026
wLLM: Native Windows LLM Inference Server
View sourceEngagement Snapshot
What I built, who it is for, and where my ownership starts and stops.
Native Windows LLM Inference Engine
vLLM does not run natively on Windows. wLLM reimplements its core ideas (PagedAttention, continuous batching, CUDA graphs) as a native CUDA/C++ and PyTorch engine with an OpenAI-compatible API, and no WSL2 or Docker.
Role
Sole designer and implementer
Scope
CUDA kernel, KV-cache manager, scheduler, API server, benchmarking
CUDAC++FastAPIPydanticWindowsNVIDIACustomer & Problem
Who this is for, the messy reality they work in, and why the system needed to exist.
Who it is for
Windows developers and workstation users with a single consumer GPU who want production-style serving and an OpenAI-compatible endpoint.
What was wrong before
Serving-engine optimisations (paged KV cache, continuous batching, CUDA graphs) exist only on Linux. On Windows the choice is a plain HuggingFace loop with poor GPU utilisation, or a WSL2/Docker layer.
Who needed it
Windows developers and workstation users with a single consumer GPU who want production-style serving behaviour and an OpenAI-compatible endpoint.
If nothing changed
Consumer GPUs stay under-used: memory is wasted on over-allocated KV cache, decode is launch-bound, and one long prompt stalls everyone.
The environment I had to work in
- vLLM has no native Windows support, so the usual path is WSL2 or Docker.
- A consumer GPU has limited VRAM, so KV-cache memory is the binding constraint.
- Users already have OpenAI-based tools and expect them to work unchanged.
- Small models are launch-bound, large models are compute-bound, so one optimisation does not fit all.
Discovery & Requirements
The questions I ask first, and the requirements every decision below traces back to.
Discovery questions
- 01Which GPU and how much VRAM?
- 02Which models and context lengths are needed?
- 03Which existing clients must keep working?
- 04What counts as good enough versus vLLM?
Requirements
System Architecture
The centerpiece. The diagram is the easy part - the value is in why each boundary exists.
Why this boundary exists · OpenAI client
Compatibility is the product surface. Any existing OpenAI SDK or tool works by changing a base URL.
Click any component to see why it is a separate boundary.
Integrations & Data
Where each category of information lives, what it connects to, and why.
Critical Architecture Decisions
Architecture decision records: the problem, the options, the call, and when to revisit it.
ADR-001 - KV-cache layout
Options considered
Decision
Paged blocks.
- Problem
- Contiguous per-request KV allocation wastes memory on short outputs and fragments as requests come and go.
- Reasoning
- Fixed-size blocks allocate on demand, eliminate over-allocation and make prefix sharing a block-table operation.
- Trade-offs
- Needs a custom attention kernel and block-table bookkeeping.
- Revisit when
- If block size tuning shows meaningful overhead on very small models.
Deployment & Handoff
How it gets installed and shipped, what it runs on, and how it is handed over so the customer can run it.
Git.
CUDA runtimeRollout and handoff
Drop-in for existing clients
An OpenAI-compatible API means adoption is a base-URL change, with five endpoints covered.
No virtualisation layer
Runs natively on Windows without WSL2 or Docker, which removes the biggest setup barrier.
Benchmarks users can reproduce
Measured against a HuggingFace baseline and real vLLM on the same GPU, so claims can be checked.
Reliability Engineering
Failure thinking: what breaks, what catches it, and what actually went wrong.
Served by CUDA graph decode
Security Architecture
Every request passes these gates in order.
Bind to localhost.
Local service
Intended as a local/workstation server; binds locally by default.
Input validation
Pydantic schemas and prompt/response length limits guard the engine.
Supply chain
Native CUDA extension built from source, with pinned PyTorch/CUDA versions.
Results
Only outcomes that can be substantiated. Adoption figures are added when they are real.
Performance
Engineering
Field Lessons
What this project taught me about shipping AI into someone else's environment.
L1
Benchmark against the real thing
Comparing against real vLLM, not a strawman, is what makes the numbers believable.
L2
Gains depend on model size
CUDA graphs help most on small models, and the gap to vLLM narrows as matmul compute dominates.
L3
Compatibility beats novelty
Matching an existing API removes the need to convince anyone to change their code.
Request / Execution Flow
One real request traced through the system - dynamic behaviour, not just boxes.
POST /v1/chat/completions.Standard OpenAI request.
Deep-Dive Architecture
The technically difficult pieces, step by step.
Request.Arrives on an OpenAI-compatible endpoint.
AI Architecture
Model strategy, retrieval, and evaluation - the parts specific to an AI system.
Model strategy
Small models
Launch-bound decode where CUDA graphs give the largest gain (3-7× throughput).
Larger models
Compute-bound; graph gains narrow and the gap to vLLM closes.
Baseline
HuggingFace transformers loop used as the reference for speedup.
Evaluation
Observability
The observability model, not a tool name.
Decode stepsIllustrative span layout - shows nesting, not measured durations.
Performance Engineering
Where the bottlenecks were, and what moved the numbers.
What caused the improvement
- CUDA graphs remove per-step kernel-launch overhead, the dominant cost for small models.
- Paged KV blocks avoid over-allocation, letting more sequences fit on the GPU.
- Prefix caching skips recomputing shared system prompts.
- Chunked prefill stops long prompts stalling decode.
Measured on one RTX 3060 against vLLM under WSL2. The first bar is a relative illustration of the published 3-7× range, not an absolute tok/s figure.
Architecture Principles
The architecture followed from these, not from a shopping list of technologies.
P1
Windows-native
Removing WSL2/Docker is the feature. Every dependency is judged against that.
P2
Memory first
On a 12 GB-class GPU the KV cache, not compute, limits concurrency, so memory layout drives design.
P3
Measure against the real thing
Claims are benchmarked against actual vLLM (under WSL2) and a HuggingFace baseline on the same GPU.
P4
Compatibility over novelty
An OpenAI-compatible API means adoption needs no client changes.
P5
Scheduler isolated from I/O
Network handling never sits on the token-generation critical path.
Alternatives Rejected
Understanding the solution space, not just the final implementation.
It works, but adds a VM layer, extra memory and setup friction. Native Windows support was the point.
What I Would Change Today
A self-critique of the architecture.
- Add multi-GPU support (tensor parallel) before any claim of serving beyond one card.
- Automate benchmark runs in CI so regressions against vLLM are visible per commit.
- Investigate speculative decoding, which targets the same decode bottleneck from another angle.
My Role
What I owned, co-designed, influenced, and implemented personally.
Owned
- · Engine architecture
- · Scheduler design
- · Benchmark methodology
Co-designed
- -
Influenced
- -
Implemented personally
- · PagedAttention CUDA kernel
- · KV block manager
- · Continuous batching
- · CUDA-graph decode
- · OpenAI-compatible API