Forward Deployed Engineer - Project Case Study · Jan – May 2025
TITANS: Long-Context Dialogue Memory
Engagement Snapshot
What I built, who it is for, and where my ownership starts and stops.
Memory-Augmented Transformer for Long Conversations
A memory-augmented transformer that lets attention read across unbounded history through a persistent memory module, without paying quadratic attention cost. A fused Triton kernel merges attention and memory read into a single pass.
Role
Research engineer: architecture, kernel, evaluation
Scope
Model architecture, Triton kernel, quantisation, serving integration, benchmarking
Customer & Problem
Who this is for, the messy reality they work in, and why the system needed to exist.
Who it is for
Teams building conversational systems that need to remember long multi-turn exchanges without paying quadratic attention cost.
What was wrong before
A standard KV cache discards context beyond the window, which is the failure point in long multi-turn conversations. Extending the window makes attention cost quadratic.
Who needed it
Conversational AI systems that need to remember what was said many turns ago, especially for multi-hop reasoning across turns.
If nothing changed
Assistants forget earlier facts, or pay memory and latency costs that make long conversations unaffordable to serve.
The environment I had to work in
- A standard KV cache loses context beyond the window.
- Longer windows make attention cost grow quadratically.
- Results only count if they hold up in a real serving stack.
Discovery & Requirements
The questions I ask first, and the requirements every decision below traces back to.
Discovery questions
- 01How long are real conversations?
- 02Which tasks need recall across turns?
- 03What latency and throughput targets does serving have?
Requirements
System Architecture
The centerpiece. The diagram is the easy part - the value is in why each boundary exists.
Why this boundary exists · Tokens
Standard tokenised input keeps the model drop-in compatible with existing tooling.
Click any component to see why it is a separate boundary.
Integrations & Data
Where each category of information lives, what it connects to, and why.
Critical Architecture Decisions
Architecture decision records: the problem, the options, the call, and when to revisit it.
ADR-001 - How to carry long context
Options considered
Decision
Persistent memory module.
- Problem
- Context beyond the window is discarded by a KV cache; enlarging the window is quadratic.
- Reasoning
- Lets attention read across unbounded history at fixed cost and stays inside the model, avoiding retrieval hops.
- Trade-offs
- Custom architecture and kernel to maintain; not a drop-in checkpoint.
- Revisit when
- If off-the-shelf long-context models match recall at comparable cost.
Deployment & Handoff
How it gets installed and shipped, what it runs on, and how it is handed over so the customer can run it.
Git.
Rollout and handoff
Served through vLLM and FastAPI
The model is evaluated in a realistic serving stack, not only in a notebook.
Quantised for deployment
Lower-precision weights reduce memory and latency at serving time.
Reliability Engineering
Failure thinking: what breaks, what catches it, and what actually went wrong.
Served by Fused kernel
Security Architecture
Every request passes these gates in order.
Authenticate.
Research scope
Serving is behind FastAPI with standard input validation; the project is model research, not a multi-tenant platform.
Results
Only outcomes that can be substantiated. Adoption figures are added when they are real.
Performance
Latency
Field Lessons
What this project taught me about shipping AI into someone else's environment.
L1
Measure where it will run
Latency and throughput are only meaningful in the serving engine you will deploy.
L2
Fuse what shares data
Merging attention and memory read into one pass removed duplicated I/O.
Request / Execution Flow
One real request traced through the system - dynamic behaviour, not just boxes.
Turn arrives.New user message.
Deep-Dive Architecture
The technically difficult pieces, step by step.
Current window.Recent tokens attend locally.
AI Architecture
Model strategy, retrieval, and evaluation - the parts specific to an AI system.
Model strategy
Baseline
Standard transformer with KV cache: the comparison point.
TITANS variant
Memory-augmented model with fused kernel.
Quantised serving
Lower-precision weights for latency and throughput.
Evaluation
Observability
The observability model, not a tool name.
Illustrative span layout - shows nesting, not measured durations.
Performance Engineering
Where the bottlenecks were, and what moved the numbers.
What caused the improvement
- Fusing attention and memory read into one Triton pass avoids double I/O.
- Persistent memory replaces ever-growing KV context for old turns.
- Quantisation reduces weight bandwidth.
Bars are relative to the baseline (=100) using the published percentages.
Architecture Principles
The architecture followed from these, not from a shopping list of technologies.
P1
Memory as a module
Long-term recall is an explicit component, not an ever-larger window.
P2
Fuse what shares data
If two ops read the same tensors, do them in one kernel.
P3
Validate on reasoning, not just perplexity
BoolQ and GSM8K require using remembered information across steps.
P4
Measure end to end
Latency and throughput are measured in a serving stack, not a notebook.
Alternatives Rejected
Understanding the solution space, not just the final implementation.
Attention cost grows quadratically with window length; memory cost here stays fixed.
What I Would Change Today
A self-critique of the architecture.
- Evaluate on more purpose-built long-dialogue benchmarks than BoolQ and GSM8K.
- Study memory saturation and eviction behaviour over very long conversations.
- Compare against newer long-context and state-space models on identical hardware.
My Role
What I owned, co-designed, influenced, and implemented personally.
Owned
- · Architecture
- · Evaluation design
Co-designed
- -
Influenced
- -
Implemented personally
- · Fused Triton kernel
- · Quantisation
- · Serving integration
- · Benchmarks