Forward Deployed Engineer - Project Case Study · 2024 – 2025
MedRAG: Clinical Dialogue System
Engagement Snapshot
What I built, who it is for, and where my ownership starts and stops.
Guardrailed RAG for Clinical Q&A
A production RAG system for multi-turn clinical questions, grounded in 18K+ medical papers. Every claim must cite a retrieved passage; ungrounded answers are blocked at the guardrail layer. Built on top of four peer-reviewed healthcare AI publications.
Role
Lead engineer: retrieval, guardrails, serving
Scope
Ingestion, two-stage retrieval, guardrail layer, dialogue memory, API
Customer & Problem
Who this is for, the messy reality they work in, and why the system needed to exist.
Who it is for
Clinicians and medical researchers who need fast, cited answers from a large body of literature.
What was wrong before
Clinical questions demand grounded, citable answers. A plain LLM hallucinates confidently and cannot show its evidence.
Who needed it
Clinicians and researchers who need multi-turn Q&A over a large medical literature, with patient-safety guardrails.
If nothing changed
Unsupported claims in a clinical context are a safety risk, and a system that cannot cite sources cannot be trusted or audited.
The environment I had to work in
- A wrong answer is a safety issue, so unsupported claims cannot be allowed through.
- Medical language is dense: abbreviations, drug names and study jargon defeat generic embeddings.
- Users ask follow-ups, so context has to carry across turns without drowning the prompt.
- Clinical users will not trust what they cannot trace back to a paper.
Discovery & Requirements
The questions I ask first, and the requirements every decision below traces back to.
Discovery questions
- 01What questions do clinicians actually ask, and how do they phrase them?
- 02What must the system refuse to answer?
- 03Which sources count as authoritative?
- 04How is a good answer judged, and by whom?
- 05What is acceptable latency at the point of care?
Requirements
System Architecture
The centerpiece. The diagram is the easy part - the value is in why each boundary exists.
Why this boundary exists · Clinician query
A typed API boundary validates input and streams responses.
Click any component to see why it is a separate boundary.
Integrations & Data
Where each category of information lives, what it connects to, and why.
Critical Architecture Decisions
Architecture decision records: the problem, the options, the call, and when to revisit it.
ADR-001 - Retrieval design
Options considered
Decision
Dense + cross-encoder rerank.
- Problem
- A single retriever trades recall for precision and latency.
- Reasoning
- MedCPT dense retrieval shortlists cheaply; a cross-encoder fine-tuned on clinical QA reranks for precision.
- Trade-offs
- An extra model in the latency path.
- Revisit when
- If lexical matching of drug names and codes shows gaps, add hybrid retrieval.
Deployment & Handoff
How it gets installed and shipped, what it runs on, and how it is handed over so the customer can run it.
Git.
Rollout and handoff
Corpus ingestion is repeatable
Papers are embedded once and the index can be rebuilt from the source corpus.
Fail closed by default
The guardrail blocks any answer without a citation, so the safe behaviour is the default behaviour.
Citations in every answer
Users can open the source passage behind each claim, which is what makes the system reviewable.
Golden question set
A fixed set of clinical questions is re-run on every change to catch regressions in retrieval and groundedness.
Reliability Engineering
Failure thinking: what breaks, what catches it, and what actually went wrong.
Served by Grounded answer
Security Architecture
Every request passes these gates in order.
Authenticate.
Patient safety
The guardrail fails closed: no citation, no answer.
PII
The system answers from published literature; user inputs are treated as sensitive and minimised in logs.
Prompt injection
Retrieved text is treated as evidence, not instructions.
Results
Only outcomes that can be substantiated. Adoption figures are added when they are real.
Performance
Latency
Engineering
Field Lessons
What this project taught me about shipping AI into someone else's environment.
L1
Refusing is a feature
A cited refusal is better than a fluent guess when the domain is clinical.
L2
Two-stage retrieval pays for itself
A cheap high-recall shortlist followed by a precise reranker beats either alone.
L3
Safety should not depend on the generator
The guardrail sits outside the model so it cannot be prompted away.
Request / Execution Flow
One real request traced through the system - dynamic behaviour, not just boxes.
Question.Clinician asks in a multi-turn chat.
Deep-Dive Architecture
The technically difficult pieces, step by step.
Query.Clinician question with recent turns.
AI Architecture
Model strategy, retrieval, and evaluation - the parts specific to an AI system.
Model strategy
Embedding model
MedCPT bi-encoder for biomedical retrieval.
Reranker
Cross-encoder fine-tuned on clinical QA pairs.
Generator
LLM constrained to cite retrieved passages.
Retrieval
Evaluation
Observability
The observability model, not a tool name.
Illustrative span layout - shows nesting, not measured durations.
Performance Engineering
Where the bottlenecks were, and what moved the numbers.
What caused the improvement
- Rerank only the shortlist instead of the full corpus.
- Stream tokens as soon as generation begins.
- Vector search on a dedicated ANN index.
Only the < 180 ms TTFT figure is a published measurement; the comparison bar is illustrative.
Architecture Principles
The architecture followed from these, not from a shopping list of technologies.
P1
Grounded or silent
If it cannot cite evidence, the system says so instead of guessing.
P2
Two-stage retrieval
Cheap recall first, expensive precision second.
P3
Safety is a separate layer
Guardrails sit outside the generator so they cannot be prompted away.
P4
Domain-tuned components
Medical embeddings and a clinically fine-tuned reranker beat generic ones.
P5
Multi-turn aware
Salience decays over turns so old context fades rather than pollutes.
Alternatives Rejected
Understanding the solution space, not just the final implementation.
The literature changes and answers must cite sources; retrieval provides both freshness and traceability.
What I Would Change Today
A self-critique of the architecture.
- Add hybrid lexical + dense retrieval for drug names and codes.
- Build a larger golden set with clinician review to track groundedness over time.
- Track guardrail refusals to find retrieval gaps rather than treating them as final.
My Role
What I owned, co-designed, influenced, and implemented personally.
Owned
- · Retrieval architecture
- · Guardrail design
Co-designed
- · Evaluation approach
Influenced
- · Healthcare AI publications
Implemented personally
- · Ingestion
- · Two-stage retrieval
- · API