Forward Deployed Engineer - Project Case Study · 2024 – Present
LUNA: Conversational AI Engine
View sourceEngagement Snapshot
What I built, who it is for, and where my ownership starts and stops.
Local-first Conversational AI Engine
A production-grade AI assistant that runs entirely on the user's machine: multi-turn dialogue, an on-device voice pipeline, persistent memory, health-device data, desktop automation, and skills written in plain text. Zero cloud dependency by default.
Role
Sole architect and engineer
Scope
Electron shell, React UI, FastAPI backend, memory, voice, skills, integrations, packaging
Customer & Problem
Who this is for, the messy reality they work in, and why the system needed to exist.
Who it is for
Individuals and small teams who want an assistant they own: private by default, model-agnostic, and able to act on their desktop.
What was wrong before
Assistants were either cloud products that own your conversations, memory and health data, or local demos with no memory, no voice and no way to extend them.
Who needed it
Individuals and small teams who want an assistant they fully control: private by default, model-agnostic, and able to actually act on their desktop and calendar.
If nothing changed
Every capability stays rented: per-token bills, subscriptions, data leaving the machine, and a hard dependency on one vendor's model and uptime.
The environment I had to work in
- Users run wildly different hardware, from laptops to GPU desktops, and Windows, macOS and Linux.
- Some want fully local models; others want a hosted frontier model. Both must work.
- Voice, camera and desktop automation need OS-level access a browser cannot give.
- A personal tool and a multi-user business tool have very different security needs.
Discovery & Requirements
The questions I ask first, and the requirements every decision below traces back to.
Discovery questions
- 01Local only, or is a cloud provider acceptable?
- 02What hardware is available for local models?
- 03Which integrations matter: calendar, health devices, messaging channels?
- 04Single user, or a team that needs auth and rate limits?
Requirements
System Architecture
The centerpiece. The diagram is the easy part - the value is in why each boundary exists.
Why this boundary exists · Inputs
Browser, Electron, Telegram, Discord, Slack and webhooks all normalise into one message shape, so every surface gets identical behaviour.
Click any component to see why it is a separate boundary.
Integrations & Data
Where each category of information lives, what it connects to, and why.
Critical Architecture Decisions
Architecture decision records: the problem, the options, the call, and when to revisit it.
ADR-001 - Memory storage
Options considered
Decision
SQLite + ChromaDB, split by access pattern.
- Problem
- Conversation needs exact recall of what happened and fuzzy recall of what is relevant, on a single-user machine with no ops team.
- Reasoning
- SQLite is the durable episodic record (embedded, transactional, zero setup). ChromaDB serves semantic retrieval. Each store does the one thing it is best at.
- Trade-offs
- Two stores to keep consistent; a fact written to one must be reflected in the other.
- Revisit when
- If Business mode needs many concurrent writers, move the relational side to PostgreSQL.
Deployment & Handoff
How it gets installed and shipped, what it runs on, and how it is handed over so the customer can run it.
Git.
Rollout and handoff
One-command setup
A setup wizard picks the variant, configures the LLM provider, installs dependencies and pulls local models. The app opens locally in about two minutes.
Packaged for the platform
Desktop installers for the Electron shell, plus Docker compose variants for cloud, GPU and business deployments.
Personal vs Business by config
One setting switches on JWT auth, rate limiting and messaging channels, so there is one codebase to support.
Documented for others
A docs site, contributing guide and security policy ship with the repository.
Reliability Engineering
Failure thinking: what breaks, what catches it, and what actually went wrong.
Served by Primary model
Security Architecture
Every request passes these gates in order.
Authenticate.
Local by default
No telemetry or cloud calls unless a hosted provider is configured. Secrets live in a local .env that is never committed.
Business mode
JWT auth, rate limiting and per-user isolation for multi-user deployments.
Tool authorization
Desktop automation and integrations run behind a permission manager and are recorded in an audit log.
Prompt injection
Fetched web content is treated as untrusted data, never as instructions to the tool layer.
Results
Only outcomes that can be substantiated. Adoption figures are added when they are real.
Cost
Latency
Engineering
Field Lessons
What this project taught me about shipping AI into someone else's environment.
L1
Setup is part of the product
If installation is painful, nothing else matters. A wizard and sensible defaults matter as much as any feature.
L2
Local-first is a constraint that shapes everything
It forces honest choices about memory, model size and what runs where.
L3
Keep providers behind one interface
Model choice changes monthly, so no feature should depend on one vendor.
Request / Execution Flow
One real request traced through the system - dynamic behaviour, not just boxes.
User speaks / types.Any surface: desktop app, browser, Telegram, Discord, Slack, webhook.
Deep-Dive Architecture
The technically difficult pieces, step by step.
Input.A message arrives from the UI, voice, or a messaging channel.
AI Architecture
Model strategy, retrieval, and evaluation - the parts specific to an AI system.
Model strategy
Local model
Default for everyday chat and privacy-sensitive content. Runs on-device.
Hosted frontier model
Opt-in for hard reasoning and coding tasks when the user chooses a cloud key.
Fast / cheap inference
Latency-sensitive or high-volume paths through OpenAI-compatible endpoints.
Memory
Skill loading
Observability
The observability model, not a tool name.
Illustrative span layout - shows nesting, not measured durations.
Performance Engineering
Where the bottlenecks were, and what moved the numbers.
What caused the improvement
- Streaming first token to the UI instead of waiting for the full reply.
- Memory update runs after the response, off the critical path.
- Local inference removes network round-trips for the default path.
Only the UI-response figure is a published measurement; the baseline bar is relative.
Architecture Principles
The architecture followed from these, not from a shopping list of technologies.
P1
Local-first
Nothing leaves the machine unless the user picks a cloud provider. Privacy is the default, not a setting.
P2
Model independence
The engine never assumes one provider. Eight backends sit behind one interface.
P3
Plain-text extensibility
Skills are text files. Adding a capability should not require redeploying the core.
P4
Memory is a system, not a prompt
Episodic and semantic stores, fact extraction and compaction, so context survives beyond the window.
P5
One codebase, two modes
Personal and Business differ by configuration, so security fixes land once.
P6
Failure isolation
A failing provider, tool or integration must not take down the conversation loop.
Alternatives Rejected
Understanding the solution space, not just the final implementation.
The product promise is ownership: private data, no subscriptions, no vendor lock-in. A hosted backend would contradict that.
What I Would Change Today
A self-critique of the architecture.
- Move Business-mode persistence off SQLite before multi-user write concurrency becomes the bottleneck.
- Add sandboxing for skills that execute code, rather than relying on permissions alone.
- Introduce an evaluation harness so memory and retrieval changes are regression-tested, not judged by feel.
My Role
What I owned, co-designed, influenced, and implemented personally.
Owned
- · Overall architecture
- · Memory system
- · LLM router
- · Voice pipeline
Co-designed
- -
Influenced
- -
Implemented personally
- · Backend services
- · Electron shell and React UI
- · Skills runtime
- · Packaging and Docker variants