Skip to main content
Architecture Decision Records (ADRs) capture significant architectural decisions made during the development of agtOS. Each ADR describes a single decision, its context, the rationale behind it, and the consequences of adopting it.

​
Why ADRs?

agtOS went dormant from August 2025 to March 2026 — seven months during which the AI landscape shifted dramatically. MCP moved to the Linux Foundation, Piper TTS was archived, new voice architectures emerged, and local models became viable for agentic tool calling. ADRs ensure that:
  • Future sessions have context — when returning after any gap, ADRs explain why things are the way they are
  • Decisions are traceable — every choice links to specific technical context and trade-offs
  • Alternatives are documented — knowing what was not chosen (and why) is as valuable as knowing what was
  • Onboarding is faster — new contributors can read the ADR index to understand the system’s evolution
ADRs are immutable once accepted. If a decision is reversed or significantly modified, the original ADR is marked as deprecated and a new ADR references it. This preserves the historical reasoning behind every decision.

​
ADR Index

The first 10 ADRs (001-010) were created together on 2026-03-22 to document decisions made upon resuming development after the 7-month hiatus. ADRs 011-020 were created individually during active development. ADRs 021-027 (2026-04-06 through 2026-04-08) cover the Memory Enhancement, Zod 4, timezone-aware scheduling, atomic profile updates, multi-tenant readiness, provider catalog, and NLI hybrid contradiction pipeline work. ADRs 028-037 (2026-04-10 through 2026-04-12) cover the Personal AI Gateway thesis, PACT capture protocol, entity-centric memory, speaker intelligence, legal compliance, consumer desktop experience, health tiers, billing exhaustion, desktop chat UI, and provider-agnostic thinking/vision.

​
Key Architectural Principles

These principles emerge from the ADR collection and guide ongoing development:

​
Protocol-First Design

Every integration is protocol-defined, not provider-specific. Protocols define interfaces; implementations are swappable. This applies to voice providers (ADR-002, ADR-010), LLM providers (ADR-003, ADR-004), tool integration (ADR-001, ADR-009), and audio architectures (ADR-008).

​
Local-First Where Possible

The model router (ADR-004) routes simple queries to local Ollama models, reducing cost and latency while enabling offline operation. Privacy-sensitive requests never leave the local network. Cloud is reserved for complex reasoning that exceeds local capabilities.

​
Infrastructure/Orchestration Separation

The dual-layer architecture ensures the orchestration layer does not know or care which voice pipeline mode is active (ADR-008). Whether using cascade STT->LLM->TTS, half-cascade audio LLMs, or native end-to-end models, the orchestration logic remains identical.

​
Backward-Compatible Extension

New capabilities are added as optional extensions to existing interfaces. Platform-aware routing (ADR-015) adds a secondary lookup map without changing default behavior. Dynamic tool selection (ADR-009) filters tools before the LLM sees them without changing tool definitions.

​
Security by Default

BYOK credential management (ADR-011) encrypts keys at rest with AES-256-GCM, scrypt key derivation, and AAD-bound ciphertext. Credential operations are instrumented with Prometheus metrics and structured correlation IDs. API security (ADR-014) uses timing-safe comparison and token bucket rate limiting. Device authentication uses per-device SHA-256 tokens. These are not bolt-on features — they were designed into the architecture from the start.

​
ADR Deep Dives

Problem: MCP was the sole integration protocol, but the protocol landscape shifted. MCP joined the Linux Foundation alongside Google’s A2A. MCP has a context window problem (tool schemas consume 72% of 200K context). MCP does not address agent-to-agent coordination or frontend streaming.Decision: Build a gateway abstraction (OrchestratorGateway interface) with protocol-specific adapters. MCP adapter is first and primary. A2A and AG-UI adapters can be added without modifying orchestration logic.Trade-off: Adds an abstraction layer that may be premature until a second protocol is needed. But the cost of the abstraction is low, and the cost of restructuring later is high.
Problem: The voice pipeline has two fundamentally different interaction patterns — real-time conversation (streaming, low latency) and background tasks (multi-step, autonomous).Decision: Use both Anthropic SDKs. Client SDK (@anthropic-ai/sdk) for voice path with streaming. Agent SDK (@anthropic-ai/claude-agent-sdk) for background tasks with agentic loops. Both connect to the same MCP servers.Trade-off: Two SDK integrations to maintain, two authentication flows. But optimizes each path for its requirements — voice gets minimum latency, background gets full agentic capability.
Problem: Cloud API calls cost money, add latency, and send data off-device. Local models (Qwen3.5 27B) now score competitively on function calling benchmarks. Most voice interactions are simple enough for local models.Decision: Three-tier routing. Tier 1: intent classification via a micro-model (under 50ms). Tier 2: local Ollama for simple requests. Tier 3: Claude for complex reasoning. Automatic fallback between tiers.Trade-off: Routing complexity is higher than a single API call. Misclassification degrades experience. But cost savings are substantial for high-volume voice interactions.
Problem: True AI agents need memory that persists across sessions. Working memory (in-context) is insufficient for long-term recall.Decision: Three-tier memory architecture. Working memory: per-session conversation history with automatic LLM summarization. Episodic memory: cross-session recall via Redis with heuristic save decisions. Semantic memory: embedding-based vector search using Redis Vector Search and Ollama embeddings.Trade-off: Redis Vector Search is less capable than dedicated vector databases (Pinecone, Weaviate). But it avoids adding another infrastructure dependency — Redis is already required for sessions and scheduling.
Problem: Voice AI has evolved from cascade-only (STT->LLM->TTS) to three architectures with different cost/latency/quality profiles. Locking into cascade limits future options.Decision: Support all three modes through the infrastructure layer. CASCADE (default, ~500ms, ~0.15/min),HALFCASCADE(Ultravox, 300ms),NATIVE(GeminiLive/OpenAIRealtime, 200ms, 0.15/min), HALF_CASCADE (Ultravox, ~300ms), NATIVE (Gemini Live / OpenAI Realtime, ~200ms, ~1.50/min). The orchestration layer does not change.Trade-off: Three variants multiplied by multiple providers creates a large test matrix. But this validates the dual-layer architecture and positions agtOS for the native audio future.
Problem: MCP tool definitions consume 550-1,400 tokens each. With 20+ tools, 72% of a 200K context window is consumed before any conversation. This is MCP’s structural context window problem.Decision: Intent-to-category mapping with top-N tool selection. The intent classifier determines which tool categories are relevant, and only those tools are loaded into context. Achieves 80-90% context reduction.Trade-off: If the classifier picks the wrong category, the needed tool is not available. Mitigation: always include a “general” fallback category.
Problem: agtOS runs across multiple platforms (Node.js server, desktop, ESP32), but some tools are only available on specific platforms. There was no way to restrict tool availability per platform.Decision: Add platform-specific adapter overrides in the gateway. Tools can declare which platforms they support via metadata. The gateway filters unavailable tools before the LLM sees them. Backward compatible — tools without platform metadata are available on all platforms.Trade-off: Adds a secondary lookup map and increases tool metadata complexity. But keeps the orchestration layer platform-agnostic while enabling platform-specific optimizations.
Problem: agtOS depended on speaches, a Python-based STT/TTS server with a single maintainer who disappears for months. Critical bugs (transcription hallucination, silence crash) sat unmerged for 39-67+ days. The v0.9.0 release was stuck in RC for 6+ months. Every STT/TTS call required a cross-process HTTP round-trip to the Python sidecar.Decision: Replace speaches with sherpa-onnx-node (v1.12.34), a pre-compiled N-API native addon that runs STT, TTS, and VAD directly in the Node.js process via ONNX Runtime. This brings 17+ STT models (Whisper, Moonshine, SenseVoice, Zipformer, Paraformer), 7 TTS families (Kokoro, Piper, Matcha, ZipVoice), and Silero VAD into a single process. GPU acceleration is available via CUDA (Linux), CoreML (macOS), and DirectML (Windows). speaches is preserved as a config-switchable fallback.Trade-off: Whisper uses greedy decoding only (no beam search), model files require ~460MB local storage, and TypeScript type declarations (~300 lines) must be maintained manually. But the elimination of the Python sidecar, HTTP overhead, and single-maintainer dependency risk far outweighs these costs.
Problem: agtOS has multiple specialized AI tasks (embedding, classification, reasoning, consolidation, summarization) that each have different cost/latency/quality trade-offs. Using a single provider for all tasks was inefficient.Decision: Allow independent provider selection for each cognitive task. Each task can route to its optimal provider (ollama for local inference, claude for reasoning, openrouter for fallback). This is configured via environment variables and can be changed at runtime via PUT /api/settings.Trade-off: More configuration complexity and multiple API keys to manage. But enables fine-grained optimization — e.g., local Ollama for embeddings (cost-free) while routing complex reasoning to Claude.
Problem: The two-tier LOCAL/CLOUD routing model forced a single cloud provider for all requests. With OpenAI added as an alternative (ADR-019), there was no way to use different providers for different request types — e.g., Claude for reasoning and OpenAI for tool calling. The AGTOS_CLOUD_PROVIDER env var was a global switch that couldn’t express per-capability preferences.Decision: Replace the two-tier model with a Model Slot Registry. Each slot is a named capability position (chat, reasoning, coding, tool_calling, creative) that maps to a configured provider + model pair. The intent classifier routes requests to slot names, and the registry resolves slots to live provider instances. Task slots (embedding, classifier, summarization, consolidation, dialectic, maintenance) handle background cognitive tasks. Configuration lives in ~/.agtos/config.json under a slots key. Each slot supports fallback chains to handle provider failures.Trade-off: More configuration complexity — users must understand slots to customize routing. But the default agtos setup wizard configures the chat slot (required) and optional reasoning slot, which covers the common case. Power users get fine-grained control over which provider handles which kind of request.
Problem: The MemoryDreamer consolidation engine was implemented with 60+ tests but triggerConsolidation() was never called from anywhere in the codebase — user profiles were always empty. Even when wired, the Dreamer would compete with active voice sessions for local Ollama GPU/VRAM (Ollama queues requests when VRAM is full). And there was no mechanism to age out stale conclusions, detect contradictions, or prune redundant facts as the knowledge base grew.Decision: Five complementary changes that preserve the three-tier architecture (ADR-007):
  1. Wire the Dreamer — call triggerConsolidation() in endVoiceSession() plus a server-level sessionEnded event listener as defense-in-depth.
  2. ResourceGuard — gate every background LLM call via a deterministic decision tree: policy override → cloud/remote short-circuit → active sessions → session cooldown → system load → Ollama VRAM probe (GET /api/ps). Configurable policy (auto / always / idle-only). Windows caveat: os.loadavg() returns zeros, so VRAM probe is the only strong signal.
  3. qmd MCP configuration + tool category inference — AGTOS_MCP_SERVERS JSON env var for external MCP servers. inferCategory() maps tool name/description to intent categories so search tools auto-participate in dynamic tool selection (ADR-009).
  4. Query-as-Ingest — a new RESPONSE_INGEST episode type. Heuristic scoring (tool calls, multi-step reasoning, length, synthesis patterns) decides when to persist a high-quality agent response so synthesis compounds across sessions.
  5. Maintenance Mode — Dreamer.maintain() runs a six-step sweep (stale detection, confidence decay, redundancy merge, orphan flagging, LLM contradiction check, low-confidence prune). Auto-registered cron task (default 0 3 * * *), POST /api/memory/maintain for on-demand runs, memory-maintenance health check flags runs older than 48 hours.
Trade-off: More moving parts in the MemoryCoordinator, and maintenance LLM calls cost tokens on cloud providers. Mitigated by per-sweep caps, the cron schedule, and ResourceGuard’s skip-on-load. The contradiction detection portion of Step 5 is superseded by ADR-027’s NLI hybrid pipeline when its dependencies are wired — ADR-021 still owns ResourceGuard, the cron schedule, the kill switch, dangling-source detection, and the maintenance health check.
Problem: agtOS ships single-user today (one credential file, one global API key, one Tauri sidecar per machine), but the long-term roadmap includes shared deployments for business teams and family households. Without discipline, adding real multi-user support later means hunting down every hardcoded 'default' literal and migrating Redis keys — painful and bug-prone. Industry postmortems (Kestra 2024, WorkOS guidance) converge on: “the data layer must be multi-tenant-aware from day one, even if the operational layer is not.”Decision: Adopt seven code-review rules enforced by discipline rather than infrastructure:
  1. Never hardcode 'default' as a userId — use resolveUserId(config) from @/core/types.
  2. Tenant-first Redis keys: agtos:{subsystem}:{userId}:{entity} for all user-scoped data.
  3. No optional userId in client-facing API request schemas; the server derives it from profileManager.getDefaultUserId(). Schemas are .strict().
  4. No premature auth middleware — the single global AGTOS_API_KEY is correct for single-user. The only acceptable infrastructure is type UserId = string and resolveUserId(config).
  5. Document the contract in any new ADR that introduces persistence.
  6. Every vector search must accept a userId parameter.
  7. Background jobs carry userId at task creation time, not at execution time.
Trade-off: Requires code-review vigilance to enforce (no linter catches all violations). But the migration path when real multi-user ships is small: only resolveUserId() and the credentials store change. All other code stays identical because the data layer is already correct.
Problem: The dashboard’s slot configuration UI, the agtos setup wizard, and the CLI all need to discover what models are available from each provider — but every provider SDK has a different shape. Anthropic exposes rich ModelInfo with capability flags; OpenAI exposes only id and owned_by; Ollama needs list + show fan-out; OpenRouter has its own /api/v1/models endpoint with string-encoded pricing. Meanwhile, OpenRouter was being treated as “OpenAI with a different baseURL”, which caused credential scope confusion, lost access to rich catalog metadata, and mis-layered ranking headers (HTTP-Referer, X-Title).Decision: Define a ProviderCatalog interface at src/core/providers/catalog/types.ts with listModels() (returns Result<ModelInfo[]>), optional getAccountInfo(), and optional validateModel(). ModelInfo carries a 13-entry finite capability union (including 'contradiction' for KB maintenance), context length, pricing per 1M tokens, and provider-specific fields (sizeBytes for Ollama, upstreamProvider for OpenRouter). Four implementations ship: OpenRouterCatalog, OllamaCatalog, ClaudeCatalog, OpenAICatalog, all with 1-hour TTL caching. OpenRouter is promoted to a first-class provider under src/providers/openrouter/ with its own credential scope (provider-openrouter), a client-provider (OpenAI SDK with baseURL override plus attribution headers), and a new maintenance task slot (6th entry in TASK_SLOTS) for the Stage 3 LLM judge in the NLI hybrid pipeline.Trade-off: Four catalog implementations to maintain plus a capability map for OpenAI (whose API doesn’t expose capabilities). But the dashboard, setup wizard, and NLI hybrid pipeline (ADR-027) all get a single provider-agnostic query path, and OpenRouter finally gets its own credential scope and pricing data.
Problem: The Memory Maintenance V1 design (ADR-021) sent every conclusion in the profile to a single LLM call to audit for contradictions. At ~500 conclusions per profile, modern frontier models drop pairs from the middle of the list (“attention dilution”) and hallucinate conclusion IDs. Published benchmarks show ~71% recall for this flat-list approach vs. ~87.8% for ContraGen + cross-encoder hybrids. The single-stage path is also expensive — paying frontier rates to sift through pairs that a cheaper pre-filter could rule out — and there’s no principled way to upgrade only the contradiction step.Decision: Ship a 3-stage hybrid contradiction detection pipeline that replaces the single-LLM detector inside Dreamer.maintain() Step 5 when its dependencies are wired:
  1. Stage 1 — Candidate selection: CandidateSelector computes an in-memory cosine over conclusion embeddings, caches by id + textHash, selects top-K nearest plus an “interesting pair” priority heuristic, deduplicates, and truncates to 500 pairs.
  2. Stage 2 — NLI cross-encoder + pair cache: NliClassifier runs onnxruntime-node@^1.24.3 on a CPU session with a quantized DeBERTa-v3-base MNLI model (223 MB, SHA-256 pinned, atomic-rename download). PairCache uses Redis HEXPIRE on 7.4+ with STRING+EX fallback, content-addressed pair keys, and a secondary index for O(k) invalidation.
  3. Stage 3 — Batched LLM judge: LlmJudge sends 10 pairs per call with a structured-JSON Zod-validated prompt. Per-batch isolation means a single failure logs a warning and contributes zero without aborting the sweep. Uses the hot-swappable maintenance task slot from ADR-026 so users can pin a cheaper / faster model for the judge.
Opt-out via AGTOS_NLI_ENABLED=false; legacy callers fall through to the V1 single-LLM path. A new memory.contradiction.detected event fires once per confirmation. MaintenanceReport.summary.contradictionPipeline carries per-stage counters and latencies.Trade-off: 223 MB model file to ship (prebuild via npm run prebuild:nli) and additional CPU work on sweep days. But recall goes up, token cost goes down, and each stage can be upgraded independently. Backwards compatibility is preserved: the existing 77 dreamer tests run unchanged against the V1 fall-through path.
Problem: agtOS evolved into a sophisticated orchestration system but lacked a unifying strategic thesis for feature prioritization, resulting in ad hoc decisions around voice hardening, memory improvements, and dashboard polish.Decision: Adopt the “Personal AI Gateway” framing as the core positioning — the identity + memory + routing layer between a person and every AI they touch. Prioritizes capture infrastructure, entity-centric memory, open protocols, and multi-surface support over single-surface optimization, guided by a 12-18 month market window.Trade-off: Scope expansion risk (capture, entities, protocols, multi-surface support each require significant effort) versus differentiation opportunity (no competitor occupies this exact position combining protocol-agnostic capture + NLI contradiction detection + local-first architecture).
Problem: Existing protocols (MCP, MCAP, LSL, openEHR) cover portions of capture but none address the “first mile” from device sensors to structured memory with provenance, presence attestation, and consent as first-class requirements.Decision: Design PACT (Presence Attestation & Capture Telemetry) with 11 core concepts: presence signals (10 mechanisms with trust grades), per-modality consent envelopes (audio/video/neural), jurisdiction-aware validation (defaulting to strictest two-party consent), witness delegation, local-first-only v1 architecture, and extensible mechanism/modality registries.Trade-off: Local-first-only v1 limits cloud sync/multi-device reach versus ensuring legal defensibility.
Problem: The three-tier memory system stores everything as free-form text, unable to answer structured queries like “what do I know about Alice?” or “who does Alice work with?” because there is no entity-level indexing or relationship graph.Decision: Implement EntityManager and RelationshipManager using Redis JSON + RediSearch, with automatic NER extraction via @huggingface/transformers (bert-base-NER), entity deduplication via alias matching and embedding similarity (0.85 threshold), and a wiki UX for browsing/editing entities and relationships.Trade-off: Storage overhead (~2 MB at 1000 entities) and NER accuracy ceiling (general-purpose model will miss domain-specific entities) versus structured entity queries and relationship traversal.
Problem: Without speaker identification, all voice input is attributed to “the user” and multi-person conversations become unusable for entity-aware memory; active-participant verification for PACT legal compliance requires proof the device owner’s voice is present.Decision: Wrap sherpa-onnx SpeakerEmbeddingExtractor with Redis JSON persistence, enrollment lifecycle (5-10s voice sample → embedding → re-enrollment via EMA), active-participant verification, and post-session diarization with per-speaker transcript sections.Trade-off: Enrollment friction and cold-start latency versus multi-speaker attribution in episodic memory and legal compliance for ambient capture.
Problem: PACT handles audio, video, and future neural data with significant legal implications across multiple jurisdictions — federal one-party consent, 12 US two-party states, EU AI Act biometric classification, Colorado/California neurorights, and COPPA age-detection paradox.Decision: Adopt 7 principles: (1) default active-participant mode only, (2) mandatory consent metadata in every envelope, (3) per-modality consent (audio ≠ video ≠ neural), (4) jurisdiction-aware validation (default to two-party), (5) explicit no-age-detection rule, (6) tool-not-service architecture (local-only, user control), (7) neural data requires explicit opt-in.Trade-off: User configuration burden and feature limitations (no age detection, no cloud sync) versus strong legal defensibility and future-proofing for neural data.
Problem: agtOS shipped with a 7-step setup wizard assuming Docker, terminal access, and API key familiarity; competitors deliver “just works” experiences where users double-click and start chatting.Decision: Simplify to a 3-step flow (Mode Selection → Redis → Done), implement just-in-time downloads (voice models on first mic click), SSE progress for all downloads, hardware-aware model recommendations via SystemCapabilities detection, and slot auto-configuration for cloud/local/hybrid modes.Trade-off: Redis still requires Docker for now (SQLite adapter deferred) versus reducing setup time from 30+ minutes to ~2 minutes and eliminating terminal commands.
Problem: Health aggregator treated all failures equally (any unhealthy = degraded); 6 capture subsystems always unhealthy on fresh install; Redis reconnection required full server restart.Decision: Add priority field to health checks (critical: ollama/sherpa/provider; important: cloud-providers/redis/mcp; optional: capture/memory/nli), return 200 only when critical services are healthy. Add POST /api/system/reconnect-redis to lazily create and connect Redis-dependent services on demand without restart.Trade-off: Fresh install shows green status even with optional capture unavailable versus slightly more complex ServerContext with mutable state indirection.
Problem: When cloud provider credits run out, users see cryptic errors indistinguishable from transient failures; each provider signals exhaustion differently (Anthropic billing_error, OpenAI insufficient_quota, OpenRouter 402).Decision: Create provider-specific mapApiError() to classify billing errors into PROVIDER_BILLING_EXHAUSTED, centralize detection in BillingDetector that marks all slots for that provider unhealthy simultaneously, and enable user-configured fallback strategy (cloud-backup/ollama-local/none).Trade-off: OpenAI SDK wastes 2 retries before detection versus proactive balance monitoring and cross-provider fallback chains.
Problem: Desktop app had no text chat interface; users without microphones or in noisy environments could only interact via CLI.Decision: Create POST /api/chat/stream SSE endpoint with fetch + ReadableStream, accumulate token deltas in rAF buffer before flushing to React state, and defer markdown AST construction until streaming completes. Includes thinking/reasoning block display and tool call visualization.Trade-off: SSE via POST requires manual stream parsing (no native EventSource support for POST) versus real-time streaming feedback in the dashboard.
Problem: Desktop chat lacked thinking/reasoning display and image/vision input; Claude, OpenAI (o-series), Ollama (DeepSeek-R1, Qwen3), and OpenRouter all support thinking in different formats.Decision: Add onThinking callback to AgentStreamCallbacks with per-provider extraction (Claude stream.on, Ollama think field, OpenRouter message.reasoning, OpenAI Responses API reasoning_summary_text). Preserve continuity tokens in Message type. Base64-encode images in request body with per-provider formatting.Trade-off: Per-provider extraction logic for incompatible thinking formats versus unified user experience with thinking visibility and multi-turn reasoning continuity.

​
Proposing a New ADR

1

Create a file

Copy the template into docs/adr/NNN-short-description.md using the next sequential number.
2

Fill all sections

Context, Decision, and Consequences (positive, negative, risks). Include version numbers, benchmarks, and references.
3

Set status to Proposed

Until reviewed and accepted.
4

Add to the index

Update the table in docs/adr/README.md.
5

Link to a GitHub issue

Every ADR should relate to a tracked issue.

​
ADR Template

​
Status Lifecycle

Full ADR text is available in the GitHub repository. Each ADR is self-contained — read only the ones relevant to your work.