Tatvik

Tatvik is a local-first AI assistant. Everything — hotword detection, speech-to-text, intent parsing, and language generation — runs on-device. It was born from a simple question: can a useful voice assistant exist without a cloud dependency?

The answer turned out to be yes, with some hard trade-offs that are worth documenting.

The Problem

Most assistants are thin clients for remote models. That architecture is convenient, but it has real costs:

• Privacy — audio and text never leave the machine.
• Latency — no round-trips for the audio path.
• Reliability — works without internet.

The obvious trade-off is compute. Consumer laptops can run small LLMs comfortably, but a voice assistant also needs real-time audio processing and low-latency wake detection.

Architecture

The system runs in four layers, each owning a single responsibility:

Audio API → Rust Core (hotword) → Whisper STT → Intent Router → LLM / Tools

1. Audio capture — Rust owns the mic, pushes frames through a lock-free channel. No GC pauses.
2. Hotword detection — DSP runs in the Rust core for sub-300ms wake latency.
3. Speech-to-text — Whisper runs in a Python worker, streaming partial transcripts.
4. Intent routing — Low-confidence utterances go to the LLM; high-confidence ones hit matched tools directly.

Key decisions:

• Rust for the audio path. Hotword detection must be interruption-free; a GC pause in the middle of an audio buffer is unacceptable. The Rust core owns the mic and pushes frames to a lock-free channel.
• Python for the NLP layer. Whisper and the model ecosystem are strongest in Python. Keeping them in a separate worker process isolates crashes and lets each layer scale independently.
• One model, one process. The local LLM (a 1.2B-parameter Q4 model) runs in its own process so a long generation never stalls the STT pipeline.

Engineering decisions

1. Chunked streaming STT. Transcribing the whole utterance after recording adds startup latency. Whisper streaming gives partial transcripts that let the intent router start early.
2. Fallback chain. If intent confidence is low, route to the LLM with the raw transcript. If the model is busy, queue the request rather than dropping it. These fallbacks matter more than accuracy benchmarks.
3. Disk as memory. Session state and user profile live in SQLite, so Tatvik survives restarts and reboots without a server.

@dataclass
class Turn:
    transcript: str
    confidence: float
    wake_time: float
 
def route(turn: Turn) -> Intent:
    if turn.confidence < 0.6:
        return Intent.LLM_GENERAL
    return intent_classifier(turn.transcript)

Benchmarks

These numbers are on a M1-class laptop — a 2020 baseline. The point isn't that the numbers are exceptional; it's that they're good enough on commodity hardware.

Challenges

The hardest problem was audio ownership. The OS may remix or re-route audio (e.g., a notification) mid-turn. Rebuilding a clean audio pipeline that survives OS-level interrupts took longer than the entire rest of the stack.

The second was VRAM contention. STT and the LLM both want the GPU. The fix was strictly serializing heavy kernels and moving STT to CPU-bound workers when the GPU is generating.

Lessons

• Local-first is a constraint that sharpens design. Every abstraction had to justify itself against the "run on one machine" budget.
• Latency budgets beat accuracy curves. A 30 ms STT improvement moves the product more than a +1 accuracy point on a benchmark nobody feels.
• Open-source it early. Shipping to GitHub forced better docs, cleaner APIs, and honest READMEs.

Roadmap

• Tool calling so Tatvik can actually do things (calendar, files).
• Multi-language wake word and STT.
• A GUI control panel to tune models and latency without touching config files.