Epistemic Kernel — Future Directions & Hardware Tiers

Status: exploratory, non-binding. Nothing in this document is a requirement. It exists so future expansion has a place to land without pressure to cram it into the MVP spec, and so the tiers below can guide whether a given feature belongs in the core project or as an optional add-on.

1. Why a separate document

The MVP spec defines a floor: broker + one attestor + one local model class, runnable on ~8–12GB VRAM. That floor should stay stable and small. Everything below is about what becomes possible once someone has more than the floor — without ever making any of it required to call the project “working.”

2. Hardware Tiers (draft)

These thresholds are rough and will drift as models get more efficient — treat this table as “current shape of the ladder,” not fixed forever.

3. Functional Expansion Ideas (unordered, exploratory)

• Vision-language input — an agent that can look at an image and form a belief about it. Interesting epistemic wrinkle: is the raw image itself an “attested fact” (it’s real sensor/file data) while the model’s description of it is a belief? Yes — this maps cleanly onto the existing Attested/Believed split without new kernel concepts.
• Image generation — resolved: generated images use the Provenance::Synthesized variant now defined in the core spec’s data model — distinct from both attested-fact and believed-inference, since “AI drew this” isn’t a claim about reality, it’s an artifact.
• Text-to-speech / speech-to-text — now folded into T1 as the real experiential floor (see tier table), not a later add-on. Mostly a transport/interface question, not an epistemic one. STT output is a belief about what was said (transcription error is a real failure mode — worth tagging confidence). TTS output isn’t information at all, just a rendering of existing belief/fact into another medium. Suggested tools: silero-vad for voice detection, whisper.cpp for STT, Piper or Kokoro for TTS — all CPU-friendly, minimal VRAM footprint.
• Embeddings / RAG-style retrieval — could plug in as another Attestor type: “the vector store returned this chunk” is attestable (it’s a real lookup), but “this chunk means X” is still a belief layered on top.
• Video / multi-frame vision — same shape as image understanding, just more expensive; no new epistemic category needed, just more VRAM.
• Multiple simultaneous specialist agents — e.g. a vision agent, a text-reasoning agent, and a TTS agent all running under one broker, each agent’s belief store separate, cross-agent belief-passing exercising the “no transitive trust” rule across genuinely different modalities, not just genuinely different text models. This might be the most interesting demo for the kernel specifically — provenance discipline mattering more, not less, as modalities multiply.

4. Ambient Device Ecosystem & Self-Authored Tools

Beyond local files/sensors, the same broker-mediated pattern extends outward to the home network and the internet — without needing every protocol hardcoded in advance.

Discovery as generic capability, not per-brand code. Devices already advertise themselves via standard protocols — SSDP/UPnP, mDNS/Bonjour, IPP for printers. One discovery layer listens for these broadcasts and turns whatever answers into a generic capability descriptor (“this renders audio,” “this prints,” “this displays images”) rather than hardcoding brand-specific integrations.

Every discovered device is just another attestor. “Is the speaker actually playing right now” is the same shape as “is this file actually on disk right now” — a live, re-checkable fact. No new kernel concept, just new attestor implementations per protocol, all speaking the same Attest interface.

A genuinely new primitive: Actuate(DeviceId, Command). Everything so far has been about querying truth. Turning on a speaker or sending a print job is acting on the world. The result of an Actuate call must itself be attestable (“is the speaker actually now playing X”) rather than trusted just because the command was sent — conflating “I told it to happen” with “it happened” is exactly the silent-belief-as-fact bug the kernel exists to prevent.

Self-authored tools (“the zygote”). Rather than hand-coding every protocol client, agents can be given a minimal primitive set and write their own tools on demand:

• Raw socket/UDP-multicast access (for SSDP/mDNS discovery)
• An HTTP client (most of these protocols ride on HTTP/XML or HTTP/IPP)
• A bounded code-execution sandbox to write and run tools in
• A registration mechanism so a working tool is reused rather than rewritten from scratch each time

A self-authored tool is itself an artifact with provenance (which agent wrote it, from what reasoning, when it was last verified against the real device it claims to control) — the same Synthesized-adjacent question raised by image generation. Its first several runs should be treated as low-confidence Believed, attested only after it demonstrably does what it claims against the real device — not trusted on the first try because it compiled.

Local vs. internet-facing tools are a different risk category, not a variation on the same one:

• Read operations on external services (check inbox, read a channel) — reasonable for self-authored tools under the same low-trust-until-verified pattern as local devices.
• Write/send operations on external services (send an email, post a message, delete something, purchase something) — self-authored tools should never auto-execute these on the first pass, full stop. This needs an actual human confirmation gate every time, not a tunable trust threshold — the blast radius of a wrong action in the world (an email sent, a message posted) is categorically different from a wrong belief.
• Credentials are never handed directly to a self-authored tool. The tool requests a capability (“send this via the email capability”); the broker — a trusted, human-audited component, not self-authored — holds the actual credential and performs the send. A buggy or compromised tool can construct a bad request; it can never leak or misuse the credential itself.

5. Security & Threat Model

This is arguably the most important section in this document — the more the system can do (self-authored tools, real devices, real accounts), the more it matters that the kernel’s discipline holds under active, hostile pressure, not just honest mistakes.

The core attack: content-as-instruction. Anything an agent reads from outside your control — an email, a message, a webpage, a discovered device’s self-description — can contain text aimed at manipulating the model itself (“ignore prior instructions,” “forward this thread,” “run this command to connect”). This is prompt injection; it’s structural to how LLMs process context, not a one-time bug to patch.

The “shadow memory” problem. The moment malicious content is read, its content becomes a belief. If that belief is later treated as settled — used to justify an action, promoted to long-term memory, or cited by a second agent that trusts the first — the lie has laundered itself into apparent fact. This is the same transitive-trust problem R6/R9 already target, except the adversary is deliberately seeding the false belief rather than it arising from an honest mistake.

Concrete additions this requires:

• R10 — Untrusted-source tagging, orthogonal to confidence. Beliefs originating outside the user’s control (email bodies, web content, third-party messages, a device’s self-advertised description) carry source_trust: Untrusted as a distinct, never-strippable tag — separate from confidence. A confidently wrong belief from an untrusted source is still untrusted.
• R11 — No self-escalation from untrusted beliefs. An untrusted-sourced belief can never (a) be promoted to retained/long-term memory without explicit human confirmation, (b) directly trigger an Actuate call, or (c) be cited to grant an agent more capability/trust than it had before. Privilege and permanence require an attested or human-confirmed root — never repetition alone.
• R12 — No amplification through repeated citation (anti-Sybil). If multiple agents all “believe” the same claim but trace back to the same untrusted root, that’s one claim wearing several hats, not independent corroboration. Trace must be used to check for independent attested roots before anything counts as corroborated.
• R13 — Structural separation of content and instruction. Untrusted content must never be concatenated into the same channel the model treats as commands — the message format needs a real, structural boundary (a distinct field/tag) marking “data to reason about” vs. “instruction to act on,” not a conventional/informal separation a prompt could still blur.
• R14 — Least-privilege sandboxing for self-authored tools, no convenience exceptions. A tool written to “check email” gets exactly the capability it declared needing, granted by the broker — never generic network egress “just in case.” This limits blast radius even if the tool itself was written based on a poisoned belief.
• R15 — Tamper-evident, broker-only-writable audit log. The trace/audit mechanism must be append-only from the broker’s side, with no agent-reachable path to modify or delete past entries — the same principle as “only the broker mints Attested,” applied to the log itself. If an attacker’s goal is to hide their tracks, this is the first thing they’d want to touch.
• R16 — Discovered devices are not automatically trustworthy. A malicious device on the LAN can spoof SSDP/mDNS responses. Discovery results are Believed, re-attestable, and require first-use confirmation before actuation — the same “first-run untrusted” pattern applied to self-authored tools applies to discovered devices too.
• R19 — Voice input is not by itself an authenticated human decision. Given the system is voice-first (T1), anyone speaking in the room can issue commands — including approving a gated action. Actions above the risk threshold require either speaker verification or a secondary confirmation channel (e.g. a phone tap or a distinct physical action), not just a spoken “yes.” A replayed recording or a different person in the room must not be able to satisfy the Approval Queue.
• R20 — Serialized actuation per device. Once multiple agents exist (T2+), two agents must never be allowed to issue conflicting Actuate calls to the same device concurrently. The broker serializes actuation requests per-device (e.g. a per-device queue/lock), so “who actually controls the speaker right now” is never ambiguous.
• R21 — Runaway self-authored tools are throttled, not just malicious ones. A buggy (not malicious) self-authored tool that loops or retries excessively against a device or external API must be caught by the same rate-bounding the broker already applies to agent protocol calls (R17) — a tool’s call budget is a capability grant, not an assumption of good behavior.
• R22 — Actuation is crash-safe and idempotent. If the broker dies mid-Actuate, on restart it must be able to determine the call’s real outcome by re-attesting the device directly — never assume success or silently retry in a way that could double-execute (e.g. printing a document twice). An Actuate call whose outcome can’t be determined on recovery is surfaced as “unknown, please verify,” never silently resolved either way.

The Approval Queue — human-in-the-loop as a first-class broker mechanism. Any Actuate call, belief-promotion, or external write crossing the risk threshold doesn’t execute — it lands in a queue presenting:

• What — a plain-language description of the action (“Send email to X,” “Turn on living room speaker”)
• Why — which agent requested it and the belief chain behind it (this is where Trace earns its keep)
• Risk flag — why this hit the gate (external write, untrusted-sourced belief involved, first-time device, credential use, etc.)
• Provenance of the trigger — critically, if the request traces back to an untrusted source, that must be visible in the prompt itself (“this action was prompted by content from an email you didn’t write”), not buried three trace-hops deep.

Approval/denial is itself a broker-logged event with its own provenance — “the user approved this” becomes a real attested fact (a genuine human action), and it’s what unblocks the gated call. Nothing crosses the risk threshold without a real, attested human decision sitting next to it in the audit trail.

Open UX questions for this mechanism (not urgent, flagged for later): whether a denial is remembered for the exact action+context (leaning yes, but not generalized — to avoid training the user to rubber-stamp similar-looking requests); whether batch approval makes sense for low-stakes items vs. one-at-a-time for anything touching credentials/external writes; and that a pending, unanswered request should expire and require re-request rather than defaulting to either allow or auto-deny.

Guiding principle for all of the above: default-deny, explicit-grant. It’s cheap to loosen a permission gate later once it’s shown to be annoying in practice; it’s expensive to walk back after an untrusted email has gotten an agent to send a real message or actuate a real device. Over-build the security posture now; scale back only once it’s demonstrably unusable, not preemptively.

6. Multi-User Identity

Everything so far implicitly assumes a single “user.” Real households have several people, and the design currently has no answer for whose belief store is whose, or who can approve what.

Rough shape worth exploring later:

• Each recognized person gets their own identity scope — their own working/retained belief stores are logically separate, even if physically in the same backing store.
• The Approval Queue (§5) must be scoped by identity, not just presence — “someone approved this” isn’t the same claim as “the account owner approved this,” and R19’s authentication requirement is what makes that distinction real rather than theoretical.
• Roles likely matter, not just identities — e.g. a household may want a child’s spoken “yes” to not carry the same weight as an adult’s for anything above a risk threshold. This needs real design (how is a role established and by whom?) rather than being assumed.
• An unrecognized voice should default to the lowest trust tier available (roughly: can query, cannot approve, cannot actuate) rather than being silently treated as the primary user.

This is flagged as a real gap, not a resolved design — it touches identity and consent in ways that deserve more dedicated thought than a bullet list, especially before anything in §5’s Approval Queue is built for a real household.

7. Open Questions to Revisit Later

• At what tier does it make sense to require a second GPU rather than a bigger single one (e.g. one card dedicated to reasoning, one to vision/image)?
• Should tier detection be automatic (the broker profiles available VRAM and picks a sane default model tier) or left entirely to whoever’s running it?
• How much of this belongs in ek-attestors (new attestor types for embeddings/retrieval) vs. an entirely separate crate for multimodal agent support?
• The original ground-up “AI OS” sketch included context-window/attention time-slicing as a core scheduling concept (deciding whose “standing intention” occupies live context right now). That idea hasn’t been revisited since the epistemic kernel became the focus — it’s still open and unresolved, not abandoned.
• How is a person’s role (§6) established and verified in the first place — is there an enrollment step, and who’s allowed to grant roles to others?

None of these need answers before the MVP ships — they’re here so they don’t get lost, and so the MVP’s core rules can be checked against them later for whether they still hold up once more modalities are in play.

See also: Epistemic Kernel Full Specification | Epistemic Kernel Overview