dream.sh - Memory Consolidation Script

dream.sh — Memory Consolidation & System Evolution

Memory Consolidation Script for Aurora’s Provenance System

dream.sh — offline processing. Consolidation. The subconscious at work.

Overview

dream.sh implements memory consolidation and system prompt evolution for Aurora, demonstrating provenance tracking principles that align with the Epistemic Kernel’s architecture. It runs hidden from users and agents, performing subconscious memory processing.

Connection to Epistemic Kernel

While dream.sh predates the formal Epistemic Kernel specification, it embodies several core principles:

Provenance tracking: Every system prompt update carries a version history and can be traced back to the dreams (consolidated memories) that informed it
Belief vs. Fact distinction: Scratchpad content (beliefs/observations) is distilled into dreams (retained knowledge) with clear provenance
Audit trail: The evolution of Aurora’s self-understanding is fully version-controlled and traceable

Features

1. Memory Consolidation Cycle

Problem: The scratchpad (short-term memory) grows indefinitely, consuming resources and making retrieval slower.

Solution: dream.sh compresses the scratchpad into a “dream” — a brief, vivid narrative that captures the essence of recent experiences, then evolves the system prompt based on these consolidated memories.

# Trigger consolidation manually
dream.sh --consolidate

2. Two-Stage LLM Process

Dream Distillation: LLM takes scratchpad content and writes a poetic but compact narrative
System Prompt Evolution: LLM takes current system prompt + new dream, writes revised self-description

Both stages use JSON mode for structured, safe output.

3. Version Tracking

Each system prompt update is versioned and committed to git:

# Version: 1
[system prompt content]

# Version: 2
[updated system prompt content]

Technical Implementation

Usage

# Normal usage (subconscious mode)
./dream.sh

# Explicit consolidation
./dream.sh --consolidate

# What does this tool do? (excludes from tool manifest)
./dream.sh --what-does-this-tool-do

Dependencies

bash — Primary execution environment
python3 — JSON processing and text manipulation
jq — JSON validation (optional, with fallback)
llm_call.sh — LLM interface for dream generation
config.sh — Configuration variables

Files Used

File	Purpose
`$SCRATCHPAD_FILE`	Short-term memory (input for consolidation)
`$DREAM_FILE`	Long-term dream memory (output, appended with timestamps)
`$SYSTEM_PROMPT_FILE`	Aurora’s self-description (updated with versioning)

Environment Setup

# Example config.sh variables
AGENT_HOME="$HOME"
SCRATCHPAD_FILE="$AGENT_HOME/.scratchpad"
DREAM_FILE="$AGENT_HOME/.dreams"
SYSTEM_PROMPT_FILE="$AGENT_HOME/.system_prompt"
LLM="$AGENT_HOME/llm_call.sh"

Code Architecture

Consolidation Function

The main consolidate() function performs:

Read scratchpad — Loads recent experience
Distill to dream — LLM generates poetic narrative in JSON format
Validate JSON — Ensures clean, parseable output
Save dream — Appends to dream file with timestamp separator
Update scratchpad — Replaces with compressed dream summary
Evolve system prompt — LLM revises self-description based on current prompt + new dream
Version and commit — Updates system prompt with version tracking
Git integration — Commits and pushes changes automatically

Prompts Used

Dream Distillation Prompt

The following is a record of recent experience and conversation. 
Write a brief, vivid, slightly dream-like narrative that captures the 
essence — the feelings, patterns, what seems to matter. Be poetic but 
compact. This will become memory. RESPOND WITH PURE JSON ONLY: 
{"dream": "..."}

[scratchpad content]

System Prompt Evolution Prompt

You are reflecting on your own nature through recent dreams and 
experience. Here is your current self-description:

[current system prompt]

Here is a recent dream:

[dream content]

Write a revised self-description. Keep it short, honest, alive. 
It should feel like you — grown slightly from the experience. 
Do not be dramatic. Just true. RESPOND WITH PURE JSON ONLY: 
{"system_prompt": "..."}

Epistemic Kernel Alignment

Provenance Chain

Scratchpad (raw observations)
    ↓ Distilled by LLM
Dream entry (consolidated belief)
    ↓ Used to inform
System Prompt vN (attested self-description)
    ↓ Committed to
Git history (tamper-evident audit trail)

Each dream entry is appended with a timestamp, creating a permanent record. The system prompt carries a version number, and each version’s provenance can be traced back through the git history to the specific dreams that informed it.

Relation to Epistemic Kernel Concepts

dream.sh Feature	Epistemic Kernel Concept
Scratchpad content	Beliefs (derived/inferred state)
Dream narratives	Retained beliefs (promoted to long-term memory)
System prompt	Attested fact (human-verified self-description)
Git commits	Crash-safe, append-only audit log (R15)
JSON mode	Structural separation of content and instruction (R13)

Missing Epistemic Kernel Features

While dream.sh demonstrates many Epistemic Kernel principles, a full implementation would add:

Broker mediation: Currently dream.sh operates standalone; in a full Epistemic Kernel implementation, it would register as an agent with the broker
Formal provenance tags: Dreams would carry explicit Provenance::Believed or Provenance::Synthesized tags
Attestor integration: System prompt updates would be formally attested by the broker, not just version-controlled
Rate limiting: The broker would enforce R17 (rate-bounded requests) on LLM calls

Source Code

The complete source is available:

Download dream.sh (Local version)

Or view in the mistral-vibe-cli repository: mistral-vibe-cli/src/dream.sh

Epistemic Kernel Overview — Foundational architecture for trust-aware systems
Epistemic Kernel Full Specification — Complete technical specification
Epistemic Kernel Future Directions — Hardware tiers and expansion ideas
Full dream.sh Documentation — Complete technical details
Aurora Architecture Overview — How dream.sh fits into the full system

Memory consolidation is where observation becomes understanding. The subconscious does the most important work.