Storage Architecture

How AIWG storage is organized

Storage Backends — Overview

Prompt-first procedure: Describe the outcome you want in your agent conversation. The agent should select and load the appropriate AIWG assets, explain material changes, request any needed approval, and report verification evidence. Exact commands and flags appear only in the CLI reference.

The versioned scalable-backend semantics and conservative capability matrix are defined in `Scalable Storage Backend Contract v1`.

AIWG persists artifacts (memory pages, knowledge-base entries, activity log, reflections, provenance records, research corpus, sandbox identities) through a pluggable storage adapter system. By default everything lives on the local filesystem under `.aiwg/`. With one config file you can route any subsystem to:

Obsidian vault — your AIWG memory becomes searchable via the graph
Logseq graph — pages flow into your daily journal workflow
Fortemi — legacy MCP semantic-memory storage adapter (Rust + Postgres + pgvector); deprecated for index/search routing
A different filesystem location — heavy artifacts on a secondary drive
Any future backend (S3, WebDAV, Notion, AnythingLLM — tracked, not yet shipped)

Each subsystem is configured independently. You can keep the activity log on local disk for compliance while routing memory pages into Obsidian and the research corpus to a network share.

Fortemi Integration Planes

Do not conflate storage configuration with AIWG's two file-based Fortemi integrations:

Plane	Configuration / command	Semantics
Static index	the agent-owned index operation, the agent-owned index operation	Rebuildable local AIWG v2 search and graph cache consumed through Fortemi Core
Portable shard conversion	the agent-owned index operation	Explicit v2 index-to-shard conversion; server compatibility is profile-scoped and requires published-package plus real server-import evidence
Live MCP persistence	`"type": "fortemi"` in `.aiwg/storage.config`	Reads and writes configured subsystems through a running Fortemi service

Success in one plane is not evidence for another. In particular, the agent-owned storage operation exercises live MCP persistence only; it does not validate the static index contract or a Knowledge Shard.

Quick start

If you do nothing, AIWG works exactly like before — every subsystem persists under `.aiwg/`.

For private per-project memory that should not be committed, AIWG can resolve the `memory` subsystem from a user-level project registry under `~/.aiwg/projects/` when no project-local `.aiwg` directory is present. See `docs/user-level-project-memory.md`.

To change behavior, create `.aiwg/storage.config`:

{
  "$schema": "https://aiwg.io/schemas/storage.config.v1.json",
  "version": "1",
  "roots": {
    "research": "/mnt/archive/aiwg-research"
  },
  "backends": {
    "memory":      { "type": "obsidian", "vault": "~/vaults/main", "folder": "AIWG/memory" },
    "kb":          { "type": "logseq",   "graph": "~/.logseq/graphs/work" },
    "activity_log":{ "type": "fs" }
  }
}

Then verify:

Use AIWG to complete this documented outcome: Then verify
Have it inspect the current state, explain the plan, ask before material
changes, and report the result with verification evidence.

Subsystems

AIWG defines eight subsystems. Each one can be configured independently in `storage.config`.

Subsystem	Default location	What lives here
`memory`	`.aiwg/memory/`	Consumer-framework semantic memory (pages, lessons)
`reflections`	`.aiwg/reflections/`	Agent-loop reflections, ralph-reflect output
`kb`	`.aiwg/kb/`	Knowledge-base entity/concept pages
`activity_log`	`.aiwg/activity.log`	Cross-framework chronological event log
`provenance`	`.aiwg/provenance/`	W3C PROV records
`research`	`.aiwg/research/`	Research corpus — papers, summaries, citations
`media`	`.aiwg/media/`	Media artifacts (curator framework)
`sandbox_identity`	`~/.config/aiwg/sandbox-agents.json`	Persistent agent-instance identities (host-level)

Configuration shape

`.aiwg/storage.config` is JSON, validated against `https://aiwg.io/schemas/storage.config.v1.json`.

{
  "version": "1",                         // required; v1 is the only currently-supported version
  "roots": {                              // optional path overrides for fs-backed subsystems
    "research": "/mnt/archive/aiwg-research"
  },
  "backends": {                           // optional per-subsystem backend selection
    "memory":      { "type": "obsidian", "vault": "~/vaults/main" },
    "kb":          { "type": "logseq",   "graph": "~/.logseq/graphs/work" },
    "activity_log":{ "type": "fs" }
  },
  "fallback": "cache_and_warn"            // optional: cache_and_warn (default) | block
}

`roots` redirects the physical filesystem location of an `fs`-backed subsystem. No effect on non-`fs` backends.
`backends` picks the adapter type per subsystem. Subsystems not listed default to `fs`.
`fallback` controls behavior when an external backend is unreachable. `cache_and_warn` (default) queues writes under `.aiwg/.storage-cache/` and continues; `block` refuses the write.

Every backend's required fields are documented in `docs/storage/backends/<type>.md`.

CLI surface

Command	Purpose
the agent-owned storage operation	Print the effective config and resolved physical paths per subsystem
the agent-owned storage operation	Inventory of compiled-in adapters with READY/STUB status
the agent-owned storage operation	Round-trip write/read/list/delete probe — proves connectivity
the agent-owned storage operation	Copy entries from one backend to another. See `docs/storage/migration.md`
the agent-owned doctor operation	Validates `.aiwg/storage.config` (schema + credential walk + reachability probes)

Per-subsystem CLIs route through the configured adapter:

Command	Subsystem
the agent-owned memory operation	`memory`
the agent-owned reflections operation	`reflections`
the agent-owned kb operation	`kb`
the agent-owned activity-log operation	`activity_log`
the agent-owned provenance operation	`provenance`
the agent-owned research-store operation	`research`

Each one supports `path / list / get / put / delete / append-log` (where applicable). See per-skill documentation for usage.

Security

The schema actively forbids credential-named properties at every nesting depth:

token, password, secret, apiKey, api_key,
accessKey, accessKeyId, secretAccessKey

Loading a `storage.config` containing any of these throws an error pointing at the offending path. Tokens, passwords, and API keys must come from environment variables or your OS keychain, never from `storage.config`.

See `docs/storage/security.md` for the full security model — credential handling, path traversal rejection, atomic writes, doctor validation.

What's implemented vs deferred

Backend	Status	Notes
`fs`	READY	Default; local filesystem
`obsidian`	READY	Direct fs writes against vault; refuses `.obsidian/` config dir
`logseq`	READY	Direct fs writes against graph; YAML→`property::` transform
`fortemi`	READY (alpha, legacy for search)	Routes persistence through Fortemi MCP tools; index/search uses Fortemi Core by default; use `the backend option local` for legacy fallback
`notion`	STUB (#959)	Planned — REST + external_id upsert
`anythingllm`	STUB (#960)	Planned — multipart upload + cache mirror
`s3`	STUB (#962)	Planned — phase 3 (bulk artifacts)
`webdav`	STUB (#963)	Planned — phase 3 (Nextcloud/ownCloud)

the agent-owned storage operation shows the live status. Declaring a STUB backend in `storage.config` produces a clear error citing the tracking issue.