ADR-0093: External memory adapter fidelity contract¶

• Status: Proposed
• Kind: Aspirational
• Area: substrates
• Date: 2026-07-09
• Relations: supersedes v0-0091; extends ADR-0092

Context¶

ADR-0092 intentionally keeps MemoryStore structural and small. That makes adapters easy to inject, but identical method names do not prove identical storage or search behavior.

P1 — Structural compatibility is not semantic compatibility. A runtime-checkable Protocol can see store, retrieve, and search; it cannot see whether metadata is dropped, tags survive search, writes are immediately searchable, results are ranked, or a service error was silently converted into an empty result.

P2 — External services have asymmetric native shapes. A service may accept tags on write but not return them in ranked hits, expose direct identity retrieval separately from recall, normalize timestamps, reject arbitrary metadata, or add native fields that do not fit MemoryItem. An adapter must declare that projection rather than imply a lossless round trip.

P3 — Query portability is incomplete. MemoryQuery.filters is an open mapping. External services frequently support a closed filter set, different comparison operators, or no unfiltered scan. Unsupported keys must fail visibly; ignoring them would broaden a query and return records the caller intended to exclude.

P4 — Search and identity consistency can differ. A successful store should support direct identity retrieval, while a secondary search index may be eventually consistent. Treating an immediate search miss as a failed write, or treating ranked recall as identity retrieval, produces incorrect recovery behavior.

P5 — Explicit persistence cannot use best-effort prompt-injection failure semantics. lionagi/tools/khive_injection.py is a ContextProvider: it recalls and optionally composes text for a prompt, optionally writes selected events, and deliberately degrades to no injection on transport failure. It does not implement MemoryStore, return MemoryItem, or participate in Branch/Session store ownership. An explicit store() call must not appear successful when its transport failed.

P6 — Core must remain dependency-light. An external adapter can live in a connector or service-owned package and be passed to Branch(memory=...) or Session(memory=...). Lionagi core should not acquire service-specific clients, connection pools, retry policies, or field mappings to make that possible.

The target boundary is:

flowchart LR
    B["Branch / Session"] --> MS["MemoryStore"]
    MS --> A["ExternalMemoryStore adapter"]
    A --> S["external service"]
    A --> M["MemoryAdapterManifest"]
    T["conformance suite"] --> MS
    T --> A
    CP["ContextProvider"] --> S
    CP -. "not a MemoryStore" .-> B

This ADR deliberately does not decide:

• Graph traversal, task lifecycle, messaging, or knowledge composition. Those operations do not fit the three-method memory contract.
• A service transport. MCP, HTTP, an SDK, or an embedded client may sit behind an adapter if the manifest and tests remain true.
• A core connection manager or close hook. Branch and Session continue accepting the smaller MemoryStore Protocol.
• One portable search score or ordering. External search may be ranked; callers decide whether a declared adapter meets their needs.
• The package, release, or installation mechanism for a concrete adapter. The interface is independent of distribution.
• Automatic prompt injection. ContextProvider owns that optional pre-turn concern.

Decision¶

D1 — Require an immutable fidelity manifest¶

Every external adapter implements MemoryStore and a companion protocol with one immutable manifest property. The manifest is not added to the base Protocol, so existing core and minimal in-process stores remain valid.

The target contract:

from collections.abc import Iterable
from typing import Literal, Protocol, runtime_checkable

from pydantic import BaseModel, ConfigDict, Field, model_validator

MemoryItemField = Literal["id", "created_at", "metadata", "content", "tags"]
SearchConsistency = Literal["immediate", "eventual"]
Fidelity = Literal["full", "native_fields", "lossy"]

class MemoryAdapterManifest(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    adapter: str = Field(min_length=1)
    native_item_fields: frozenset[MemoryItemField]
    dropped_item_fields: frozenset[MemoryItemField]
    supported_filters: frozenset[str]
    search_consistency: SearchConsistency
    retrieve_fidelity: Fidelity
    search_fidelity: Fidelity
    search_returns_tags: bool
    preserves_metadata: bool

    @model_validator(mode="after")
    def coherent_projection(self) -> "MemoryAdapterManifest":
        fields = {"id", "created_at", "metadata", "content", "tags"}
        if self.native_item_fields & self.dropped_item_fields:
            raise ValueError("native and dropped item fields must be disjoint")
        if self.native_item_fields | self.dropped_item_fields != fields:
            raise ValueError("every MemoryItem field must be classified")
        if not {"id", "content", "tags"} <= self.native_item_fields:
            raise ValueError("id, content, and tags are required native fields")
        if self.preserves_metadata and "metadata" not in self.native_item_fields:
            raise ValueError("preserved metadata must be a native field")
        if self.retrieve_fidelity == "full" and (
            self.dropped_item_fields or not self.preserves_metadata
        ):
            raise ValueError("full retrieve fidelity must preserve every item field")
        if self.search_fidelity == "full" and (
            self.dropped_item_fields
            or not self.search_returns_tags
            or not self.preserves_metadata
        ):
            raise ValueError("full search fidelity must preserve all fields and tags")
        return self

@runtime_checkable
class ExternalMemoryStore(MemoryStore, Protocol):
    @property
    def manifest(self) -> MemoryAdapterManifest: ...

Field meanings are normative:

The fidelity values mean:

• full: every MemoryItem field is represented and round-trips without adapter-defined loss.
• native_fields: direct retrieval round-trips every field in native_item_fields; search round-trips the operation's documented native projection, with tag presence governed by search_returns_tags. Every globally dropped field is absent/defaulted as documented.
• lossy: at least one accepted native field is normalized, truncated, synthesized, or otherwise not exactly reconstructible. The transformation must have named test fixtures and documentation; the enum alone is not sufficient disclosure.

Exact manifest semantics:

• The manifest is stable for the lifetime of an adapter instance. A runtime service downgrade that invalidates it makes the adapter unavailable; the adapter does not mutate capabilities mid-call.
• The manifest describes adapter behavior, not service marketing or optional features that were not configured for this instance.
• Every MemoryItem field is classified exactly once. An adapter cannot omit an awkward field from both sets.
• Additional service-native fields are projected only into documented MemoryItem fields or omitted. They do not appear as undeclared dynamic attributes.
• Applications may reject an adapter before first use when its manifest does not meet required fidelity, filter, consistency, tag, or metadata properties.

Why this way: a typed manifest makes semantic differences machine-checkable without expanding the base core Protocol. Immutable instance-level truth prevents a caller from adopting an adapter under one contract and observing another later.

D2 — Make conformance executable¶

An external package runs the same semantic matrix against two fixtures:

1. a deterministic fake transport that can force misses, unsupported queries, eventual-index delay, malformed responses, and transport failure; and
2. an opt-in live service environment that checks the adapter's published manifest against the deployed service version.

The reusable suite consumes this harness shape:

class MemoryAdapterHarness(Protocol):
    async def make_store(self) -> ExternalMemoryStore: ...
    async def reset(self) -> None: ...
    async def make_search_index_visible(self) -> None: ...
    async def make_transport_unavailable(self) -> None: ...
    async def close(self) -> None: ...

The suite does not require the adapter itself to expose reset, index, or failure controls; those are test-harness capabilities over a disposable namespace or fake transport.

Required conformance matrix:

The external package may add service-specific tests, but it cannot weaken or skip a shared case without changing its claim that the adapter satisfies this ADR. Live tests may be opt-in; fake tests are deterministic and required in ordinary CI.

Why this way: prose capability declarations drift. A fake makes rare failure and eventual consistency paths repeatable, while a live check catches wire/schema drift the fake cannot. The separation avoids making routine core tests depend on external availability.

D3 — Make projection and query semantics explicit¶

An adapter may be lossy, but never silently so.

Write contract:

• store(item) validates field support before issuing a remote write.
• A field the adapter cannot represent is either rejected or included in manifest.dropped_item_fields. Documentation states the default value observed on retrieval and search.
• id, content, and tags are the minimum direct-retrieval anchors already exercised by ADR-0092's protocol fence. If the service cannot preserve them across store then retrieve, the adapter rejects the write; it must not return a different id or silently lose tags while claiming core conformance. Search hits may still omit tags when the manifest declares search_returns_tags=False.
• A rejected remote write raises MemoryAdapterError without returning a UUID. A successful transport response that lacks a usable identity is also a write failure.
• Retrying a write is adapter-owned. If the service cannot make repeated writes by the same UUID idempotent, the adapter documents duplicate/conflict behavior; the portable layer does not auto-retry.

Retrieve contract:

• retrieve(item_id) uses a direct identity operation, not ranked recall with the UUID as query text.
• Unknown identity returns None.
• A service object with that identity but the wrong native record kind returns None; the adapter does not coerce unrelated records into MemoryItem.
• A successful store() must be directly retrievable according to the declared retrieve fidelity without waiting for the search index. Otherwise the write is not acknowledged as successful.
• Projection yields a new MemoryItem and no transport-native response object.

Search contract:

• MemoryQuery.text, tags, and positive limit are portable input axes. An adapter documents its backend-specific ordering and tag-match semantics; ADR-0092 does not claim those are equal to InMemoryStore.
• MemoryQuery.filters accepts only keys in manifest.supported_filters. Validation considers the full key set before transport so partial filtering cannot occur.
• Unsupported keys raise UnsupportedMemoryQuery with all rejected keys sorted. They are never ignored, stripped, or moved into free-form query text.
• limit=0 returns [] without transport. limit<0 raises UnsupportedMemoryQuery naming limit; external adapters do not reproduce the reference backend's negative-slice quirk.
• Empty text/tags/filters is a valid request for the backend's documented first page. If a service cannot express that operation, its adapter cannot claim conformance until it implements a safe list path; substituting a wildcard with different scope is not allowed.
• search_consistency="eventual" permits a newly stored item to be absent before index convergence. It does not permit a stale direct retrieve or an empty result on transport error.
• Search scores, rank breakdowns, and service-native query options do not enter MemoryItem or MemoryQuery implicitly. They require a separately versioned extension.

Stable errors:

class MemoryAdapterError(RuntimeError):
    """Base class for an explicit external MemoryStore operation failure."""

class UnsupportedMemoryQuery(MemoryAdapterError):
    def __init__(self, rejected_keys: Iterable[str]) -> None:
        self.rejected_keys = tuple(sorted(set(rejected_keys)))
        super().__init__(
            "unsupported memory query keys: " + ", ".join(self.rejected_keys)
        )

class MemoryAdapterUnavailable(MemoryAdapterError):
    """Transport or service availability prevented an explicit operation."""

Native exceptions are chained as causes but do not escape as the public error type. Validation errors in constructing MemoryItem, MemoryQuery, or the manifest remain Pydantic errors; these adapter exceptions describe operation semantics after valid input reaches the adapter boundary.

Why this way: field loss and ignored predicates are correctness failures, not mere feature differences. Direct retrieval and eventual search answer different questions and need different miss semantics. Stable exceptions let applications distinguish “not found,” “unsupported,” and “temporarily unavailable” without knowing the service SDK.

D4 — Keep operational lifecycle adapter-owned and documented¶

The base MemoryStore does not gain open, close, health, retry, or transaction methods. An external adapter owns any client, pool, authentication, namespace, timeout, and retry policy.

Every adapter's companion documentation must state:

• when connections are created (constructor, first call, or injected client);
• whether one adapter instance is safe for concurrent Branch calls;
• how the owning application closes it and whether close is idempotent;
• request/connect timeout values and why those values were chosen;
• retry count, backoff, jitter, and which operations are safe to retry;
• namespace/collection scope and whether it is fixed at construction;
• write-conflict and duplicate-id behavior; and
• what availability/health evidence invalidates the manifest.

There are no portable timeout, retry, pool-size, or connection-budget numbers in this ADR. The adapter package must provide real values with service evidence or explicitly state that it inherits client defaults with no recorded rationale. Core must not silently supply a universal retry loop: a retried write may duplicate state while a retried search is normally read-only.

Exact failure semantics:

• Transport creation failure raises MemoryAdapterUnavailable from the first explicit store, retrieve, or search that requires it.
• Mid-call transport failure raises MemoryAdapterUnavailable; it does not return None, [], or a generated UUID.
• Malformed service payload raises MemoryAdapterError with the native error chained.
• Close failure follows the adapter's documented owner lifecycle; it is never swallowed by Branch or Session because they do not own close.
• An adapter may expose an additional async context manager or close() method, but callers using the base Protocol must still arrange ownership externally.
• There is no fallback to InMemoryStore after an external write or read failure. Such fallback would split state and make later retrieval nondeterministic.

Why this way: lifecycle policy differs by transport and deployment. Adding it to the three method core Protocol would force the zero-dependency store to mimic network concerns and would not solve application ownership. Stable operation errors are portable; connection mechanics are not.

D5 — Keep the khive store target separate from prompt injection¶

A khive-backed adapter is the first named application of D1-D4, but this ADR does not claim that it exists in this repository. It remains in an external package, satisfies ExternalMemoryStore, publishes a source-verified manifest, and maps store, identity retrieval, and ranked recall to their distinct native operations. The exact verb ids, request fields, response fields, defaults, and loss profile are DEFERRED until implementation time, when they can be verified against the external adapter and service source. Historical mappings are not treated as current wire truth.

KhiveInjectionProvider is an adjacent, implemented ContextProvider. Its relevant current contract (lionagi/tools/khive_injection.py) is:

class KhiveInjectionProvider:
    def __init__(
        self,
        policy: KhiveInjectionPolicy,
        mcp_config: dict | None = None,
    ): ...

    async def provide(
        self, branch: Branch, instruction: Instruction
    ) -> str | None: ...

    async def writeback(
        self, branch: Branch, action_responses: list
    ) -> None: ...

Its policy defaults are also concrete:

@dataclass(frozen=True)
class RecallPolicy:
    limit: int = 5
    min_score: float = 0.4
    max_tokens: int = 800

@dataclass(frozen=True)
class ComposePolicy:
    enabled: bool = False
    max_tokens: int = 2000

@dataclass(frozen=True)
class WritebackPolicy:
    enabled: bool = False
    salience_cap: float = 0.4
    tags: tuple[str, ...] = ()

The query task text is truncated to 400 characters. Injection output is token-truncated to the recall cap plus the compose cap when enabled. These values are inherited implementation defaults; the module records no measured rationale for 5, 0.4, 800, 2,000, or 400. They are not memory-store limits and must not be copied into an adapter manifest.

Exact separation semantics:

• provide() returns rendered prompt text or None, never MemoryItem.
• It calls recall and optional composition, and it swallows transport failure into no injection so the conversational turn continues.
• Optional writeback derives tool error/resolution pairs and issues memory writes. A writeback failure is logged and stops the remaining writeback loop; it is not surfaced as an explicit persistence result.
• It can emit best-effort feedback after recall. Feedback failure is logged and does not fail the injection.
• It has cadence, token, and salience policy; it has no MemoryAdapterManifest and does not participate in Session's first-claim store sharing.
• A caller wanting explicit persistence injects the external adapter. A caller wanting optional prompt enrichment registers the context provider. Using one does not prove the other exists.

Why this way: prompt enrichment should degrade gracefully because it is optional context. Explicit persistence must surface failure because the caller relies on state. Sharing a transport or native service does not make those contracts interchangeable.

The explicit operation sequence is:

sequenceDiagram
    participant A as Application
    participant M as ExternalMemoryStore
    participant X as Manifest / validator
    participant S as External service

    A->>X: inspect fidelity and filter support
    X-->>A: accept or reject adapter before use
    A->>M: store(MemoryItem)
    M->>M: validate field projection
    M->>S: native write
    S-->>M: native identity response
    M-->>A: same UUID or typed error
    A->>M: retrieve(UUID) or search(MemoryQuery)
    M->>M: validate query/filter/limit
    M->>S: identity read or ranked/list search
    S-->>M: native payload
    M-->>A: MemoryItem / list[MemoryItem] / None / typed error

Consequences¶

Applications can compare adapters by executable guarantees instead of assuming identical method names imply identical memory semantics. Core remains dependency-light, while lossy projection, eventual consistency, filter support, and service availability become visible adoption choices.

Adapter authors must maintain an immutable manifest, deterministic fake, live compatibility check, field projection documentation, and error mapping as either side evolves. Search hits may still differ in ordering or rank even when two adapters both conform; the manifest is a capability contract, not a claim that all backends are interchangeable.

Callers requiring full metadata, tags in search hits, immediate search visibility, or a specific filter set must reject a manifest that does not provide them. They must also own connection lifecycle because Branch and Session remain intentionally unaware of it.

Reversing D1-D3 after adapters ship requires a manifest/schema version and conformance migration. Reversing D4 by adding lifecycle to core would change MemoryStore and every implementation. Reversing D5 would couple optional prompt enrichment to explicit persistence and create ambiguous failure semantics.

Counting the core protocol, conformance suite, external adapter, external service, and context provider gives five components. The adapter depends on the protocol and service, the suite depends on the protocol and adapter, and the context provider depends directly on the service, giving five directed dependencies and κ = 5 / (5 × 4) = 0.25. Its testability target is τ = 0.90: protocol, projection, filters, consistency, and errors are testable through a fake transport, with only live-service compatibility requiring an opt-in integration environment.

Alternatives considered¶

Put service-specific adapters in lionagi core¶

This would make installation and examples easy and could share Lionagi's connection utilities. It lost because every service client, wire change, credential mode, and retry policy would enter core's dependency and release surface. Constructor injection already provides the needed seam.

Add the manifest to base MemoryStore¶

One universal protocol would let every caller inspect capabilities, including InMemoryStore. It lost because the existing core contract deliberately has three methods and structurally valid minimal stores need no service-fidelity declaration. ExternalMemoryStore extends rather than breaks that contract.

Treat direct context injection as a store adapter¶

Both surfaces may call the same service, and prompt recall resembles search. It lost because the context provider returns rendered text, performs optional feedback/writeback, follows cadence and token budgets, and swallows transport errors. A store returns typed records and must surface explicit persistence failure.

Permit undocumented lossy projection¶

This minimizes adapter code and lets callers inspect returned objects empirically. It lost because missing metadata or tags can alter provenance and filtering silently. The manifest and fixtures make the adoption tradeoff explicit before data is written.

Ignore unsupported filters and run the broader query¶

Best-effort filter mapping would keep more services usable. It lost because dropping a predicate broadens result scope and can return records the caller explicitly excluded. Visible rejection is the only safe portable behavior.

Put arbitrary native query options in MemoryQuery.filters¶

Pass-through options would expose every service feature without new types. It lost because filter keys would become an unversioned backend escape hatch and callers could no longer know which queries are portable. Native extensions require their own versioned API.

Rehydrate every search hit through identity retrieval¶

This could upgrade low-fidelity search results, such as missing tags, into full MemoryItems. It lost as a hidden default because one ranked call becomes N+1 service calls, changing latency, failure, and rate behavior. A caller may explicitly retrieve selected hits when needed.

Map identity retrieval onto ranked search¶

Using one native recall operation would reduce adapter mapping work. It lost because rank search can miss a valid id, return a different record, or depend on index convergence. Identity miss and search miss have different contracts.

Fall back to InMemoryStore on external failure¶

This would keep the application available during an outage. It lost because successful-looking writes would split across two stores and later retrieval would depend on which process handled the call. Availability cannot override persistence truth silently.

Notes¶

Interpretation uses expressio unius to limit this contract to the three MemoryStore verbs and their fidelity. Graph traversal, task operations, prompt enrichment, and service-native features remain outside the seam unless separately specified.

Primary current-code anchors: lionagi/protocols/memory.py; tests/protocols/test_memory.py; lionagi/session/branch.py; lionagi/session/session.py; lionagi/tools/khive_injection.py; lionagi/protocols/context_providers.py.