ADR-0051: LNDL language and operations boundary¶

• Status: Accepted
• Kind: Retrospective
• Area: utilities
• Date: 2026-07-09
• Relations: none

Context¶

LNDL is a structured-output language intended to let a model mix natural reasoning, typed value declarations, action-call declarations, and an explicit OUT{} projection. The shipped code must separate deterministic language processing from stateful model and action execution. Four problems force that boundary.

P1 — Syntax must remain deterministic and independently testable. Tokenization, parsing, AST construction, normalization, extraction, and assembly should run on text and Python values. Requiring a Branch, a model endpoint, or session history to parse one LNDL block would make grammar behavior depend on runtime state.

P2 — Action declarations must be representable before execution. A <lact> body describes a function call, but the language package does not know whether that action is permitted or how it should execute. The assembler therefore needs a typed placeholder that can be collected, replaced, rejected, or revalidated after the operations layer supplies results (lionagi/lndl/types.py, lionagi/lndl/assembler.py).

P3 — Model-produced syntax is imperfect but repair must be bounded and explainable. Models produce curly-brace tags, XML-like attributes, missing >, capitalized note namespaces, scientific notation, duplicate aliases, malformed constructors, and incomplete OUT{} blocks. The language package owns conservative normalization and typed diagnostics; it must not hide operational retries behind the parser (lionagi/lndl/normalize.py, lionagi/lndl/lexer.py, lionagi/lndl/parser.py, lionagi/lndl/errors.py).

P4 — The two production consumers need different behavior. The ordinary parse operation recognizes an explicitly supplied Operable and extracts the first fenced LNDL block before the normal JSON/Pydantic route. The opt-in LNDL Middle runs a multi-round loop, assembles values, sends action calls through normal branch action handling, and validates the final result. They consume the same language package but are not interchangeable (lionagi/operations/parse/parse.py, lionagi/operations/lndl_middle/lndl_middle.py).

lionagi.lndl therefore has active integration paths, but it is not an independent execution runtime.

This ADR deliberately does not decide:

• the multi-round action-selection policy, permission policy, history mutation, suppression behavior, or retry budget; those belong to the operations ADR on LNDL execution semantics;
• a scheduler, action runtime, session store, or independent LNDL service;
• whether LNDL becomes the default response format. Both current integrations are opt-in;
• arbitrary XML or Python compatibility. Normalization and constructor parsing accept only the documented subset;
• an input-size limit for the LNDL lexer/parser. The core currently has no byte/character cap; transport and operation callers must bound untrusted input where required.

Decision¶

D1 — lionagi.lndl owns the language, not execution¶

The package is public and intentionally decomposed by deterministic responsibility.

The contract is the shipped module tree and export surface (lionagi/lndl/__init__.py):

lionagi/lndl/
├── __init__.py              explicit public exports
├── lexer.py                 TokenType, Token, Lexer
├── parser.py                ParseError, Parser
├── ast.py                   Lvar, RLvar, Lact, OutBlock, Program
├── assembler.py             assemble, action/note collection and replacement
├── _parse_function_call.py  restricted Python-style lact constructor parser
├── extract.py               fenced-block extraction
├── normalize.py             conservative surface repair
├── types.py                 metadata, ActionCall, LNDLOutput, revalidation
├── errors.py                LNDLError family
├── diagnostics.py           chunk/round/trace inspection values
├── round_outcome.py         Success/Continue/Retry/Exhausted/Failed values
└── prompt.py                LNDL_SYSTEM_PROMPT and accessor

The public API explicitly exports the AST/value/error types and the language helpers through __all__; it does not export _parse_function_call internals.

flowchart LR
    Text --> Extract[extract / normalize]
    Extract --> Lexer
    Lexer --> Parser
    Parser --> AST[Program AST]
    AST --> Assemble[assembler]
    Assemble --> Value[Python mapping / ActionCall]
    Value -. consumed by .-> ParseOp[operations.parse]
    Value -. consumed by .-> Middle[operations.lndl_middle]
    Middle --> Branch[Branch chat and act]

Exact semantics:

• lionagi.lndl imports Pydantic value types plus deterministic JSON and boolean coercion helpers, but no Branch, Session, provider, action execution, or operation module.
• Parsing and assembly hold no global conversation/session state. A Lexer and Parser are mutable single-use objects and are documented as not thread-safe; callers create one per parse.
• The parser's only process-global mutable value is the set used to emit a reserved action-name warning once per name. It does not affect AST content.
• Prompt, diagnostic, and round-outcome objects are data contracts. Importing or constructing them does not call a model or execute an action.
• The package boundary does not mean zero integration coupling: changes to ActionCall, Program, assemble(), prompts, or error classes can break both operation consumers.

Why this way: deterministic language behavior can be unit tested with plain text and Python types. Runtime state remains in the layer that already owns model calls, action permissions, messages, and failure policy.

D2 — Fenced extraction, normalization, lexing, and parsing define the syntax¶

The core pipeline is explicit; callers choose whether to normalize before lexing.

The primary signatures are:

def extract_lndl_blocks(text: str) -> list[str]: ...
def normalize_lndl_text(text: str) -> str: ...

@dataclass
class Token:
    type: TokenType
    value: str
    line: int
    column: int

class Lexer:
    def __init__(self, text: str): ...
    def tokenize(self) -> list[Token]: ...

class Parser:
    def __init__(self, tokens: list[Token], source_text: str | None = None): ...
    def parse(self) -> Program: ...

The AST contract (lionagi/lndl/ast.py) is:

@dataclass(slots=True)
class Lvar:
    model: str
    field: str
    alias: str
    content: str

@dataclass(slots=True)
class RLvar:
    alias: str
    content: str
    extra_id: str | None = None

@dataclass(slots=True)
class Lact:
    model: str | None
    field: str | None
    alias: str
    call: str
    extra_id: str | None = None

@dataclass(slots=True)
class OutBlock:
    fields: dict[str, list[str] | str | int | float | bool]

@dataclass(slots=True)
class Program:
    lvars: list[Lvar | RLvar]
    lacts: list[Lact]
    out_block: OutBlock | None

Accepted declaration forms are:

OUT{} accepts explicit mappings (field: [alias], string, finite number, boolean, or identifier), bare-alias shortcuts, flat alias lists, and nested list groups. The parser resolves shortcuts using the declared field, then model, then two-token hint; with no context it falls back to the bare identifier.

Exact semantics:

• extract_lndl_blocks() recognizes complete backtick or tilde fences whose language label case-folds to lndl; it returns code bodies in source order. An unfenced input or a differently labelled fence yields an empty list.
• normalize_lndl_text("") returns the empty input. If fenced blocks exist, it joins all of them with two newlines, then repairs only the documented patterns: a missing opening > when a parenthesized body and closing tag exist, curly lvar/lact tags, XML-style opening attributes (promoting name=), and Note. casing in tag bodies. It does not attempt arbitrary markup correction.
• The lexer emits positional tokens and a terminal EOF. It accepts quoted strings with common escapes and numbers with a valid e/E exponent. A trailing exponent marker without digits is not consumed as part of the number.
• Parser.parse() requires source_text; omitting it raises ParseError. Non-LNDL tokens outside declarations are skipped. Parsing stops after the first OUT{}; later declarations are not part of the Program.
• Aliases must be unique across both lvars and lacts. A duplicate raises ParseError. Missing </lvar> or </lact> raises ParseError; content is stripped at both ends.
• An action alias that is a Python reserved word or selected builtin is accepted but warns once per process/name.
• Nested OUT lists are limited to 32 levels. At level 32 the parser raises ParseError; the source records protection from excessive nesting but no rationale for exactly 32.
• Numeric literals containing . or an exponent become float; others become int. Non-finite float results (for example overflow to infinity) raise ParseError.
• Reaching EOF inside OUT{} returns the fields parsed so far; unlike unclosed lvar or lact tags, a missing final } is not currently rejected.
• An input containing no declarations returns Program([], [], None). An input with declarations but no OUT block preserves those declarations and sets out_block=None.

Why this way: the grammar is permissive around surrounding prose and a small set of common surface mistakes, but strict about reference identity, constructor shape, and bounded nesting. This gives operation-level repair loops a useful typed failure instead of embedding retries in the parser.

D3 — Assembly yields mappings and explicit action placeholders¶

Assembly resolves a Program against an optional target class. It does not execute actions and does not itself return a target-model instance.

The value contracts (lionagi/lndl/types.py) are:

@dataclass(slots=True, frozen=True)
class ActionCall:
    name: str
    function: str
    arguments: dict[str, Any]
    raw_call: str

@dataclass(slots=True, frozen=True)
class LNDLOutput:
    fields: dict[str, BaseModel | ActionCall | Scalar]
    lvars: dict[str, str] | dict[str, LvarMetadata]
    lacts: dict[str, LactMetadata]
    actions: dict[str, ActionCall]
    raw_out_block: str

The assembly and post-execution signatures are:

def build_action_call(alias: str, node: Lact) -> ActionCall: ...

def assemble_spec_value(
    refs: list[str],
    target_type: Any,
    lvars_by_alias: dict[str, Lvar | RLvar],
    lacts_by_alias: dict[str, Lact],
    action_results: dict[str, Any] | None = None,
    scratchpad: dict[str, Any] | None = None,
) -> Any: ...

def assemble(
    program: Program,
    target: Any,
    action_results: dict[str, Any] | None = None,
    scratchpad: dict[str, Any] | None = None,
) -> dict[str, Any]: ...

def collect_actions(value: Any) -> list[ActionCall]: ...
def replace_actions(value: Any, results_by_name: dict[str, Any]) -> Any: ...
def has_action_calls(model: BaseModel) -> bool: ...
def ensure_no_action_calls(model: BaseModel) -> BaseModel: ...
def revalidate_with_action_results(
    model: BaseModel,
    action_results: dict[str, Any],
) -> BaseModel: ...

A lact body is parsed as a restricted Python-style function call. The resulting constructor payload is:

{
    "action": str,
    "service": str,          # omitted when unqualified
    "arguments": dict[str, Any],
}

Positional arguments are stored as _pos_0, _pos_1, and so on. Reserved keyword arguments in the supported set are rewritten with a trailing underscore. Literal strings, numbers, booleans, nulls, dictionaries, lists, and tuples are accepted; arbitrary names and **kwargs are rejected. service.action(...) becomes the qualified function name service.action in ActionCall.

Exact semantics:

• assemble() returns {} when program.out_block is absent. It indexes lvars and lacts by alias and makes note declarations from the current round immediately available in a copied scratchpad.
• An lvar reference resolves to stripped text. A lact reference resolves to a supplied historical/current result when present, otherwise to an ActionCall. A historical action result can be referenced by alias in a later round without redeclaring the lact.
• note.X resolves from the scratchpad. A missing scratchpad note contributes no value. Any other undeclared alias raises MissingLvarError.
• If the target exposes model_fields, assembly uses field annotations for scalar, list, dictionary, and nested-model shaping. Required target fields absent from OUT{} raise MissingFieldError. Optional or unknown fields are not synthesized.
• list[Model] groups repeated fields into successive dictionaries; explicit nested OUT groups define item boundaries. Raw values fill unused model fields in declaration order. A dict[K, V] retains only references that carry a declared field name.
• A single string targeting list[scalar] is coerced in this order: JSON list, Python list literal, non-empty newline-separated lines, bracketed comma list, then one-item list. Ordinary comma-containing prose is not split.
• A malformed lact constructor raises InvalidConstructorError. A valid unresolved call remains a value until an operational caller executes it.
• collect_actions() recursively walks dictionaries and lists. replace_actions() recursively replaces calls whose names exist in the result mapping and leaves unmatched calls unchanged.
• has_action_calls() and ensure_no_action_calls() inspect nested models, dictionaries, and sequences. ensure_no_action_calls() raises ValueError, reports up to three field paths plus a remaining count, and otherwise returns the same model.
• revalidate_with_action_results() replaces direct action-valued model fields and nested Pydantic models, coerces scalar results to field annotations, and calls the model class's model_validate(). A missing named result raises ValueError. Boolean strings use LionAGI boolean validation rather than Python truthiness; dictionary results targeting a scalar are deterministic-JSON encoded; None passes through.

Why this way: a placeholder is an honest description of work not yet performed. The language layer can assemble and inspect a result without acquiring action permissions or runtime state, and the operations layer has explicit points to collect, execute, replace, and validate.

D4 — Operations own both consumption seams¶

The language package exposes values; operations decide whether and how to use them.

The narrow parse seam (lionagi/operations/parse/parse.py) is selected only when response_format is an Operable:

def _is_lndl_operable(response_format: Any) -> bool: ...
def _extract_lndl(text: str, operable: Operable) -> Any: ...

Its exact behavior is deliberately small:

• it extracts all fenced LNDL blocks and returns the first block;
• if there is no fenced block, it returns the original text;
• it does not lex, parse, assemble, execute lacts, or materialize/validate the supplied Operable;
• non-Operable response formats continue through ordinary JSON extraction, fuzzy mapping, and Pydantic validation.

The multi-round seam is opt-in through lionagi/operations/lndl_middle/lndl_middle.py:

DEFAULT_ROUND_BUDGET = 3

def build_lndl_middle(round_budget: int = DEFAULT_ROUND_BUDGET): ...

async def generated_middle(
    branch: Branch,
    instruction: JsonValue | Instruction,
    chat_param: ChatParam,
    parse_param: ParseParam | None = None,
    clear_messages: bool = False,
    skip_validation: bool = False,
) -> Any: ...

sequenceDiagram
    participant Op as LNDL Middle
    participant Core as lionagi.lndl
    participant Act as normal Branch action path
    participant Model as target model
    Op->>Core: extract, normalize, lex, parse, assemble
    alt no OUT block
        Core-->>Op: Continue + declared lacts
        Op->>Act: execute through normal action handling
    else parse/assembly error
        Core-->>Op: Retry(error)
    else OUT candidate
        Core-->>Op: Success mapping + reachable ActionCalls
        Op->>Act: execute reachable calls
        Op->>Core: replace actions
        Op->>Model: model_validate(mapping)
    end

The operations seam strips native tool schemas and the native response format from each inner chat call, supplies the LNDL prompt and rendered target spec as guidance, and chooses API communication or CLI run-and-collect through the normal endpoint family decision. Actions are translated into normal action requests and passed to the ordinary action operation, so existing permissions and hooks remain in force.

Current round behavior is recorded here only to make the boundary exact:

• no fenced LNDL block produces Continue() with no actions;
• a parsed program without OUT{} produces Continue() and schedules every declared lact through the operations bridge;
• an LNDL parse/assembly error produces Retry(error=...) and no action dispatch;
• an OUT candidate schedules only ActionCall values reachable from the assembled output, replaces their results, and then validates the caller's target unless validation is skipped or no model-like target exists;
• inner validation failure consumes another round; committed messages/action results remain visible through the operation-owned history;
• action dispatch is concurrent and suppresses individual errors according to the operation's shared ActionParam; the language package does not implement that policy;
• exhausting the default three rounds raises LNDLError rather than returning a raw placeholder, raw error string, or None. The value three is inherited; no rationale for exactly three is recorded in the current source.

These loop and action-selection semantics belong to the operations contract (see the operations ADR on LNDL execution semantics), not to the language package. This ADR fixes which side of the boundary owns them.

Why this way: the parse operation needs a low-cost opt-in extraction path, while the Middle needs history, model calls, action policy, and validation. Keeping both in operations prevents the grammar package from becoming a second Branch runtime.

D5 — Typed language failures and inert outcome values¶

The shipped error family (lionagi/lndl/errors.py, lionagi/lndl/parser.py) is:

LNDLError
├── ParseError                 includes token line and column
├── MissingLvarError           OUT references an undeclared alias
├── MissingFieldError          required target field absent from OUT
├── TypeMismatchError          constructor/spec type mismatch
└── InvalidConstructorError    lact body cannot become a supported call

ensure_no_action_calls() intentionally raises ValueError, because it is guarding a caller-supplied Pydantic value rather than reporting syntax construction.

The round outcome algebra is data only:

@dataclass(slots=True, frozen=True)
class Success: output: Any

@dataclass(slots=True, frozen=True)
class Continue: notes_committed: tuple[str, ...] = ()

@dataclass(slots=True, frozen=True)
class Retry:
    error: str
    note_keys: tuple[str, ...] = ()

@dataclass(slots=True, frozen=True)
class Exhausted: last_error: str | None = None

@dataclass(slots=True, frozen=True)
class Failed: error: BaseException

LndlChunkHealth, LndlRoundRecord, and LndlTrace similarly classify and summarize chunks/rounds without executing anything. The current Middle constructs Success, Continue, and Retry; it raises directly at budget exhaustion rather than returning an Exhausted, and it does not currently construct Failed.

Exact failure semantics:

• syntax position failures include the current token's one-based line and column;
• normalization does not suppress later parser/assembler errors;
• an operational loop may classify an LNDLError as retryable, but the language error itself carries no retry count or side effect;
• constructor parsing wraps SyntaxError/ValueError as InvalidConstructorError at the assembly boundary;
• Pydantic validation failures remain Pydantic ValidationError until the operations seam decides whether to retry or raise.

Why this way: typed errors let deterministic callers distinguish bad syntax, missing references, missing schema fields, and bad constructors. Inert outcomes let an operation implement a state machine without putting that state machine in the parser.

Consequences¶

• Lexer, parser, assembler, normalization, extraction, and diagnostics can be tested without constructing a Branch or invoking a model.
• Operations can change model-call or action policy without placing runtime state in the grammar package, provided the shared AST/value contracts remain compatible.
• Action placeholders make incomplete execution visible. Callers must use ensure_no_action_calls() or explicit replacement/revalidation before treating a model as final.
• The two production paths are intentionally unequal: the parse path extracts one block and stops; the Middle parses, assembles, executes, and validates. Replacing one with the other changes semantics.
• The forgiving grammar has maintenance cost. Changes must test accepted tag forms, shortcut resolution, nested grouping, finite numbers, missing closers, and normalization before changing the repair surface.
• The core currently has no text-size cap and accepts EOF-terminated OUT blocks. Callers needing stricter untrusted-input behavior must enforce it outside the core or change the language contract explicitly.
• Reversing D1 would couple every syntax test to runtime construction. Reversing D3 would require a new representation for unexecuted work. Reversing D4 would move permission/history policy into a second runtime and is therefore high cost.

Current-vs-ideal delta¶

Alternatives considered¶

Embed lexer, parser, and assembler in operations¶

This would co-locate the language with both production consumers and make private runtime helpers easy to call. It lost because deterministic text-to-value behavior would then depend on an operations package that imports model, Branch, message, and action machinery. Grammar tests would no longer define a reusable public language contract.

Give LNDL its own scheduler and action runtime¶

An independent runtime could own rounds, scratchpad state, tool execution, and retries and present one end-to-end LNDL API. It lost because Branch operations already own permissions, hooks, messages, API/CLI model selection, and failure policy. A second runtime would duplicate those invariants and create a bypass risk.

Represent lacts as immediately executed callbacks¶

The assembler could accept an executor and invoke it while resolving OUT{}. This would remove ActionCall replacement steps. It lost because assembly would no longer be deterministic or framework-neutral, and actions not reachable from the final output would be difficult to reason about. Typed placeholders keep execution explicit.

Reject all malformed surface syntax¶

A strict XML-like grammar would be smaller and every deviation would trigger an operation retry. It lost because the shipped model-facing format sees a small, repeatable set of harmless syntax drifts. Conservative repair handles those cases without relaxing duplicate-alias, reference, constructor, or schema checks.

Normalize arbitrary XML/Python-like input¶

Broader repair would accept more model output and perhaps reduce retries. It lost because every new rewrite creates ambiguity about what the model meant and makes failures harder to reproduce. The current normalizer is a closed set of local surface repairs followed by ordinary typed errors.

Make the Operable parse path perform full LNDL execution¶

One parse path could extract, parse, execute actions, and validate the generated model. It lost because parse() is also used as a value conversion operation and should not silently gain action side effects. Full execution remains an explicit Middle choice.

Keep LNDL internal-only¶

An internal package would reduce compatibility obligations. It lost because the AST, value types, prompts, diagnostics, and helpers are deliberately exported and used by both integration paths. Downstream deterministic parsing and inspection are supported uses, not repository-local accidents.

Return None or raw text when the round budget is exhausted¶

This would let an outer caller decide what to do and avoid an exception. It lost at the operations boundary because a caller expecting a validated model could receive a value of the wrong type. The current seam raises a typed LNDL failure instead.

Notes¶

Primary source anchors for this retrospective record are all modules under lionagi/lndl/, plus lionagi/operations/parse/parse.py and lionagi/operations/lndl_middle/lndl_middle.py. The latter two are read here to document the boundary; their round/action policy remains owned by operations.