RuVector validation and benchmark evidence¶

Measured on 2026-10-11 using the LionAGI source implementation on Linux x86_64, Python 3.10.22, RuVector 0.1.2 and NumPy 2.2.6. No embedding provider, remote service or LLM was called.

Acceptance¶

44 integration and optional-import tests plus 29 existing MemoryStore/Branch memory regression tests passed: 73 passed. The real Rust extension and real Branch.act implementation are exercised. Ruff check and formatting pass for all added Python files. The lockfile passes uv lock --check. An isolated source distribution and wheel build succeeded.

The full repository suite was attempted with the RuVector extra and development dependencies. Collection stopped with five missing-dependency errors in unrelated Studio/MCP tests (fastapi and jsonschema), with 81 skips. Therefore this is not a claim that the full LionAGI suite or platform CI passed. Maintainer CI uses all extras and remains the merge gate. The change does not alter existing CI or release permissions.

Tests cover identity, payload roundtrip, tag/filter semantics, real tools, concurrency, cancellation during embedding, missing and malformed vectors, bounded finite JSON, resource limits, detached snapshots, native insertion poisoning and deletion rollback. No fuzzing campaign, penetration test or new independent native-code audit is claimed.

Synthetic retrieval benchmark¶

10,000 normalized Gaussian vectors, 64 dimensions, 200 independent queries, k=10, dataset seed 20261011. Ten queries warm the index. Exact NumPy cosine ranking provides ground truth. BLAS runs one thread. Each configuration runs once in its own process; rerun for variance. Dataset hashes and environment details are in benchmarks/ruvector/gaussian-10k-ef*.json.

Exact NumPy was faster on this small dataset. No speed advantage over NumPy is claimed. Higher ef_search trades additional search work for recall. The adapter honors the configured ef_search directly and uses bounded top-k selection for exact filtered queries. HNSW graph randomness is not fully controlled by the dataset seed. This is an exploratory benchmark, not a statistically established optimization claim.

Measurements include Python adapter overhead and exclude embedding, network and LLM costs. They do not cover production relevance, concurrent throughput, filtered scans, deletion latency, or other operating systems. High-water RSS includes Python and native allocations and is not exclusive index memory.

uv sync --extra ruvector
uv run pytest -n0 tests/integrations tests/protocols/test_memory.py tests/session/test_memory_access_surface.py
uv run python examples/ruvector_memory.py
uv run python benchmarks/ruvector/retrieval.py --ef 100 --output /tmp/ruvector-benchmark.json

The benchmark uses the Unix resource module and runs on Linux/macOS. The integration itself does not depend on that module.

Exact filtered path optimization¶

The native SDK filter applies after approximate retrieval. Using it here would change the existing exact pre-ranking filtering contract, so this change retains the exact path. The hot loop now clamps scalar cosine distances using Python min/max instead of dispatching NumPy clip for each eligible record. It preserves the original subtraction precision, output bounds, deterministic tie order and metadata isolation.

Paired comparison against commit cd6e3686c1586a271864c323d4998a1e3c9369df: same store/index, 5,000 vectors, 64 dimensions, 40 queries, three rounds per configuration. Execution order alternates baseline/optimized. Two independent dataset seeds and two filter selectivities produce 480 timed pairs. Every pair has identical complete results, including IDs, order, distances and metadata. A new regression test pins cosine endpoints and tie ordering. Pyright reports zero errors/warnings; Ruff passes.

Observed median reduction is 34% to 59%; every configuration exceeds the 10% acceptance target. Raw p95 measurements and environment details are in benchmarks/ruvector/filtered-ab.json. These are synthetic filtered retrieval measurements, not an unfiltered HNSW or native SDK speedup. Host predicate cost and metadata size can change the benefit. No approximate-search recall tradeoff or security check removal is involved.

Reproduce the acceptance check from the repository root:

uv run python benchmarks/ruvector/filtered_ab.py --output /tmp/filtered-ab.json --min-reduction 0.10

The command rejects any result mismatch and exits unsuccessfully if any configuration misses the requested median reduction. Timing gates belong in a controlled benchmark environment; this command is not an unconditional wall-clock gate in general CI.