Benchmarks

AEROS' guarantees aren't asserted.
They're measured.

Three benchmarks pin the production claims of the runtime, measured on the same code that ships in the open-source repo. The packaged benchmark suite is being reworked for public release.

Test baseline: 2,483 passing tests · 0 failing on main · identity-hash regression on every PR.

Benchmark 1

Audit-first vs fail-open recovery

200 trials of a 4-level structured rollback harness, comparing two execution modes: audit-first (every state-mutating action gates on the audit log persisting first) vs fail-open (action proceeds; audit catches up).

The chart is filled with one tile per trial. Green = rollback succeeded; red = rollback failed. 60ms is the median latency the audit-first mode pays per call.

audit-first fail-open 100% 200 / 200 trials 25% 50 / 200 trials audit overhead, p50: 60ms
Takeaway

Audit overhead is small. Audit gap is total. The 60ms p50 cost is the price of going from a 25% rollback floor to a 100% one.

Source: AEROS rollback harness Trials: N=200
Benchmark 2

Byte-identical replay determinism

N=100 replays of the same scenario, with the same governance config and frozen identity manifest. We hash the full run output (per-step decisions, signed audit chain, terminal state) and bucket the hashes.

The histogram below is the actual output: 99 buckets stay empty, one bucket holds all 100 replays. No RNG, no clock-as-input, no LLM in the consolidation path.

100 50 0 100 modal hash 49 empty buckets 50 empty buckets distinct output hashes distinct output hashes: 1 / 100
Takeaway

Determinism isn't aspirational. It's the default. The consolidator does deterministic SQL aggregation; the persona engine signs every event; the cursor advances atomically with the facts upsert via outbox. Same input, same output, every time.

Source: AEROS replay benchmark Replays: N=100
Benchmark 3

Grounded planner cuts unproductive motion

V1 / V2 / V3 are three planner variants with progressively tighter grounding in the embodied agent's identity manifest and current capability set. We measure the rate of unproductive actions (actions that the watcher rolls back, repeats, or marks as no-ops) across a fixed task suite.

V1 is the ungrounded baseline. V3 reads the identity manifest, the live ECM registry, and the current persona — then plans against that envelope. The chart shows how the unproductive-action rate falls.

100% 75% 50% 25% 0 100% V1 ungrounded — V2 capability-aware * illustrative 25.4% V3 grounded −74.6% unproductive actions, V3 vs V1: 3.9× fewer
Takeaway

Grounding the planner in the embodied identity cuts wasted motion 3–4×. The identity manifest isn't just an audit anchor; it's a planning input that pays for itself.

Source: AEROS planner benchmark Variants: V1 / V2 / V3 Validation: V1, V2, V3 PASS
Reproducibility

Run the test baseline yourself

The runtime and its full test suite are in the open-source repo. Clone, install, run.

1. Clone & set up

git clone https://github.com/s20sc/aeros-core.git
cd aeros-core
make setup   # venv + runtime/dev/bridge/mcp deps

2. Run the test baseline

make test        # full pytest run
# v0.15.0 baseline: 2,483 passed (suite keeps growing)
make test-fast   # minus integration tests

3. Benchmark suite

# The packaged benchmark suite (scenario YAMLs,
# matrix runner, scoring) is being reworked for
# cross-runtime comparability — public release
# is on the roadmap. Perf gates already run in CI.

The figures above were measured on the internal baseline that became the v0.15.0 open-source release.

Want to add your own benchmark?

The next public benchmark is EmbodiedGovBench v2 — a public suite for governance overhead, replay determinism, and recovery latency. It's on the roadmap; contributors welcome.

View the repo →