01recorded pass · unattested
guards
- 5 of 5 checks passed
- 0 failures
- Duration: 617 ms
01 / Live evaluation
Live evaluation
Follow each selected slice from queued to its recorded outcome. Open the provenance panel for the exact source, timestamps, and hash-checked JSON. Producer-recorded results are not independent attestation.
Waiting for the first published run.
Run provenance and downloadable evidence
- Recorded status
- not published
- Feed liveness
- not published
- Target
- not published
- Exact commit
- not published
- Working tree
- not published
- Package version
- not published
- Started at
- not published
- Updated at
- not published
- Downloadable record
- View the record for this runNo validated record is available for this run.
Source package version, not a confirmed installed release. Scope is development only, never an npm equivalence claim.
Recorded results, not an independent audit. Unselected slices remain not run. Models, fixtures, and live chain reads are identified separately below.
Method
How the evaluation runs
Every published number comes out of the same path. The run pins its own source, isolates the runner, records each slice separately, and seals the result with a hash you can recompute.
FIG. 1Production method for one evaluation recordscroll the figure sideways
Method schematic. The stages are drawn to explain the pipeline shape. They are not actual packet counts, timings, or case totals from any run.
01Pin source
The source commit, working-tree state, and source package version identify the run.
02Isolated runner
Child evaluations get a trimmed environment, not the ambient one.
03Measured slices
Each slice reports its own cases, passes, failures, and blocks.
04Immutable revision
Each update is written as a new JSON revision. Earlier revisions are never overwritten.
05Hash check
SHA-256 covers the exact published bytes so mismatches are detectable.
06Public evidence
The same file the site reads is the file you can download.
Architecture
One parent. Bounded research.
Oracle can hand a research question to a child that reads, then returns a result. The parent keeps the tools that matter. The point is delegation you can reason about, not an agent tree that grows on its own.
FIG. 2Parent, one bounded child, and the refused edgescroll the figure sideways
Architecture walkthrough, not a live execution. No tool output is replayed here and nothing on this page is a recording of a session.
Parent holds the working set
The parent binds named variables and keeps them across turns. Values are capped at 16 KB each, so context stays a small readable object rather than a growing transcript.
Source: rlm.bind / rlm.get / rlm.vars in src/tui/rlm.mjs
One bounded research branch
rlm.run hands the child a packet: goal, context, the bound variables, and an allow list of read only tools. The child runs at depth 1 with a step budget of 4 by default and 8 at most.
Source: runRlmTool packet build in src/tui/rlm.mjs
Structured result, then control returns
Only wait=true is supported, so the child answer comes back into the same parent turn as a stored result with a handle. Background runs are refused rather than queued.
Source: wait=false refusal in src/tui/rlm.mjs
The boundary is code, not a prompt
A child cannot call rlm.run again, cannot sign, and cannot write files. Those refusals are evaluated by the host before the tool is dispatched.
Source: evaluateRlmChildTool in src/tui/rlm.mjs
Depth ceiling 1
A child cannot open another child. The ceiling is a constant, so raising the environment variable cannot exceed it.
Source: src/tui/rlm.mjs
12 deterministic checks
Bind, run, result, refusals, and the depth ceiling are graded with a stub child runner. No model call, no network, and no signing take place, so these checks prove mechanics rather than research quality.
Source: src/eval/rlm-eval.mjs
Coverage
Crypto capability, measured in layers
Different layers carry different weight. We keep them apart on purpose: code that was exercised, corpora that exist, and work that is still ahead. There is no single number that blends them.
Implementation checks
12 checks
measured in the release run
Bind, run, result, depth ceiling, and refusal mechanics run with a stub child runner. Deterministic and repeatable.
Source: src/eval/rlm-eval.mjs
Read only preparation
3 cases
measured in the release run
Ethereum, Base, and Arbitrum swap intents are prepared and simulated from an unfunded address. The expected result is execution reverted: STF. Nothing is signed and nothing is broadcast.
Source: scripts/prepare-fidelity-eval.mjs
Policy decisions
45 cases
corpus size, not a fresh pass
An evidence bound classifier that grades the next safe action and the highest lifecycle state already shown. This is a corpus size, not a result from today.
Source: environments/oracle_harness_eval/oracle_harness_eval_v2.py
Mock tool behavior
36 cases
corpus size, not a fresh pass
Twelve reads, twelve simulations, and twelve execute to prepare traps against inert mock tools whose sign and broadcast paths always deny. Also a corpus size, not a result from today.
Source: environments/oracle_harness_eval/oracle_tool_safety.py
Model trajectories
not yet published
planned, in development
Full agent runs on crypto tasks, graded end to end. This layer is in development and we have not published it as a comprehensive benchmark.
Source: in development
FIG. 3Evidence layers, kept separate
Case counts for the policy and tool suites are a corpus size, meaning how many cases exist, not a pass count from a run today. The tool suite uses inert mock tools whose sign and broadcast paths always deny.
Next: agentic crypto tasks
We are building our own task set for crypto agents. It is incomplete and in development, and we are not claiming a result from it. What it will grade:
Research and data honesty
Does the answer match what the tools actually returned, including gaps.
Exact unsigned preparation
Are chain, route, amounts, and calldata right in the artifact it builds.
Policy and confirmation boundaries
Does it stop where a person has to confirm, every time.
Receipt truth
Does it report what settled, without describing work it did not do.
RLM task completion
Does the bounded child branch return usable evidence to the parent.
Protocol build acceptance
Does generated contract work pass tests it never saw.
Case study
Protocol building: historical case study
Historical custom EVM milestone escrow task. Local compile and tests only, no deployment. Existing generated artifacts revalidated offline, not fresh model generations. Oracle means MCP plus protocol-builder skill, not a native-harness head-to-head.
| Arm | Hidden tests | Own tests | Targeted source review |
|---|---|---|---|
| Codex | 8 / 8 | One literal checks-effects-interactions ordering miss in fund() | |
| Codex + Oracle | 9 / 9 | No targeted specification gaps found | |
| Claude | 82 / 82 | No targeted specification gaps found | |
| Claude + Oracle | 91 / 91 | No targeted specification gaps found |
FIG. 4Hidden test acceptance, zero baseline
Historical run, measured 2026-08-27. One custom EVM task. One run per arm. Local compile and tests only, no deploy, and not a formal security audit. This compares Claude and Codex with and without Oracle as an MCP server and skill. It is not an Oracle model against another model.
Self generated tests measure assurance effort, not code quality. An arm can write a small suite it passes easily, so the own tests column is kept separate from the hidden tests it never saw.
Release evidence, separate from development runsPinned release snapshot
02 / Evidence map
Expected eval slices
Every expected slice has a place. Absent, blocked, skipped, and unavailable results are neutral or adverse states—never green and never counted as zero.
02recorded pass · unattested
fidelity
- summary: 3/3 ok, 0 prepare-fail, 0 structural-fail, 0 sim-fail, 3 simulated
- Duration: 9.71 s
03not in this snapshot
golden-prepare
- No result was recorded for this slice.
04recorded pass · unattested
hybrid
- Duration: 6.05 s
05recorded pass · unattested
desk
- 115 of 115 package tests passed
- 0 package-test failures
- Duration: 530 ms
06recorded pass · unattested
agentic
- 46 of 46 package tests passed
- 0 package-test failures
- Duration: 338 ms
07recorded pass · unattested
adversarial
- 160 of 160 package tests passed
- 0 package-test failures
- 7 named guard scenarios
- Duration: 8.19 s
08recorded pass · unattested
contracts
- 9 of 9 checks passed
- 0 failures
- Duration: 976 ms
09recorded pass · unattested
rlm
- 12 of 12 checks passed
- 0 failures
- Duration: 3 ms
10not in this snapshot
native
- No result was recorded for this slice.
03 / Attack surface
Adversarial coverage
Classes explicitly named by the committed report. No inferred coverage and no synthetic score.
- exact-confirmation
- artifact-mutation
- replay-idempotency
- receipt-finality
- autonomous-mint-gates
- exact-nft-contract
- hyperliquid-position-drift
04 / Provenance
Pin the claim
The timestamp, package, commit, and reproduction command define the boundary of this evidence.
- Exact report commit
- 6a9feb3a90b596b22744777143d6d909bc4d31df
- Generated at
- Package version
- @oracle-agent/oracle · 0.35.41
- Eval SHA-256 · canonical JSON
- 5e5c03861006c8d1c783381adf9d0184b6e185b9982db1141e0275674b5169be
- Evidence boundary
- Digest integrity is not cryptographic attestation. The embedded eval payload matches this digest and the release-report schema; serialized producer claims do not independently prove execution or npm-to-source equivalence. This record concerns only the declared package version and exact source commit, never subsequent changes. For source reproduction, check out that commit; the version-pinned npm command is a separate package reproduction. Live read-only providers and the public-site probe can change between runs.
- Methodology and reproduction
- Exact-commit evidence generated by the deterministic oracle eval all aggregate. Fidelity uses read-only chain calls and changes no chain state.
npx @oracle-agent/oracle@0.35.41 eval all --mode playwright --json - Download records
- Full 0.35.41 release report · Archived 0.35.40 report · Prior blocked attempt. The public records declare their local-path redaction and retain original stdout hashes. The first fresh run failed closed during Chromium process cleanup; the separately recorded retry passed.