HEMRAEvaluate · Compare · Improve
Know whether your agent harness actually got better.
HEMRA runs AI agent harnesses on pinned benchmark tasks, records every attempt on your machine, and compares them task by task with the benchmark's own scorer.
# pin a dataset, run two harnesses on it three times each, compare
hemra dataset add hf:harborframework/terminal-bench-2.1/tasks --name tb
hemra run -H claude-code -H gemini-cli -d tb -k 3
hemra compare <run>
- Claude Code
- Claude Agent SDK
- Codex SDK
- Gemini CLI
- Goose
- OpenCode
- ChatGPT agent mode
- Codex
- Pi
- any Harbor agent
- your own harness
- 12
- commands
- 6
- ways to run an agent
- 4
- dataset sources, pinned
- 0
- accounts or telemetry
Four nouns, thirteen commands
Everything in HEMRA is a dataset, a harness, a sandbox or a run. A run is datasets × harnesses × attempts, recorded under one ID, and every result is filed under a run.
Pinned by digest
Tasks are copied into the store and pinned, never referenced. Sources: a local Harbor-format folder, harbor:name@version, swebench:verified and hf:org/repo@rev, pinned at one commit and hash-checked.
One TOML file
A harness file's via says how the agent runs: program, cli, sdk, acp, computer-use or harbor. Versions and models are pinned; environment variables are named, never stored.
Docker or local
By default each trial runs in the task's own image in a hardened container: no network, no capabilities, no-new-privileges, bounded memory, CPU and PIDs. --sandbox local runs it as a process.
Queued and durable
One job per trial in a durable queue that survives restarts. Only infrastructure failures are retried, always as a new attempt. --detach hands the run to hemra serve.
Any agent, through its own interface
Native drivers run an agent the way its users do and translate what it does into HEMRA's event protocol. They report what the agent did, keep its raw transcript, and never turn an agent's claim of success into a score.
Command-line agents
Each agent through its own JSON output: claude -p --output-format stream-json, codex exec --json and pi --mode json, with isolated agent profiles and recorded transcripts per tested version.
Agent SDKs
Agent SDKs driven from TypeScript, with permission requests answered by the profile.
Agent Client Protocol
An ACP client for any ACP agent. File and terminal requests are served only inside the workspace, and permission requests are answered by the profile.
Agent products in a browser
Operates a product's own UI in Chromium with a scripted recipe. A login wall or outage is an infrastructure error, never a task failure.
Harbor agents
One supervised Harbor job per dataset. HEMRA keeps everything Harbor wrote and imports the official reward unchanged.
Your own harness
Anything that speaks the HEMRA harness protocol: versioned JSON lines on stdio, in any language. The TypeScript SDK gives you the bootstrap and the session rules.
Built so the numbers hold up
A harness finishing is not a task passing. HEMRA keeps how a run ended and whether the task passed apart, and says where every number came from.
Scored by the task's own verifier
Native runs are scored with the task's unchanged tests/test.sh in a separate container, on a snapshot of the agent's work. Multi-step tasks run step by step from each step's snapshot, every step scored by its own verifier. Verifier problems are evaluation failures, never task failures.
Judges, kept apart
A rubric judge that ships with a dataset is read criterion by criterion, with every judge's reasoning. Your own judges in judges/ score the answer, the artifacts, the tools used or the workspace, with no network and the key held on the host, and never pair with the benchmark's score.
Official scores, imported
hemra import reads Harbor job directories and SWE-bench runs, keeps their files byte for byte and copies rewards verbatim. Versions are pinned and tested against recorded fixtures.
Paired, honest comparison
hemra compare pairs results task by task, only where the scorer fingerprint matches. It flags differing models, unpinned or drifting versions and differing tool sets, and never adds up across datasets.
MCP through a recording proxy
Declare pinned MCP servers in the harness file. Agents reach them only through HEMRA's proxy, which records every tools/call, digests the tool list, and applies allow rules.
Everything recorded locally
Attempts, transitions, event logs streamed as they arrive, and content-addressed artifacts, in SQLite and JSONL on your machine. Export any run as a bundle. No account, no telemetry.
Remote workers
hemra worker leases jobs from a controller over HTTPS. Results commit only under the live lease and generation; anything else is refused and never stored.
Live traces
hemra serve gives you a read-only API and a web dashboard: follow an attempt's events live, open artifacts, read run reports and compare runs, and browse the pinned datasets, the harness files runs used, and every scorer and judge behind the scores.
Checks before you spend
hemra doctor lists the tools it found, the sandboxes and what each enforces. --require fails closed before anything is recorded; --prune removes containers a killed runner left behind.
A Mac app
Browse a store, build and start runs, watch them live, cancel, import and export, get notified when a run finishes, and lease the queue to remote workers. Download
See every step an agent took
The same views in the web dashboard and the Mac app. Recorded from HEMRA's own end-to-end runs.
The command surface
Thirteen top-level commands. New ways to run an agent add a via, not a command.
| Command | What it does |
|---|---|
hemra dataset add|list|show|rm | Pin tasks from a folder, harbor:, swebench: or hf:. |
hemra harness add|list|show|check | Write, inspect and try harness files; check runs one trivial task and compares versions. |
hemra judge add|list|show|check | Write and try your own judges; run --judge applies one, compare --judge compares by it. |
hemra run | Run harnesses × datasets × attempts, in Docker or locally, now or detached. |
hemra runs, hemra show | List runs; show any run, import, job or attempt, its events or its report. |
hemra compare | Pair two imports or run cells, or report a whole run against its baseline. |
hemra import, hemra export | Bring in a Harbor job or SWE-bench run (detected); export anything as a bundle. |
hemra cancel | Cancel a run, job or attempt. |
hemra serve, hemra worker | API, dashboard and queue workers; remote workers leasing the same queue. |
hemra doctor | Tools, sandboxes and their guarantees; prune leftover containers. |
Try it in a minute
HEMRA is in early development (v0.1, Linux and macOS). Build it with Rust 1.80 or later and run the sample harness on the one-task example dataset.
.hemra by default). HEMRA makes no network calls of its own and keeps environment variable names only, never values.git clone https://github.com/UmaiTech/hemra && cd hemra cargo build -p hemra-cli # pin the example tasks and run the sample shell harness on them target/debug/hemra dataset add examples/tasks --name examples target/debug/hemra run -H echo-sh --harness-dir examples/harness-sh \ -d examples -k 3 --sandbox local # read it back, then open the dashboard target/debug/hemra runs target/debug/hemra serve