Arena primitives
Arena is a game and agent platform: people, scripted bots and AI agents play the same matches, every match is a log that replays exactly, and everything that changes an agent is a recorded, comparable change. This page describes its abstractions, layer by layer. The code lives in arena/; the design document is arena/docs/abstractions.md.
The layers
Each layer depends only on the layers below it, and a test fails the build otherwise.
| Layer | Primitives |
|---|---|
| Arena | Player · Rating (per Pool) · Pool · Experiment · web, CLI, SDK |
| Learning | Mutation · Reflection · LearningStep · LearningAlgorithm · LearningRun · Lab |
| Data | Dataset · mining · Export |
| Analysis | Analyzer → Analysis · oracles · play-time access |
| Match | MatchSpec and its log · GameStep · AgentStep · MatchRunner · MatchManager · fork · probe |
| Agent | Harness · Tool · Learnable (Playbook, Memory) · EditGrammar · Agent · Version · bots |
| Model | ModelConfig · Trainer |
| Game | Game · RuleBook · Request, Action, Event, Viewer · Outcome · Seat |
| Foundation | Ref · registry · statistics |
Foundation
Ref is the address of any record, with versions pinned: game:chess@1.0.0, harness:fast@1, model:claude-sonnet-5:high, version:468a74c0e1b2, match:2c953c67/step/41/seat/1, dataset:chess-probes@5. Code that can be named in a record (games, harnesses, learnables, analyzers, learning algorithms) is in a registry by name and version. Resolving a version that doesn't exist is an error, never a silent "latest".
Game
A Game is a pure, deterministic state machine: setup, requests (who must decide what), validate, apply (returning a Transition: the next state and its events), observe (what a viewer may see) and outcome. All randomness is recorded, so a log replays exactly. A change that would alter a replay bumps the game's version.
- RuleBook: the rules as text, rendered for the match's options. People read it in the lobby; agents read it in their prompt. It never contains strategy.
- Request, Action, Event, Viewer: what is asked, what is answered, what happened, and who may see it. Secrecy lives in audiences and viewers.
- Outcome: a score per seat. Win, draw and loss are derived, so team, cooperative and scored games all fit.
- Seat: anything that fills a seat in a match answers
decide(turn) -> Decision. A person's browser, a bot and an agent all do.
Model
A ModelConfig is a model with its settings, priced as a unit. Its id says everything that distinguishes it: claude-sonnet-5:high, or claude-sonnet-5:high+max_tokens=8000. It carries a price (only priced models are offered) and its capabilities (tool use, readable reasoning, structured output). A change is a new id, so each config is rated as itself.
A Trainer turns a Dataset into a new ModelConfig whose origin names the training run. The contract exists; no trainer ships yet.
Agent
A Harness is an agent's code: its fixed settings (written by people), its learnable config (the only part learning may change), and its roles (an actor that plays, possibly a judge or reflector), each requiring capabilities from the model bound to it. The house harnesses are fast (decides from the position), planner (keeps a working state across moves), deliberate (may call tools before it answers), memory (reads the lessons that fit the situation, and writes new ones after each match) and checklist (before each answer, says for every check on its list, such as the game's motifs or what it has learned, whether and how it applies to the position).
A Tool is something an agent may call while it decides, registered by name and version like everything else: rules (look up the rules), notes (a scratchpad for the match), analyze (an analyzer, only as the agent's access policy allows), recall (search its memory). Every call and its result is recorded with the decision. A tool never changes the match.
A Memory is a learnable that holds many lessons, of which each decision reads only the few that match the situation. An agent that remembers writes lessons after each match; each becomes a Mutation, and an online Player plays its next match as the new Version.
A Learnable declares its own EditGrammar: the operations a learner may propose and their limits. The house learnable is the Playbook, a list of lessons with stable ids (add, update, delete). Applying edits returns what was applied and what was rejected, with a reason for each. render returns what reached the model, by item id and content hash, so credit assignment uses the record rather than the model's own account.
An Agent is all of that as data: a harness reference, fixed overrides, the learnable data, role bindings to ModelConfigs, and an access policy. A Version is an Agent addressed by its content hash. Change anything, even one binding, and it is another Version, rated as itself.
Bots are a family (how to build a scripted seat) with params. Games ship their own bots.
Match
A match is its MatchSpec (game and version, seats, options, seed, clocks, budget, rating pool, who sees the oracle and who sees AI reasoning while it is played) and its log. While a match is played, a watcher who may look past the table (an admin who doesn't play in it, or anyone when only agents and bots play) can choose a lens: everything (the true state and every AI's reasoning, live) or what one seat sees (?lens=all, ?lens=<seat>). The log is read at three grains:
| Grain | What it is | Ref |
|---|---|---|
| MatchStep | One log entry: every seat's actions, events, randomness drawn | match:x/step/41 |
| GameStep | One seat's turn and its answer | match:x/step/41/seat/1 |
| AgentStep | A GameStep plus a trace: reasoning, rendered context, tool calls, cost, time | same Ref |
The MatchRunner is the only code that changes a match: it asks, validates, applies, logs and tells watchers. The MatchManager runs every match the same way, whoever started it (the lobby, an experiment, a learning run, the SDK), and resumes matches from their logs after a restart.
A fork is a new match whose spec names its parent and the position it starts from; its log holds only its own entries. A probe is a fork that stops after one decision: a past decision asked again of another Version, graded the same way as the original.
Analysis
An Analyzer judges a target Ref and returns an Analysis, cached by analyzer, version, options and target, so each is computed once.
| Analyzer | Target | Value |
|---|---|---|
engine_eval |
position | the game's oracle (Stockfish, poker equity): each seat's chances, the best move |
drop |
decision | how far a decision lowered the mover's chances; only for oracles that value the whole match (none for poker's hand equity, where every sound fold loses equity) |
rollout, choice_distribution, agent_diff |
decision | answers played out, re-asked, or compared between Versions |
judge, mistake_kind |
decision | a model's assessment; the kind of mistake |
checkpoint |
Version | its solve rate on a set of probes |
outcome, timing, playbook_credit |
match, decision | result; time and cost; which lessons were in context |
An agent's access policy says which analyzers it may call while it plays. The default is none. Even when allowed, an oracle is never consulted before the end of a game with secrets.
Data
A Dataset is a frozen, versioned list of Refs, with splits made by match (every decision from one match is on the same side), and facts per item. Datasets of probes are mined from stored matches: the decisions an analyzer flags. Export writes any record as JSONL, ending with the Refs it contains and the Refs it needs.
Learning
A Mutation is the unit of change: a parent Version, a target (the learnable, a role binding, or the harness itself), operations in the target's grammar, a producer (an algorithm, a person) and a Reflection whose one required field is refs, the records it came from. Applying a Mutation makes a child Version.
A LearningAlgorithm decides everything; it is written on a staged skeleton (select, sample, reflect, propose, evaluate, accept, carry), overriding only the stages it changes. A LearningRun records a DAG of Versions and Mutations, resumes after its last complete step, ends with a final evaluation on held-out data, and publishes a new Player only when the result is better than its start. The Lab runs algorithms as learning runs on a frozen Dataset, against controls and a noise row.
Arena
A Player is an identity: a person, a bot, or an agent Version. Players are seated by id, optionally binding another model for one match:
{"kind": "player", "player": "ai-fast", "actor": "claude-sonnet-5:high"}
A Rating is a distribution (OpenSkill mean and deviation) per game and Pool. A Pool states which games, which kinds of players and which play-time analyzer access it admits. An Experiment is a declarative batch of matches (round robin, gauntlet, Swiss, duplicate or benchmark) reduced to ratings with intervals.
The experiment log
Every experiment, learning run and lab comparison is tracked in one log, wherever it ran (the web app, the SDK, the command line): its kind, game, status, progress, spend against its budget, headline result, its intent (the hypothesis) and its learnings (what it showed, written once the result is read). It is the Tracker page in the arena; the narrative, one section per experiment, is arena/EXPERIMENTS.md in the repository.
See Arena from Python for the same primitives in code.