Arena from Python

The SDK (arena/sdk.py) gives Arena's primitives to a script or a notebook. Calls are synchronous: the SDK keeps its own event loop on a thread. Everything is in memory unless Arena(database=True) uses the arena service's database, where matches, players, datasets and runs made here appear in the web app. API keys come from the environment (and from the repo's .env, unless Arena(env_file=None)).

Quickstart, no keys

Bots need no model and cost nothing:

from arena import sdk

arena = sdk.Arena()
print(arena.games())                    # ['avalon', 'azul', ..., 'tictactoe', 'werewolf']
print(arena.play("tictactoe", ["tictactoe-perfect", "random"], seed=1))
# tictactoe 887164dd185b: Perfect player 1, Random 0; X wins ($0.0000)
print([p.id for p in arena.list_players("tictactoe")])
# ['random', 'tictactoe-perfect', 'tictactoe-careless', 'ai-fast', 'ai-planner', 'ai-deliberate', 'ai-memory']

The ai-* players and everything below them call models. Chess bots and the chess oracle need stockfish on the PATH.

Keys and costs

A model id names its provider, and each provider reads one key: ANTHROPIC_API_KEY (claude-*), OPENROUTER_API_KEY (openrouter/...), OPENAI_API_KEY (openai/...), GOOGLE_API_KEY (google/...). The house agents (ai-fast, ai-planner, ...) and the lab's learner default to Gemini 3.8 Flash through OpenRouter; bind another model with actor=. Only models with a registered price are offered, and every call is charged against a budget. As a guide, from the arena's own experiments: a match with a fast model at low effort costs about ten cents, and a lab comparison or a short learning run $0.20 to $3.

Arena(database=True) uses the arena service's own (production) database and its settings; it is for the team, and it logs to the experiment tracker. Without it, nothing leaves the process.

Agents and matches

from arena import sdk

arena = sdk.Arena()
fast = arena.agent("ai-fast", actor="claude-haiku-4-5:low")      # a Player, another model bound
for seed in range(4):                                             # matches, stored with their logs: short ones,
    print(arena.play("chess", [fast, "stockfish-1350"],           # from the classical openings, decided on
                     variant="openings", seed=seed))              # material at the move limit

mine = arena.make_agent("planner", actor="claude-sonnet-5:high",   # a new agent: a Version
                        fixed={"instructions": "Trade when ahead."})
better = arena.edit(mine, [{"op": "add", "text": "Castle before move ten."}],
                    why="it left its king in the centre")       # a person's Mutation: a child Version
arena.publish(better, "Careful Planner")                          # a Player: seatable, rated

Data and analysis

probes = arena.mine("chess", "chess-probes", matches=40)   # a Dataset of past mistakes, split by match
print(arena.analyze("drop", probes.items[0]).value)        # any analyzer on any Ref, cached
print(arena.score(fast, probes.split("test"), reps=3))      # a Version asked those decisions again
arena.export([probes.items[0], f"version:{better.id}"], "records.jsonl")

A learning algorithm

An algorithm overrides the stages it changes and inherits the rest:

class Hindsight(sdk.ProbeAlgorithm):
    name = "hindsight"

    class Options(sdk.ProbeOptions):
        failures: int = 4

    async def reflect(self, draft):
        parent = draft.expanded[0]
        worst = self.failures(draft.evidence, draft.measured[parent.id])[: self.options.failures]
        told = [await self.view.story(item, answer) for item, answer in worst]
        found = await self.view.learner.diagnose_probes(told, parent.agent.learnable_config())
        draft.diagnoses = [{"version": parent.id, "probes": [i for i, _ in worst], **found}] if found else []

    async def propose(self, draft):
        parent = draft.expanded[0]
        if draft.diagnoses:
            edit = await self.view.learner.edit(parent.agent.learnable_config(), draft.diagnoses)
            if edit.ops:
                m = self.mutation(draft.k, parent, edit.ops, refs=draft.diagnoses[0]["probes"],
                                  diagnoses=draft.diagnoses, rationale=edit.rationale)
                draft.candidates.append(await self.candidate(m, parent, edit.rejected))

A simple algorithm can be one function, whose keyword-only arguments are its settings:

@sdk.algorithm
async def one_lesson(view, learnable, k, *, text: str = "Develop before you attack."):
    return learnable.apply([{"op": "add", "text": text}]).learnable

Compare, then learn

lab = arena.lab(fast, probes, budget=2)                      # runs on the train split, scored on test
print(lab.compare([sdk.Baseline(), sdk.Generic(), Hindsight(failures=6)]))   # with a noise row

run = arena.learn(fast, Hindsight(), dataset=probes, steps=3, budget=2, publish="if_better")
print(run)                       # the DAG, the held-out final evaluation, the Player it published

Every model call is metered: a lab or a run stops at its budget. The same commands exist on the command line (python -m arena play | learn | lab | evaluate). See Arena primitives for what each object is.