@playerthree/bench
@playerthree/bench: benchmark series for PlayerThree games — bench configs, the headless match host, the reference
series runner, guardrails and benchmark campaigns. CLI: playerthree-bench.
Interfaces
BenchConfig
Defined in: packages/bench/src/config.ts:20
Private execution configuration. Only spec is persisted/exported; never provider endpoints or credentials.
Properties
| Property | Type | Description | Defined in |
|---|---|---|---|
definition | | GameDefinition<AnyManifest> | DefinitionModule | - | packages/bench/src/config.ts:21 |
guardrails? | readonly Guardrail[] | - | packages/bench/src/config.ts:29 |
models? | Readonly<Record<string, BenchModel>> | By entrant id. | packages/bench/src/config.ts:24 |
policies? | Readonly<Record<string, Readonly<Record<string, PolicyArtifact>>>> | By entrant id, then role. | packages/bench/src/config.ts:26 |
resolve? | (spec) => | MatchSpec | Promise<MatchSpec> | Game hook freezes actual defaults, map identity and canonical rulesHash. Required for games with matchOptions. | packages/bench/src/config.ts:28 |
spec | SeriesSpec | - | packages/bench/src/config.ts:22 |
BenchModel
Defined in: packages/bench/src/config.ts:10
An LLM entrant's provider and model. The key is read from env apiKeyEnv (default OPENAI_API_KEY / ANTHROPIC_API_KEY); scripted replays script without a provider.
Properties
| Property | Type | Description | Defined in |
|---|---|---|---|
apiKeyEnv? | string | - | packages/bench/src/config.ts:13 |
baseUrl? | string | - | packages/bench/src/config.ts:14 |
model | string | - | packages/bench/src/config.ts:12 |
price? | Price | USD per million tokens; defaults to the runner's price table. Required when the spec caps maxCostUsd and the model is not priced. | packages/bench/src/config.ts:16 |
provider | "openai" | "anthropic" | "scripted" | - | packages/bench/src/config.ts:11 |
script? | readonly Turn[] | - | packages/bench/src/config.ts:17 |
DefinitionModule
Defined in: packages/bench/src/bots-config.ts:5
A game definition loaded by dynamic import: module is an absolute file: URL, export the export name (default default).
Properties
| Property | Type | Defined in |
|---|---|---|
export? | string | packages/bench/src/bots-config.ts:5 |
module | string | packages/bench/src/bots-config.ts:5 |
Guardrail
Defined in: packages/bench/src/bots-config.ts:10
A pass/fail check over a series: metric (ticks, truncated, outcome.win|loss|draw or a game metric) per accepted
match, optionally of one scenario, at scope, aggregated and compared with threshold.
Extended by
Properties
| Property | Type | Defined in |
|---|---|---|
aggregation | "min" | "max" | "mean" | "median" | "sum" | "binomialTwoSided" | packages/bench/src/bots-config.ts:15 |
comparator | "<" | "<=" | ">" | ">=" | "==" | "!=" | "between" | packages/bench/src/bots-config.ts:16 |
id | string | packages/bench/src/bots-config.ts:11 |
metric | string | packages/bench/src/bots-config.ts:13 |
scenario? | string | packages/bench/src/bots-config.ts:12 |
scope | "match" | `entrant:${string}` | `team:${string}` | packages/bench/src/bots-config.ts:14 |
threshold | number | readonly [number, number] | packages/bench/src/bots-config.ts:17 |
GuardrailResult
Defined in: packages/bench/src/guardrails.ts:6
A guardrail's verdict. value is the aggregate (the p-value for binomialTwoSided), or null with a reason when it cannot be computed.
Extends
Properties
| Property | Type | Inherited from | Defined in |
|---|---|---|---|
aggregation | "min" | "max" | "mean" | "median" | "sum" | "binomialTwoSided" | Guardrail.aggregation | packages/bench/src/bots-config.ts:15 |
comparator | "<" | "<=" | ">" | ">=" | "==" | "!=" | "between" | Guardrail.comparator | packages/bench/src/bots-config.ts:16 |
id | string | Guardrail.id | packages/bench/src/bots-config.ts:11 |
metric | string | Guardrail.metric | packages/bench/src/bots-config.ts:13 |
passed | boolean | - | packages/bench/src/guardrails.ts:9 |
reason? | string | - | packages/bench/src/guardrails.ts:10 |
samples | number | - | packages/bench/src/guardrails.ts:8 |
scenario? | string | Guardrail.scenario | packages/bench/src/bots-config.ts:12 |
scope | "match" | `entrant:${string}` | `team:${string}` | Guardrail.scope | packages/bench/src/bots-config.ts:14 |
threshold | number | readonly [number, number] | Guardrail.threshold | packages/bench/src/bots-config.ts:17 |
value | number | null | - | packages/bench/src/guardrails.ts:7 |
RunBenchOptions
Defined in: packages/bench/src/run.ts:12
outDir receives the store and the exports; owner names the series owner (default benchmark_owner); aborting signal aborts the series.
Properties
| Property | Type | Defined in |
|---|---|---|
onProgress? | (completed, total) => void | packages/bench/src/run.ts:12 |
outDir | string | packages/bench/src/run.ts:12 |
owner? | string | packages/bench/src/run.ts:12 |
signal? | AbortSignal | packages/bench/src/run.ts:12 |
Functions
binomialTwoSided()
function binomialTwoSided(wins, total): number;Defined in: packages/bench/src/guardrails.ts:14
Exact two-sided fair-coin tail, accumulated in log space to avoid underflow at large n.
Parameters
| Parameter | Type |
|---|---|
wins | number |
total | number |
Returns
number
createGameMatchHost()
function createGameMatchHost(definition, config): object;Defined in: packages/bench/src/bot-match.ts:189
Headless GameDefinition adapter. Scheduling, retries, storage and ratings belong exclusively to SeriesRunner.
Parameters
| Parameter | Type |
|---|---|
definition | GameDefinition |
config | BenchConfig |
Returns
object
| Name | Type | Defined in |
|---|---|---|
binder | SeatBinder | packages/bench/src/bot-match.ts:189 |
host | GameMatchHost | packages/bench/src/bot-match.ts:189 |
author() | (candidateId, signal?) => Promise<AuthoredPolicies> | packages/bench/src/bot-match.ts:189 |
close() | () => Promise<void> | packages/bench/src/bot-match.ts:189 |
evaluateGuardrails()
function evaluateGuardrails(results, guardrails): GuardrailResult[];Defined in: packages/bench/src/guardrails.ts:28
Evaluates each guardrail; a guardrail that lacks any sample fails.
Parameters
| Parameter | Type |
|---|---|
results | ResultsEnvelope |
guardrails | readonly Guardrail[] |
Returns
loadBenchConfig()
function loadBenchConfig(file): Promise<BenchConfig | readonly BenchConfig[]>;Defined in: packages/bench/src/config.ts:71
Imports the config module at file (trusted operator code; Node runs TypeScript directly) and validates its default export: one config or a nonempty list.
Parameters
| Parameter | Type |
|---|---|
file | string |
Returns
Promise<BenchConfig | readonly BenchConfig[]>
loadDefinition()
function loadDefinition(ref): Promise<GameDefinition<AnyManifest>>;Defined in: packages/bench/src/bots-config.ts:37
Returns an inline definition as is, or imports a DefinitionModule and checks its export is a game definition.
Parameters
| Parameter | Type |
|---|---|
ref | | GameDefinition<AnyManifest> | DefinitionModule |
Returns
Promise<GameDefinition<AnyManifest>>
runBench()
function runBench(config, options): Promise<ResultsEnvelope>;Defined in: packages/bench/src/run.ts:15
All tracks use the same accepted schedule, lifecycle, durable store and export service; returns the full canonical download.
Parameters
| Parameter | Type |
|---|---|
config | BenchConfig |
options | RunBenchOptions |
Returns
Promise<ResultsEnvelope>
snapshotMetrics()
function snapshotMetrics(raw, teams): GameMetrics;Defined in: packages/bench/src/bot-match.ts:27
Validates and copies a game's metrics snapshot: plain maps of bounded keys and finite numbers, known teams only, at most 1 MiB of JSON.
Parameters
| Parameter | Type |
|---|---|
raw | GameMetrics |
teams | readonly string[] |
Returns
validateConfig()
function validateConfig(config, definition?): void;Defined in: packages/bench/src/config.ts:33
Throws playerthree-bench: … on an invalid config. With definition, also checks the pinned surfaceHash and policy roles against the game.
Parameters
| Parameter | Type |
|---|---|
config | BenchConfig |
definition? | GameDefinition<AnyManifest> |
Returns
void
References
benchmarkSourceDigest
Re-exports benchmarkSourceDigest
BenchmarkSources
Re-exports BenchmarkSources
BenchmarkSuiteMetadata
Re-exports BenchmarkSuiteMetadata
BoundSeat
Re-exports BoundSeat
CampaignAttempt
Re-exports CampaignAttempt
CampaignConfig
Re-exports CampaignConfig
CampaignOptions
Re-exports CampaignOptions
CampaignRecord
Re-exports CampaignRecord
ClosableSeriesRunner
Re-exports ClosableSeriesRunner
createBenchmarkSuite
Re-exports createBenchmarkSuite
createSeriesRunner
Re-exports createSeriesRunner
DEFAULT_SERIES_LIMITS
Re-exports DEFAULT_SERIES_LIMITS
exportCampaignPublication
Re-exports exportCampaignPublication
GameMatchHost
Re-exports GameMatchHost
GameMatchResult
Re-exports GameMatchResult
MatchExecution
Re-exports MatchExecution
MatchSpec
Re-exports MatchSpec
recoverSeriesRunner
Re-exports recoverSeriesRunner
ResultsEnvelope
Re-exports ResultsEnvelope
runCampaign
Re-exports runCampaign
SeatBinder
Re-exports SeatBinder
SeriesRunnerOptions
Re-exports SeriesRunnerOptions
SeriesSpec
Re-exports SeriesSpec