playerthree/benchmark

playerthree/benchmark: fixed-reference benchmark suites and publications (SPEC §20 Fixed-reference benchmark publications) — types, digests and sealing, weighted scoring with a crossed run × block bootstrap, and validation.

Interfaces

BenchmarkCost

Defined in: packages/playerthree/src/benchmark.ts:28

Properties

PropertyTypeDefined in
amountstring | nullpackages/playerthree/src/benchmark.ts:29
coverage"complete" | "partial" | "unknown"packages/playerthree/src/benchmark.ts:30
currency"USD" | nullpackages/playerthree/src/benchmark.ts:29
pricedAtstring | nullpackages/playerthree/src/benchmark.ts:31
pricingVersionstring | nullpackages/playerthree/src/benchmark.ts:31
scope"authoring-run"packages/playerthree/src/benchmark.ts:29
source"unknown" | "runner-estimate"packages/playerthree/src/benchmark.ts:30

BenchmarkExperiment

Defined in: packages/playerthree/src/benchmark.ts:22

Properties

PropertyTypeDescriptionDefined in
digeststring-packages/playerthree/src/benchmark.ts:26
executionDigeststring | null-packages/playerthree/src/benchmark.ts:24
idstringNull in anonymous exports: a digest of a guessable provider/model/price configuration re-identifies it.packages/playerthree/src/benchmark.ts:24
provenance"runner-observed" | "submitted-artifact"-packages/playerthree/src/benchmark.ts:26
replicatesnumber-packages/playerthree/src/benchmark.ts:26
suiteDigeststring-packages/playerthree/src/benchmark.ts:24
systemobject-packages/playerthree/src/benchmark.ts:25
system.configurationDigeststring-packages/playerthree/src/benchmark.ts:25
system.harnessstring-packages/playerthree/src/benchmark.ts:25
system.idstring-packages/playerthree/src/benchmark.ts:25
system.labelstring-packages/playerthree/src/benchmark.ts:25
system.modelstring | null-packages/playerthree/src/benchmark.ts:25
system.providerstring | null-packages/playerthree/src/benchmark.ts:25

BenchmarkPublication

Defined in: packages/playerthree/src/benchmark.ts:50

Properties

PropertyTypeDefined in
consentobjectpackages/playerthree/src/benchmark.ts:53
consent.artifactsfalsepackages/playerthree/src/benchmark.ts:53
consent.attributionbooleanpackages/playerthree/src/benchmark.ts:53
consent.licensestringpackages/playerthree/src/benchmark.ts:53
consent.researchbooleanpackages/playerthree/src/benchmark.ts:53
consent.resultstruepackages/playerthree/src/benchmark.ts:53
consent.trainingbooleanpackages/playerthree/src/benchmark.ts:53
digeststringpackages/playerthree/src/benchmark.ts:54
experimentBenchmarkExperimentpackages/playerthree/src/benchmark.ts:52
format"playerthree.benchmark-publication"packages/playerthree/src/benchmark.ts:51
idstringpackages/playerthree/src/benchmark.ts:51
publishedAtstringpackages/playerthree/src/benchmark.ts:51
retentionobjectpackages/playerthree/src/benchmark.ts:54
retention.policy"until-withdrawn"packages/playerthree/src/benchmark.ts:54
retention.withdrawalNotice"Downloaded copies cannot be recalled."packages/playerthree/src/benchmark.ts:54
revision1packages/playerthree/src/benchmark.ts:51
runsBenchmarkRun[]packages/playerthree/src/benchmark.ts:52
scoreBenchmarkScorepackages/playerthree/src/benchmark.ts:52
suiteBenchmarkSuitepackages/playerthree/src/benchmark.ts:52
version1packages/playerthree/src/benchmark.ts:51

BenchmarkRun

Defined in: packages/playerthree/src/benchmark.ts:37

Properties

PropertyTypeDescriptionDefined in
artifactDigeststring | null-packages/playerthree/src/benchmark.ts:41
authoringAttemptsnumberAuthoring attempts started for this replicate; infrastructure failures re-author it within the frozen retry limit.packages/playerthree/src/benchmark.ts:40
costBenchmarkCost-packages/playerthree/src/benchmark.ts:42
finishedAtstring | null-packages/playerthree/src/benchmark.ts:42
idstring-packages/playerthree/src/benchmark.ts:38
indexnumber-packages/playerthree/src/benchmark.ts:38
reasonstring | null-packages/playerthree/src/benchmark.ts:38
seriesIdstring | null-packages/playerthree/src/benchmark.ts:41
seriesRevisionnumber | null-packages/playerthree/src/benchmark.ts:41
startedAtstring | null-packages/playerthree/src/benchmark.ts:42
status"failed" | "authoring" | "pending" | "frozen" | "evaluated"-packages/playerthree/src/benchmark.ts:38
trialsBenchmarkTrial[]-packages/playerthree/src/benchmark.ts:42

BenchmarkScore

Defined in: packages/playerthree/src/benchmark.ts:44

Properties

PropertyTypeDefined in
completebooleanpackages/playerthree/src/benchmark.ts:45
completedRunsnumberpackages/playerthree/src/benchmark.ts:45
expectedRunsnumberpackages/playerthree/src/benchmark.ts:45
expectedTrialsnumberpackages/playerthree/src/benchmark.ts:46
failedRunsnumberpackages/playerthree/src/benchmark.ts:45
interval[number, number] | nullpackages/playerthree/src/benchmark.ts:47
runScores(number | null)[]packages/playerthree/src/benchmark.ts:46
scoredTrialsnumberpackages/playerthree/src/benchmark.ts:46
uncertaintyobjectpackages/playerthree/src/benchmark.ts:48
uncertainty.blocksnumberpackages/playerthree/src/benchmark.ts:48
uncertainty.method"crossed-run-block-bootstrap/1"packages/playerthree/src/benchmark.ts:48
uncertainty.replicatesnumberpackages/playerthree/src/benchmark.ts:48
uncertainty.runsnumberpackages/playerthree/src/benchmark.ts:48
uncertainty.seednumberpackages/playerthree/src/benchmark.ts:48
valuenumber | nullpackages/playerthree/src/benchmark.ts:46

BenchmarkSuite

Defined in: packages/playerthree/src/benchmark.ts:9

Properties

PropertyTypeDefined in
authoringobjectpackages/playerthree/src/benchmark.ts:17
authoring.budgetsobjectpackages/playerthree/src/benchmark.ts:17
authoring.budgets.maxCostUsdstring | nullpackages/playerthree/src/arena.ts:44
authoring.budgets.maxTokensnumber | nullpackages/playerthree/src/arena.ts:44
authoring.budgets.maxTurnsnumber | nullpackages/playerthree/src/arena.ts:44
authoring.maxWallMsnumberpackages/playerthree/src/benchmark.ts:17
authoring.practiceSeedsnumber[]packages/playerthree/src/benchmark.ts:17
authoring.replicatesnumberpackages/playerthree/src/benchmark.ts:17
blocksobject[]packages/playerthree/src/benchmark.ts:15
buildobjectpackages/playerthree/src/benchmark.ts:12
build.gameDigeststringpackages/playerthree/src/benchmark.ts:12
build.runnerDigeststringpackages/playerthree/src/benchmark.ts:12
digeststringpackages/playerthree/src/benchmark.ts:20
evaluationobjectpackages/playerthree/src/benchmark.ts:16
evaluation.permutationsstring[]packages/playerthree/src/benchmark.ts:16
evaluation.repetitionsnumberpackages/playerthree/src/benchmark.ts:16
gamestringpackages/playerthree/src/benchmark.ts:11
gameVersionstringpackages/playerthree/src/benchmark.ts:11
harnessobjectpackages/playerthree/src/benchmark.ts:20
harness.idstringpackages/playerthree/src/benchmark.ts:20
harness.versionstringpackages/playerthree/src/benchmark.ts:20
idstringpackages/playerthree/src/benchmark.ts:10
referencesobject[]packages/playerthree/src/benchmark.ts:13
rolesstring[]packages/playerthree/src/benchmark.ts:13
scoringobjectpackages/playerthree/src/benchmark.ts:18
scoring.authoringFailure"zero"packages/playerthree/src/benchmark.ts:18
scoring.method"weighted-score/1"packages/playerthree/src/benchmark.ts:18
scoring.timeout"draw"packages/playerthree/src/benchmark.ts:18
seriesDigeststringpackages/playerthree/src/benchmark.ts:11
strataobject[]packages/playerthree/src/benchmark.ts:14
surfaceHashstringpackages/playerthree/src/benchmark.ts:11
titlestringpackages/playerthree/src/benchmark.ts:10
track"authored-policy" | "submitted-policy"packages/playerthree/src/benchmark.ts:10
uncertaintyobjectpackages/playerthree/src/benchmark.ts:19
uncertainty.method"crossed-run-block-bootstrap/1"packages/playerthree/src/benchmark.ts:19
uncertainty.replicatesnumberpackages/playerthree/src/benchmark.ts:19
uncertainty.seednumberpackages/playerthree/src/benchmark.ts:19
versionnumberpackages/playerthree/src/benchmark.ts:10

BenchmarkTrial

Defined in: packages/playerthree/src/benchmark.ts:33

Properties

PropertyTypeDefined in
attemptsnumberpackages/playerthree/src/benchmark.ts:35
blockIdstringpackages/playerthree/src/benchmark.ts:34
permutationIdstringpackages/playerthree/src/benchmark.ts:34
scorenumber | nullpackages/playerthree/src/benchmark.ts:35
status"win" | "loss" | "draw" | "forfeit" | "timeout" | "incomplete"packages/playerthree/src/benchmark.ts:35
stratumIdstringpackages/playerthree/src/benchmark.ts:34
trialIdstringpackages/playerthree/src/benchmark.ts:34

Functions

benchmarkDigest()

function benchmarkDigest(value): string;

Defined in: packages/playerthree/src/benchmark.ts:57

SHA-256 hex of the canonical JSON of value.

Parameters

ParameterType
valueunknown

Returns

string

scoreBenchmark()

function scoreBenchmark(suite, runs): BenchmarkScore;

Defined in: packages/playerthree/src/benchmark.ts:62

Explicit blocks preserve all mirrors; the same sampled blocks are shared across sampled runs.

Parameters

ParameterType
suiteBenchmarkSuite
runsreadonly BenchmarkRun[]

Returns

BenchmarkScore

sealBenchmark()

function sealBenchmark<T>(value): T & object;

Defined in: packages/playerthree/src/benchmark.ts:59

value plus digest, the benchmarkDigest of value (which must not carry a digest yet).

Type Parameters

Type Parameter
T extends object

Parameters

ParameterType
valueT

Returns

T & object

validateBenchmarkPublication()

function validateBenchmarkPublication(input): BenchmarkPublication;

Defined in: packages/playerthree/src/benchmark.ts:182

Checks a publication end to end — shape, digests, suite, run lineage, adjudication, anonymity and the recomputed score; returns a copy.

Parameters

ParameterType
inputunknown

Returns

BenchmarkPublication

validateBenchmarkSuite()

function validateBenchmarkSuite(input): BenchmarkSuite;

Defined in: packages/playerthree/src/benchmark.ts:172

Checks a suite's shape, digest and bounds; returns a copy.

Parameters

ParameterType
inputunknown

Returns

BenchmarkSuite