弹性 MapReduce Benchmarks Guide（基准

配置
BenchmarkConfig 控制基准测试的运行方式：
from tinker_cookbook.eval.benchmarks import run_benchmark, BenchmarkConfig
﻿
config = BenchmarkConfig(
    max_examples=200,         # Cap number of examples (None = all)
    concurrency=64,           # Parallel rollouts (single-turn)
    agent_concurrency=8,      # Parallel rollouts (multi-turn)
    timeout_seconds=300,      # Per-example timeout
    max_tokens=32768,         # Max generation tokens
    temperature=0.6,
    save_dir="evals/run_01",  # Save trajectories and results
)
﻿
result = await run_benchmark("gsm8k", sampling_client, renderer, config)
按模型的默认值
BenchmarkConfig.for_model() 会根据内置表设置 max_tokens、timeout_seconds 和 context_window：
config = BenchmarkConfig.for_model(
    "Qwen/Qwen3.6-35B-A3B",
    save_dir="evals/qwen3.6",
)
# max_tokens=65536, context_window=65536, timeout_seconds=1800
多轮基准测试
Agent 类基准测试（swe_bench、terminal_bench、tau2_bench）需要上下文管理：
config = BenchmarkConfig(
    timeout_seconds=1800,
    max_trajectory_tokens=60000,   # Total tokens across all turns
    max_generation_tokens=8192,    # Tokens per generation step
    agent_concurrency=4,
)
沙箱基准测试
需要执行代码的基准测试（mbpp、livecodebench、swe_bench）依赖沙箱：
from tinker_cookbook.sandbox.modal_sandbox import ModalSandbox
﻿
config = BenchmarkConfig(
    sandbox_factory=ModalSandbox.create,
)
裁判模型基准测试
arena_hard 这类基准测试需要额外的 LLM 裁判。配置细节与示例见 裁判模型。
保存与查看结果
设置 save_dir 以持久化轨迹和结果：
evals/run_01/
├── summary.json                    # Combined scores (run_benchmarks only)
├── gsm8k/
│   ├── result.json                 # BenchmarkResult
│   └── trajectories.jsonl          # One StoredTrajectory per line
└── ifeval/
    ├── result.json
    └── trajectories.jsonl
加载结果
from tinker_cookbook.eval.benchmarks import (
    load_result,
    load_trajectories,
    load_summary,
    print_trajectory,
)
﻿
result = load_result("evals/run_01", "gsm8k")
print(f"Score: {result.score:.1%}, Completed: {result.score_completed:.1%}")
﻿
# Filter trajectories
wrong = load_trajectories("evals/run_01", "gsm8k", incorrect_only=True)
errors = load_trajectories("evals/run_01", "gsm8k", errors_only=True)
print_trajectory(wrong[0])
﻿
# Combined summary across benchmarks
summary = load_summary("evals/run_01")
可续跑
使用相同的 save_dir 重新运行时会跳过已完成的样本。去重基于 example_id（内容哈希），因此即使数据集被打乱也能稳定工作。
理解分数
score — num_correct / num_examples。被截断和出错的样本计为 0。
score_completed — num_correct / num_completed。分母中排除被截断和出错的样本。
对于经常触及 max_tokens 的思考型模型，用 score_completed 与公开分数对比更合适。
result = await run_benchmark("gsm8k", client, renderer, config)
print(f"Raw: {result.score:.1%}")                # 81.7%
print(f"Completed: {result.score_completed:.1%}") # 95.6%
print(f"{result.num_truncated} truncated, {result.num_errors} errors")
对比检查点
checkpoints = {
    "base": "tinker://run-abc/sampler_weights/step0",
    "step500": "tinker://run-abc/sampler_weights/step500",
    "final": "tinker://run-abc/sampler_weights/final",
}
﻿
for name, path in checkpoints.items():
    client = await sc.create_sampling_client_async(model_path=path)
    results = await run_benchmarks(
        ["gsm8k", "ifeval"], client, renderer,
        BenchmarkConfig(save_dir=f"evals/{name}"),
    )
    for bench, r in results.items():
        print(f"{name}/{bench}: {r.score:.1%}")
用 EvalStore 追踪多次运行
EvalStore 通过云端兼容的存储管理跨检查点的评测运行：
from tinker_cookbook.stores.eval_store import EvalStore
﻿
store = EvalStore("~/experiments/evals")
﻿
run_id = store.create_run(
    model_name="Qwen/Qwen3.6-35B-A3B",
    checkpoint_path="tinker://run-123/weights/step500",
    checkpoint_name="step500",
    benchmarks=["gsm8k", "ifeval"],
)
﻿
config = BenchmarkConfig(save_dir=store.run_dir(run_id))
await run_benchmarks(["gsm8k", "ifeval"], client, renderer, config)
﻿
metadata = store.finalize_run(run_id)
print(metadata.scores)  # {"gsm8k": 0.847, "ifeval": 0.923}
Pass@k 评测
设置 num_samples > 1 可计算无偏的 pass@k 估计（Codex 公式）：
config = BenchmarkConfig(num_samples=4, save_dir="evals/pass_at_k")
result = await run_benchmark("gsm8k", sampling_client, renderer, config)
print(result.pass_at_k)  # {1: 0.45, 2: 0.58, 4: 0.72}
自定义
关于系统提示词、自定义评分、答案解析和裁判模型配置，见 自定义基准测试。
新增基准测试
基准测试复用与 RL 训练相同的 Env 协议——单个 MessageEnv 即可同时驱动训练和评测。概念性说明见 共享 Env 抽象。
要新增基准测试，实现一个 BenchmarkBuilder 并注册：
from tinker_cookbook.eval.benchmarks import BenchmarkBuilder, BenchmarkConfig, register
from tinker_cookbook.eval.benchmarks._common import load_benchmark_dataset, make_example_id
from tinker_cookbook.rl.message_env import MessageEnv, MessageStepResult, EnvFromMessageEnv
﻿
class MyMessageEnv(MessageEnv):
    """Single-turn env for one example."""
﻿
    def __init__(self, question: str, expected: str, example_id: str):
        self.question = question
        self.expected = expected
        self._example_id = example_id
﻿
    @property
    def example_id(self) -> str:
        return self._example_id
﻿
    async def initial_observation(self) -> list[Message]:
        return [{"role": "user", "content": self.question}]
﻿
    async def step(self, message: Message) -> MessageStepResult:
        response = get_text_content(message).lower()
        correct = self.expected.lower() in response
        return MessageStepResult(
            reward=1.0 if correct else 0.0,
            episode_done=True,
            next_messages=[],
            metrics={"correct": float(correct)},
            logs={"expected": self.expected},
        )
﻿
class MyBenchmark(BenchmarkBuilder):
    name = "my_benchmark"
    recommended_system_prompt = "Answer concisely."
﻿
    def make_envs(self, renderer, config):
        ds = load_benchmark_dataset("my/dataset", split="test")
        envs = []
        for row in ds:
            env = EnvFromMessageEnv(
                renderer=renderer,
                message_env=MyMessageEnv(
                    question=row["question"],
                    expected=row["answer"],
                    example_id=make_example_id("my_benchmark", row["question"]),
                ),
            )
            envs.append(env)
        return envs
﻿
register(MyBenchmark())
要点：
单轮基准测试用 MessageEnv——它负责提示词构建与响应解析
通过 make_example_id() 设置 example_id，保证多次运行间可稳定续跑
logs 存放每条轨迹的诊断数据（期望答案、抽取出的答案）；metrics 存放用于聚合的数值
多轮基准测试请直接实现 Env，并在 builder 上设置 multi_turn = True
延伸阅读
﻿评测概览：可用基准测试与架构
﻿自定义基准测试：系统提示词、评分、裁判模型、共享 Env 设计
﻿评测教程：交互式演练
﻿RL 环境：基准测试使用的 Env 协议
﻿
Benchmarks Guide（基准测试指南）

本页目录：

配置

按模型的默认值

多轮基准测试

沙箱基准测试

裁判模型基准测试

保存与查看结果

加载结果

可续跑

理解分数

对比检查点

用 EvalStore 追踪多次运行

Pass@k 评测

自定义

新增基准测试

延伸阅读
