Tau2

• Airline 0.8200 (41 / 50)
• Retail 0.7632 (87 / 114)
• Telecom 0.8772 (100 / 114)
• Native user simulator gpt-4.1-2025-04-14
• GEODE recorded gpt-5.2 PAYG usage locally; user simulator cost is visible through OpenAI billing, not GEODE's usage ledger.

# Aggregate of the three per-domain native tau2 runs listed above.
# Do not average this with mock smoke or GEODE geode_user rows.

• This weighted aggregate is for internal Agent-World-style comparison only.
• The run spec differs from OpenAI's official GPT-5.2 Tau2 headline, which used an internal research setup and excludes Airline.
• The run spec differs from the earlier GEODE geode_user smoke matrix.

• DB match 31 / 114
• Write actions 471 / 496
• Generic actions 20 / 20
• Termination user_stop 114 / 114
• Duration total 28827.72s / avg 252.87s / max 818.58s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 4 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-telecom-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• Concurrency was raised from 2 to 4 mid-run and resumed from tau2 checkpoints; no rate-limit, quota, or billing errors were observed.
• Failures clustered around multi-issue MMS/mobile-data/service cases where one required APN, permission, roaming, or data-refuel action was omitted.

• DB match 88 / 113
• Read actions 320 / 354
• Write actions 140 / 174
• Termination user_stop 113 / 114, too_many_errors 1 / 114
• Duration total 23543.64s / avg 206.52s / max 873.92s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain retail \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-retail-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• The main failure mode was missing required side-effect actions even when the natural-language response looked plausible.
• One task terminated with too_many_errors; the remaining failures ended with user_stop but failed verifier assertions.

• DB match 42 / 50
• Read actions 81 / 91
• Write actions 33 / 49
• Termination user_stop 50 / 50
• Duration total 14205.02s / avg 284.10s / max 979.65s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain airline \
  --task-split-name base \
  --num-tasks 50 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-airline-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• This is the native tau2 user_simulator comparator track, not the GEODE geode_user smoke track.
• Airline is retained for internal trend comparison; OpenAI's GPT-5.2 announcement excludes Airline from its Tau2 headline due to lower-quality ground truth grading.

• DB check 1.0
• create_task action check 1.0
• Termination user_stop
• Duration 54.90s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.5 \
  --provider openai \
  --source subscription \
  --effort xhigh \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.5 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-5-xhigh-geode-user-mock-smoke-20260703-r5 \
  --log-level INFO \
  --verbose-logs

• This is a tau2 wiring/regression smoke, not a tau2 leaderboard score.
• Do not average it with native tau2 user_simulator runs using gpt-4.1 or gpt-5.2.
• Both assistant and simulated user used the GEODE subscription route.