Tau2

• Airline 0.8400 (42 / 50)
• Retail 0.6930 (79 / 114)
• Telecom 0.6930 (79 / 114)
• 51,985 canonical events / 3,964 exact tool pairs / zero orphans
• Telecom p95 957.65s / 14 max-step terminations / MMS 21 of 49

# Run once per domain with num-tasks 50 (airline) or 114 (retail/telecom).
python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain <airline|retail|telecom> \
  --task-split-name base \
  --num-tasks <50|114> \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --max-errors 1 \
  --max-retries <0|1> \
  --timeout 3600 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 600 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --trajectory-stage benchmark \
  --save-to <domain-specific-run-id>

• This GEODE-user full cycle is not comparable to the native tau2 user_simulator headline matrix.
• Tau2 results.json is score authority; the trajectory release is a privacy-reviewed diagnostic and external-loop sidecar.
• Seven Telecom transport retries created 14 extra SQLite sessions outside the final trajectory parents; no behavior-score failure was retried.
• The released trajectories are scope-complete for final task attempts and replay-incomplete for bounded bodies and retry-attempt lineage.
• The isolated Tau2 AgenticLoop records no public hook_events; the separate hook behavior E2E remains hook authority.

• Termination max_steps before native component scoring
• Duration 236.73s
• 203 canonical events / 14 exact tool pairs
• Repeated customer, line, network, usage, restriction, and VPN diagnostics

python scripts/eval/tau2_geode_agent.py   --harness-dir artifacts/eval/harnesses/tau2-bench   --domain telecom   --task-split-name small   --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]'   --num-tasks 1   --num-trials 1   --max-concurrency 1   --max-steps 50   --timeout 1800   --model gpt-5.4   --provider openai   --source subscription   --effort high   --time-budget-s 300   --user geode_user   --user-llm gpt-5.4   --user-provider openai   --user-source subscription   --user-effort high   --user-time-budget-s 180   --trajectory-stage benchmark   --save-to geode-gpt54-high-v1.0.12-f99cea63-geode-user-telecom-small-01-20260803

• The run reached 50 steps before native DB/action scoring; repeated diagnostics are preserved as behavior evidence.
• All fourteen tool calls have exactly one result, with no route, authentication, quota, or adapter failure.
• This two-task release smoke does not invalidate or replace the 200/278 full-cycle diagnostic.

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 13.75s / 31 canonical events / 2 exact tool pairs

python scripts/eval/tau2_geode_agent.py   --harness-dir artifacts/eval/harnesses/tau2-bench   --domain mock   --task-ids create_task_1   --num-tasks 1   --num-trials 1   --max-concurrency 1   --max-steps 8   --timeout 900   --model gpt-5.4   --provider openai   --source subscription   --effort high   --time-budget-s 180   --user geode_user   --user-llm gpt-5.4   --user-provider openai   --user-source subscription   --user-effort high   --user-time-budget-s 120   --trajectory-stage benchmark   --save-to geode-gpt54-high-v1.0.12-f99cea63-geode-user-mock-smoke-20260803

• The simulated user stopped before a verifier-compatible state change; DB and action checks are zero while communication is one.
• The run has no authentication, quota, provider-adapter, or harness exception and is retained without retry.
• This release smoke is not a rerun or replacement of the 278-task full cycle and is not a native user_simulator leaderboard row.

• DB check 1.0
• toggle_roaming write action 1.0
• Mobile-data and excellent-speed assertions 1.0
• Termination user_stop
• Duration 119.83s / 127 canonical events / 8 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 300 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --save-to geode-gpt54-high-afaab52b-geode-user-telecom-small-01-20260802

• The DB, toggle_roaming, mobile-data, and excellent-speed checks all passed.
• No route, provider, adapter, quota, agent, or simulated-user exception occurred.
• Tau2 results.json is the score authority; the 127-event trajectory is a digest-joined correlation and replay sidecar.
• The immutable source snapshot retains the runner-default train stage with promotion_authority=none; future benchmark commands should set --trajectory-stage benchmark explicitly.

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 25.33s
• 31 canonical events / 2 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --task-ids create_task_1 \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt54-high-afaab52b-geode-user-mock-smoke-20260802

• The model supplied unrequested description=""; Tau2's exact action and DB comparators rejected it.
• No route, provider, adapter, quota, agent, or simulated-user exception occurred.
• This fixed GEODE-user diagnostic is not a native user_simulator headline row.
• The immutable source snapshot retains the runner-default train stage with promotion_authority=none; future benchmark commands should set --trajectory-stage benchmark explicitly.

• Airline 0.8200 (41 / 50)
• Retail 0.7632 (87 / 114)
• Telecom 0.8772 (100 / 114)
• Native user simulator gpt-4.1-2025-04-14
• GEODE recorded gpt-5.2 PAYG usage locally; user simulator cost is visible through OpenAI billing, not GEODE's usage ledger.

# Aggregate of the three per-domain native tau2 runs listed above.
# Do not average this with mock smoke or GEODE geode_user rows.

• This weighted aggregate is for internal Agent-World-style comparison only.
• The run spec differs from OpenAI's official GPT-5.2 Tau2 headline, which used an internal research setup and excludes Airline.
• The run spec differs from the earlier GEODE geode_user smoke matrix.

• DB check 1.0
• toggle_roaming write action 1.0
• Mobile-data and excellent-speed assertions 1.0
• Termination user_stop
• Duration 78.52s / 117 canonical events / 8 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 300 \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --save-to geode-gpt56-sol-high-v1011-686ff372-geode-user-telecom-small-01-20260731

• The earlier premature human-transfer failure is closed for this fixed case.
• Tau2 native DB/action/assertion checks remain the score authority; the GEODE trajectory is a digest-joined replay sidecar.
• The Crucible v3 snapshot remains diagnostic with promotion_authority=none because no frozen experiment contract was supplied.

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 9.03s
• 25 canonical events / 1 exactly paired tool call and result

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt56-sol-high-v1011-686ff372-geode-user-mock-smoke-20260731

• The failure reproduces the earlier behavior: create_task includes unrequested description="" and the native exact comparator rejects it.
• The run completed normally and is retained without retry or relabeling.
• This diagnostic has promotion_authority=none and is not a native user_simulator leaderboard row.

• Required user toggle_roaming action 0.0
• Mobile-data and excellent-speed assertions 0.0
• Termination user_stop after human transfer
• Duration 51.91s

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --save-to geode-gpt56-sol-high-edb74602b-geode-user-telecom-small-01-20260731

• The agent correctly identified the customer, line, roaming state, and data usage.
• It then declared device tools unavailable and transferred to a human instead of guiding the user-side roaming/device workflow.
• No provider, quota, or adapter exception occurred; the failure is retained as behavior evidence.

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 14.58s

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --save-to geode-gpt56-sol-high-edb74602b-geode-user-mock-smoke-20260731

• The create_task tool executed, but the model supplied an unrequested optional description="".
• Tau2's exact action and DB comparators rejected the extra argument; this is retained as a behavioral failure.
• This GEODE-owned user route is not comparable to the native tau2 user_simulator headline.

• DB match 31 / 114
• Write actions 471 / 496
• Generic actions 20 / 20
• Termination user_stop 114 / 114
• Duration total 28827.72s / avg 252.87s / max 818.58s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 4 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-telecom-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• Concurrency was raised from 2 to 4 mid-run and resumed from tau2 checkpoints; no rate-limit, quota, or billing errors were observed.
• Failures clustered around multi-issue MMS/mobile-data/service cases where one required APN, permission, roaming, or data-refuel action was omitted.

• DB match 88 / 113
• Read actions 320 / 354
• Write actions 140 / 174
• Termination user_stop 113 / 114, too_many_errors 1 / 114
• Duration total 23543.64s / avg 206.52s / max 873.92s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain retail \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-retail-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• The main failure mode was missing required side-effect actions even when the natural-language response looked plausible.
• One task terminated with too_many_errors; the remaining failures ended with user_stop but failed verifier assertions.

• DB match 42 / 50
• Read actions 81 / 91
• Write actions 33 / 49
• Termination user_stop 50 / 50
• Duration total 14205.02s / avg 284.10s / max 979.65s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain airline \
  --task-split-name base \
  --num-tasks 50 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-airline-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• This is the native tau2 user_simulator comparator track, not the GEODE geode_user smoke track.
• Airline is retained for internal trend comparison; OpenAI's GPT-5.2 announcement excludes Airline from its Tau2 headline due to lower-quality ground truth grading.

• DB check 1.0
• create_task action check 1.0
• Termination user_stop
• Duration 54.90s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.5 \
  --provider openai \
  --source subscription \
  --effort xhigh \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.5 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-5-xhigh-geode-user-mock-smoke-20260703-r5 \
  --log-level INFO \
  --verbose-logs

• This is a tau2 wiring/regression smoke, not a tau2 leaderboard score.
• Do not average it with native tau2 user_simulator runs using gpt-4.1 or gpt-5.2.
• Both assistant and simulated user used the GEODE subscription route.