MCPMark

• Filesystem standard: 25 / 30, 83.3%
• Postgres standard: 20 / 21, 95.2%
• GitHub standard: 19 / 23, 82.6%
• Recorded task execution time: filesystem 13580.6s over 29 recorded tasks, postgres 8765.7s, github 16476.3s
• Notion was not included: no notion_state.json in the local harness environment.
• Playwright/WebArena was not included: required Docker images/service stack were absent.

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp <filesystem|postgres|github> \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• This is not the full MCPMark Verified leaderboard aggregate. It covers only services that were runnable in the local environment: filesystem, postgres, and github.
• The OpenAI model route was the GEODE Codex subscription route, not MCPMark's native LiteLLM OpenAI API route.
• GitHub fixture repositories were made public during execution so the Docker GitHub MCP server could use normal public-repo semantics; all transient repos were deleted by cleanup.
• The filesystem score counts papers/author_folders as a failed no-result transport run after two attempts without meta output.

• Paired outcomes: 17 both-pass / 3 both-fail / 6 GEODE-only / 4 Codex-only
• Exact token coverage: GEODE 29 / 30 attempts; Codex 30 / 30
• All-arm action wall: GEODE 8,463.252s vs Codex 5,484.322s; runner envelopes 8,536.382s vs 5,548.725s
• Native execution-log calls/errors: GEODE 644 / 51 vs Codex 678 / 17
• Normalized trajectory attempts: GEODE 645 including one recovery projection vs Codex 678
• Read/repeated-read references: GEODE 798 / 81 vs Codex 838 / 213
• One GEODE score-bearing deadline expiration; its token usage is null and its scope-incomplete trajectory is withheld

# Requires the pinned MCPMark checkout and verifier patch, frozen run spec,
# isolated runtime home, and working GPT-5.4 subscription authentication.
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark_pair   --profile filesystem30-geode-codex   --run-spec <frozen-run-spec.json>   --mcpmark-root artifacts/eval/harnesses/mcpmark   --output-dir <fresh-output-dir>   --python .venv/bin/python

• The prospective -10 percentage-point threshold was supported, but promotion_authority remains none.
• This is one direct paired repetition, not k=3 stability, a full MCPMark Verified headline, or an API-key leaderboard claim.
• The common action deadline excludes fixture setup and the post-action verifier; runtime scaffolds and tool-result policies remain different.
• Token totals have unmatched coverage and do not support a billing or token-efficiency claim.
• Fifty-nine scope-complete trajectories are admitted; the one GEODE timeout trajectory remains withheld.

• Observed fresh input: 25K 3,782,288 vs unlimited 2,202,725; exact token coverage is 13 / 15 attempts per arm
• All-attempt action wall: 25K 6,076.699s vs unlimited 4,910.217s; runner envelopes 6,119.096s vs 4,951.086s
• All-attempt MCP calls/errors: 25K 443 / 75 vs unlimited 255 / 27
• Repeated read references: 25K 211 vs unlimited 63; tool-result truncations 15 vs 0
• Two score-bearing deadline expirations per arm emitted no native token usage

# Requires the pinned MCPMark checkout and verifier patch, frozen run spec,
# isolated runtime home, and working GPT-5.4 subscription authentication.
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark_pair   --profile max-tool-result-tokens   --run-spec <frozen-run-spec.json>   --mcpmark-root artifacts/eval/harnesses/mcpmark   --output-dir <fresh-output-dir>   --python .venv/bin/python

• The prospective hypothesis was supported, but promotion_authority remains none.
• This five-task, three-repetition subset is diagnostic and is not an MCPMark suite headline.
• Token totals exclude the four timeout arms with empty native usage; wall time and MCP-call totals cover all 15 attempts per arm.
• The prior infrastructure-invalid run contributes no denominator, and scope-incomplete timeout trajectories remain withheld.

• Observed delta +1 / 30 (+3.33 pp); prospective hypothesis invalidated
• GEODE 1,646 events / 703 exact tool pairs; Codex 1,735 / 727
• GEODE fresh input 4,204,759 vs Codex 1,444,927; no token-efficiency claim
• Task wall: GEODE 7,842.4s vs Codex 6,970.2s (+12.5%); no causal efficiency claim
• Both releases scope-complete, replay-incomplete, zero orphan tool events

# Illustrative one-task invocation only; exact runner is withheld.
python -m plugins.benchmark_harness.run_mcpmark   --mcp filesystem   --task-suite standard   --tasks <one-frozen-task-id>   --models <geode-gpt-5.4|codex-gpt-5.4>   --agent <geode|codex>   --reasoning-effort high   --k 1   --timeout 1200

• All 30 pairs share the pinned task tree, fixture reset, model label, effort, and verifier.
• The frozen equal-hard-deadline claim was invalidated: GEODE timed loop.arun, while Codex timed process communication including internal MCP startup.
• Prompt, action budget, retry, compaction, and cache accounting are also not identical; outcomes are retrospective descriptions only.
• The public bundle is not independently executable because the exact runner remains digest-bound but withheld.
• This is one filesystem service slice, not the full 127-task MCPMark Verified leaderboard.
• Raw messages, logs, metadata, and provider diagnostics remain private; only digest-reduced trajectories and validated sidecars are public.

• Matched input tokens 447,376 → 314,219 (-29.8%)
• Matched output tokens 25,157 → 20,385 (-19.0%)
• Native reasoning tokens 14,174, included within output tokens
• 188 canonical events / 54 exactly paired tool calls and results
• Failure unchanged: file_context/uppercase exact-string mismatch

cd artifacts/eval/harnesses/mcpmark
PYTHONPATH=<geode-feature-tree> .venv/bin/python -m plugins.benchmark_harness.run_mcpmark   --mcp filesystem   --task-suite easy   --models geode-gpt-5.4   --agent geode   --reasoning-effort high   --k 1   --timeout 1200   --exp-name geode-gpt54-high-token-efficiency-20260812-rerun   --output-dir ./results-token-efficiency

• The score matched the pre-repair GEODE baseline at 9/10 while input and output tokens fell materially.
• Eight of ten tasks used fewer input tokens; the four tasks with identical round counts fell 12.5%.
• This is one matched diagnostic trial, not MCPMark Verified, a confidence interval, or a subscription billing claim.
• All ten public trajectories are scope-complete and intentionally replay-incomplete; the immutable release was read back from artifact main.

• Total task execution time 802.182s / average 80.218s
• 53 GEODE turns total / 5.3 average
• 302,984 input / 30,238 output tokens
• 182 canonical events / 56 exactly paired tool calls and results
• Failure: file_context/uppercase left file_01.txt incompletely uppercased

cd artifacts/eval/harnesses/mcpmark
PYTHONPATH=<geode-v1.0.12-release-tree> .venv/bin/python -m plugins.benchmark_harness.run_mcpmark   --mcp filesystem   --task-suite easy   --models geode-gpt-5.4   --agent geode   --reasoning-effort high   --k 1   --timeout 1200   --exp-name geode-gpt54-high-v1.0.12-f99cea63-20260803-mcpmark-filesystem-easy   --output-dir ./results-geode-v1012

• The official verifier found all five output files, but file_01.txt was not fully uppercased; the failure is retained without retry.
• No authentication, quota, provider-adapter, MCP transport, or harness exception occurred.
• The v1.0.11 GPT-5.6 10/10 comparison is model-confounded and cannot be attributed to the runtime release alone.
• All ten trajectories are scope-complete and intentionally replay-incomplete; manifest and native receipts are pinned to artifact commit 04ff1c4.

• Total task execution time 596.580s / average 59.658s
• 56 GEODE turns total / 5.6 average
• 700,719 input / 12,164 output / 206,848 cache-read tokens
• Recorded estimate $2.937699; not subscription billing
• 226 canonical events / 78 exactly paired tool calls and results / 0 missing required turn IDs

cd artifacts/eval/harnesses/mcpmark
GEODE_HOME=<isolated-v1011-runtime-home> \
PYTHONPATH=<geode-v1.0.11-release-tree> \
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark \
  --mcp filesystem \
  --task-suite easy \
  --models gpt-5.6-sol \
  --agent geode \
  --reasoning-effort high \
  --k 1 \
  --timeout 1200 \
  --exp-name geode-gpt56-sol-high-v1011-686ff372-20260731-mcpmark-filesystem-easy \
  --output-dir ./results-geode-v1011

• The earlier file_context/uppercase failure now passes; all ten official filesystem/easy verifiers are green.
• This remains directly comparable only to filesystem/easy, not to the MCPMark Verified standard aggregate.
• The stable geode.trajectory@1 release is scope-complete but intentionally replay-incomplete because dialogue and tool bodies are digested.
• Native receipts and stable trajectories are pinned to geode-eval-artifacts commit 16a54f0.

• Total task execution time 799.435s / average 79.943s
• 54 GEODE turns total / 5.4 average
• 799,679 input / 10,976 output / 97,792 cache-read tokens
• Recorded estimate $3.887611; not subscription billing
• Failure: file_context/uppercase left file_01.txt incompletely uppercased

cd artifacts/eval/harnesses/mcpmark
GEODE_HOME=<isolated-runtime-home> \
PYTHONPATH=<geode-edb74602b-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark \
  --mcp filesystem \
  --task-suite easy \
  --models geode-gpt-5.6-sol \
  --agent geode \
  --reasoning-effort high \
  --k 1 \
  --timeout 1200 \
  --exp-name geode-gpt56-sol-high-edb74602b-20260731-mcpmark-filesystem-easy \
  --output-dir ./results-geode-edb74602b

• This is directly comparable to filesystem/easy only, not to the MCPMark Verified standard aggregate.
• The upstream total_tokens and total_reasoning_tokens summary fields were zero despite populated input/output fields; they are not used.
• One response stream disconnected after the first task had already produced its files; that task passed every official integrity check and no 429 occurred.
• Raw receipts and ten normalized tool trajectories are pinned to geode-eval-artifacts commit 9c00ecf.

• Total task execution time 16476.3s
• Average task execution time 716.4s
• Failures: claude-code/label_color_standardization, mcpmark-cicd/deployment_status_workflow, missing-semester/assign_contributor_labels, missing-semester/find_salient_file
• All transient GitHub repositories were deleted by MCPMark cleanup.

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp github \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• The first label_color_standardization record is a fixture setup failure from GitHub state duplication; the retry produced an agent-level verification failure.
• The assign_contributor_labels failure used suffixed transient usernames in labels instead of canonical contributor labels.
• The find_salient_file failure did not create ANSWER.md on the required master branch.

• Total task execution time 8765.7s
• Average task execution time 417.4s
• Failure: employees/employee_performance_analysis

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp postgres \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• The GEODE adapter overrides MCPMark's default postgres server with postgres-mcp==0.3.0 in unrestricted mode.
• A final NoEventLoopError appeared during async cleanup after result writing; it did not affect the recorded verifier result.

• Recorded task execution time 13580.6s over 29 recorded tasks
• Average recorded task execution time 468.3s
• Failures: desktop_template/budget_computation, papers/author_folders, papers/find_math_paper, student_database/english_talent, threestudio/output_analysis

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp filesystem \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• filesystem/standard is a materially harder slice than filesystem/easy.
• papers/author_folders is counted as a failed no-result transport run because both attempts hung before meta output.
• The adapter now aliases file_path to path when the MCP schema expects path, which fixed write_file failures seen in the first filesystem pass.

• Total task execution time 1360.129s
• Average task execution time 136.013s
• 40 GEODE rounds total / 4.0 average
• 429,324 total tokens
• Category rows: file_context 3/3, file_property 2/2, folder_structure 1/1, legal_document 1/1, papers 1/1, student_database 2/2

cd artifacts/eval/harnesses/mcpmark
for category in file_context file_property folder_structure legal_document papers student_database; do
  GEODE_REPO_ROOT=<geode-worktree> \
  OPENAI_API_KEY=dummy \
  FILESYSTEM_TEST_ROOT=./test_environments \
  .venv/bin/python pipeline.py \
    --mcp filesystem \
    --task-suite easy \
    --tasks "$category" \
    --models geode-gpt-5.5 \
    --agent geode \
    --reasoning-effort xhigh \
    --k 1 \
    --timeout 900 \
    --exp-name "geode-gpt55-xhigh-20260703-ledger-$category" \
    --output-dir ./results-geode-live &
done
wait

• This rerun split filesystem/easy by category and executed the six categories in parallel.
• Only filesystem was runnable in the current local environment without additional credentials or Docker services.
• GitHub, Notion, Playwright, and Postgres MCPMark columns remain blocked until their service prerequisites are provisioned.

• Total task execution time 1706.044s
• Average task execution time 170.604s
• 40 GEODE rounds total / 4.0 average
• 266,779 total tokens

cd artifacts/eval/harnesses/mcpmark
GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp filesystem \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 900 \
  --exp-name geode-gpt55-xhigh-20260703-filesystem-easy \
  --output-dir ./results-geode-live

• MCPMark filesystem/easy is directly comparable only to the same subset.
• This is not the MCPMark Verified aggregate used by frontier leaderboards.
• OPENAI_API_KEY=dummy satisfied the harness environment check; model calls used the GEODE subscription route.

• State duplication 58.9s; agent 216.8s over 8 rounds; 62.8k input / 8.0k output tokens.
• The 2026-07-04 stall was an expired browser session: duplication page.goto to app.notion.com timed out at 120s per retry.
• Re-login used a real-Chrome-channel persistent context (Google OAuth rejects automation-flagged browsers); the session cookie lives on .app.notion.com.

set -a; source .mcp_env; set +a
OPENAI_API_KEY=dummy \
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark \
  --mcp notion \
  --task-suite easy \
  --tasks toronto_guide/simple__change_color \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Verifier-backed single-task smoke proving the notion service is runnable end to end; not a notion standard score.
• The task embeds a Notion API trap: updating a select option color returns validation_error; the agent passed by redefining options via a database schema update.

• No Notion MCPMark score was produced in this cycle.
• The harness requires source and evaluation Notion workspace credentials.

GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp notion \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Blocked before live execution because the local harness environment has no .mcp_env credentials.
• Record a measured score only after the Notion integration and paired workspaces are provisioned.

• No Playwright MCPMark score was produced in this cycle.
• Browser/WebArena service setup was not available in the local benchmark environment.

GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp playwright \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Blocked before live execution because the browser-backed service stack was not running.
• Record a measured score only after the browser environment is provisioned and health checked.