MCPMark

• Filesystem standard: 25 / 30, 83.3%
• Postgres standard: 20 / 21, 95.2%
• GitHub standard: 19 / 23, 82.6%
• Recorded task execution time: filesystem 13580.6s over 29 recorded tasks, postgres 8765.7s, github 16476.3s
• Notion was not included: no notion_state.json in the local harness environment.
• Playwright/WebArena was not included: required Docker images/service stack were absent.

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp <filesystem|postgres|github> \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• This is not the full MCPMark Verified leaderboard aggregate. It covers only services that were runnable in the local environment: filesystem, postgres, and github.
• The OpenAI model route was the GEODE Codex subscription route, not MCPMark's native LiteLLM OpenAI API route.
• GitHub fixture repositories were made public during execution so the Docker GitHub MCP server could use normal public-repo semantics; all transient repos were deleted by cleanup.
• The filesystem score counts papers/author_folders as a failed no-result transport run after two attempts without meta output.

• Total task execution time 16476.3s
• Average task execution time 716.4s
• Failures: claude-code/label_color_standardization, mcpmark-cicd/deployment_status_workflow, missing-semester/assign_contributor_labels, missing-semester/find_salient_file
• All transient GitHub repositories were deleted by MCPMark cleanup.

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp github \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• The first label_color_standardization record is a fixture setup failure from GitHub state duplication; the retry produced an agent-level verification failure.
• The assign_contributor_labels failure used suffixed transient usernames in labels instead of canonical contributor labels.
• The find_salient_file failure did not create ANSWER.md on the required master branch.

• Total task execution time 8765.7s
• Average task execution time 417.4s
• Failure: employees/employee_performance_analysis

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp postgres \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• The GEODE adapter overrides MCPMark's default postgres server with postgres-mcp==0.3.0 in unrestricted mode.
• A final NoEventLoopError appeared during async cleanup after result writing; it did not affect the recorded verifier result.

• Recorded task execution time 13580.6s over 29 recorded tasks
• Average recorded task execution time 468.3s
• Failures: desktop_template/budget_computation, papers/author_folders, papers/find_math_paper, student_database/english_talent, threestudio/output_analysis

cd artifacts/eval/harnesses/mcpmark
# Run each available MCP service through the GEODE adapter.
GEODE_REPO_ROOT=<geode-worktree> \
PYTHONPATH=<geode-worktree>:<geode-site-packages> \
GITHUB_EVAL_ORG=mangowhoiscloud \
GEODE_MCPMARK_GITHUB_REPO_VISIBILITY=public \
.venv/bin/python pipeline.py \
  --mcp filesystem \
  --task-suite standard \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 1500 \
  --exp-name geode-gpt55-xhigh-20260704-mcpmark-verified-<service> \
  --output-dir ./results-geode-agentworld

• filesystem/standard is a materially harder slice than filesystem/easy.
• papers/author_folders is counted as a failed no-result transport run because both attempts hung before meta output.
• The adapter now aliases file_path to path when the MCP schema expects path, which fixed write_file failures seen in the first filesystem pass.

• Total task execution time 1360.129s
• Average task execution time 136.013s
• 40 GEODE rounds total / 4.0 average
• 429,324 total tokens
• Category rows: file_context 3/3, file_property 2/2, folder_structure 1/1, legal_document 1/1, papers 1/1, student_database 2/2

cd artifacts/eval/harnesses/mcpmark
for category in file_context file_property folder_structure legal_document papers student_database; do
  GEODE_REPO_ROOT=<geode-worktree> \
  OPENAI_API_KEY=dummy \
  FILESYSTEM_TEST_ROOT=./test_environments \
  .venv/bin/python pipeline.py \
    --mcp filesystem \
    --task-suite easy \
    --tasks "$category" \
    --models geode-gpt-5.5 \
    --agent geode \
    --reasoning-effort xhigh \
    --k 1 \
    --timeout 900 \
    --exp-name "geode-gpt55-xhigh-20260703-ledger-$category" \
    --output-dir ./results-geode-live &
done
wait

• This rerun split filesystem/easy by category and executed the six categories in parallel.
• Only filesystem was runnable in the current local environment without additional credentials or Docker services.
• GitHub, Notion, Playwright, and Postgres MCPMark columns remain blocked until their service prerequisites are provisioned.

• Total task execution time 1706.044s
• Average task execution time 170.604s
• 40 GEODE rounds total / 4.0 average
• 266,779 total tokens

cd artifacts/eval/harnesses/mcpmark
GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp filesystem \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh \
  --k 1 \
  --timeout 900 \
  --exp-name geode-gpt55-xhigh-20260703-filesystem-easy \
  --output-dir ./results-geode-live

• MCPMark filesystem/easy is directly comparable only to the same subset.
• This is not the MCPMark Verified aggregate used by frontier leaderboards.
• OPENAI_API_KEY=dummy satisfied the harness environment check; model calls used the GEODE subscription route.

• State duplication 58.9s; agent 216.8s over 8 rounds; 62.8k input / 8.0k output tokens.
• The 2026-07-04 stall was an expired browser session: duplication page.goto to app.notion.com timed out at 120s per retry.
• Re-login used a real-Chrome-channel persistent context (Google OAuth rejects automation-flagged browsers); the session cookie lives on .app.notion.com.

set -a; source .mcp_env; set +a
OPENAI_API_KEY=dummy \
.venv/bin/python -m plugins.benchmark_harness.run_mcpmark \
  --mcp notion \
  --task-suite easy \
  --tasks toronto_guide/simple__change_color \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Verifier-backed single-task smoke proving the notion service is runnable end to end; not a notion standard score.
• The task embeds a Notion API trap: updating a select option color returns validation_error; the agent passed by redefining options via a database schema update.

• No Notion MCPMark score was produced in this cycle.
• The harness requires source and evaluation Notion workspace credentials.

GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp notion \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Blocked before live execution because the local harness environment has no .mcp_env credentials.
• Record a measured score only after the Notion integration and paired workspaces are provisioned.

• No Playwright MCPMark score was produced in this cycle.
• Browser/WebArena service setup was not available in the local benchmark environment.

GEODE_REPO_ROOT=<geode-worktree> \
OPENAI_API_KEY=dummy \
.venv/bin/python pipeline.py \
  --mcp playwright \
  --task-suite easy \
  --models geode-gpt-5.5 \
  --agent geode \
  --reasoning-effort xhigh

• Blocked before live execution because the browser-backed service stack was not running.
• Record a measured score only after the browser environment is provisioned and health checked.