feat(plugin): #2156 GAIA benchmark component in ruflo-workflows v0.3.0#2182
Open
ruvnet wants to merge 13 commits into
Open
feat(plugin): #2156 GAIA benchmark component in ruflo-workflows v0.3.0#2182ruvnet wants to merge 13 commits into
ruvnet wants to merge 13 commits into
Conversation
…solution Adds the first concrete deliverable toward ADR-133 (Real GAIA Capability Benchmark): a typed dataset loader with: - `resolveHfToken()` — mirrors the ANTHROPIC_API_KEY pattern from performance-capability.ts (env var first, gcloud fallback using the actual GCP secret name `huggingface-token`) - `loadGaia(options)` — public API returning GaiaQuestion[] with level/limit/smokeOnly/cacheDir knobs - `SMOKE_FIXTURE` — 5 offline questions for CI-without-HF testing (all 5 answer keys manually verified via `node -e`) - HF Datasets Server paginated rows endpoint (PR-1 skeleton: 100 rows, full pagination tracked for PR-3) HF_TOKEN is confirmed available in GCP Secret Manager as `huggingface-token` (36-char token). The `resolveHfToken()` function resolves this correctly. Next: gaia-tools/ (PR-2) and gaia-agent.ts multi-turn loop (PR-3). Co-Authored-By: RuFlo <ruv@ruv.net>
…le_read + types)
Adds the second slice of ADR-133 (Real GAIA Capability Benchmark):
the gaia-tools/ subsystem that the agent loop (PR-3) will consume.
New files:
- gaia-tools/types.ts — shared Anthropic tool_use / tool_result spec types
(ToolDefinition, ToolUseBlock, ToolResultBlock,
GaiaTool interface, GaiaToolCatalogue)
- gaia-tools/web_search.ts — DuckDuckGo HTML scrape (POST /html/, no API key);
regex-based title/URL/snippet extraction;
single-redirect follow; 20s timeout
- gaia-tools/file_read.ts — local fs reader with extension + magic-byte
content-type detection; 1 MB size cap; absolute-path
validation; binary stub for PDF/images (PR-4 concern)
- gaia-tools/index.ts — barrel + createDefaultToolCatalogue() factory
All logic verified via node -e before commit:
PASS: stripHtml (3 cases)
PASS: validatePath (4 cases — empty, relative, null-byte, valid absolute)
PASS: hasBinaryMagic (5 cases — PDF, PNG, JPEG, text, empty)
PASS: decodeRawUrl (DDG redirect URL + direct URL passthrough)
TypeScript: tsc --noEmit --skipLibCheck — zero errors.
Next: gaia-agent.ts multi-turn loop (PR-3) that imports createDefaultToolCatalogue()
and drives a Claude Messages API call-loop until final_answer is extracted.
Refs: ADR-133, #2156
Co-Authored-By: RuFlo <ruv@ruv.net>
Implements the SR²AM-style simulative planning layer from ADR-132: selective depth-allocation that fires a Haiku shadow pass before Tier-3 (Sonnet/Opus) dispatch when tasks exceed the horizon/MCP gate. New files: - v3/@claude-flow/hooks/src/route/simulative-planning-router.ts Gate logic (estimatedHorizon > 5 OR predictedMcpCalls >= 2), buildShadowPrompt, parseShadowResponse (fence-stripping + fallback), maybeSimulatePlan (injected HaikuClient + SonaCache collaborators). Target: <=30 ms overhead, 256-token Haiku pass, 300 s SONA TTL. - v3/@claude-flow/hooks/src/route/index.ts Barrel export for the new route/ submodule. - v3/@claude-flow/hooks/__tests__/simulative-planning-router.test.ts 25 vitest unit tests (all green): gate boundary conditions, prompt builder, JSON parser (happy path / code-fence stripping / malformed fallback), maybeSimulatePlan integration with mock collaborators, SONA cache-write failure resilience. Does NOT open PR — gated on ADR-132 doc PR #2157 merging first (per iter-1 decision D1). Co-Authored-By: RuFlo <ruv@ruv.net>
…it tests" This reverts commit 26a74c0.
… loop Implements the GAIA agent harness: multi-turn Anthropic Messages API loop with tool dispatch (web_search + file_read), final-answer extraction via FINAL_ANSWER: pattern, and a smoke runner against the 5-question fixture. - runGaiaAgent(): resolves API key (env → gcloud), drives Claude through up to 8 turns with parallel tool execution, returns GaiaAgentResult - resolveAnthropicApiKey(): mirrors resolveHfToken pattern from PR-1 - isAnswerCorrect(): substring + numeric normalisation, mirrors GAIA eval - runSmokeTest(): runs SMOKE_FIXTURE[5], reports pass/fail + cost estimate - CLI entrypoint: `node gaia-agent.js --smoke` (exit 0 if ≥3/5 pass) - Zero TypeScript errors (moduleResolution: bundler, ESNext target) - 575 lines; smoke live-run deferred until ANTHROPIC_API_KEY available Refs: ADR-133, #2156 Co-Authored-By: RuFlo <ruv@ruv.net>
…scorer Adds `judgeAnswer()` with normalised exact-match fast-path (no API call) and Claude Sonnet LLM-as-judge fallback for semantic equivalence. Results cached by (questionId, candidate, model, prompt_version) tuple under ~/.cache/ruflo/gaia/judgments/ to avoid re-judging on re-runs. Smoke: 11 assertions (6 normaliseAnswer unit + 5 exact-match path), all pass without ANTHROPIC_API_KEY; LLM cases auto-skip when key absent. Co-Authored-By: RuFlo <ruv@ruv.net>
…to-end Wires runGaiaAgent (Haiku) + judgeAnswer (exact-match → Sonnet) into a 5-question end-to-end pipeline. Reports pass rate, mean turns, and cost breakdown. Asserts ≥3/5 pass. Requires ANTHROPIC_API_KEY at runtime. Expected cost: ~$0.02 for 5 smoke questions. Co-Authored-By: RuFlo <ruv@ruv.net>
The gcloud fallback in resolveAnthropicApiKey (gaia-agent.ts) and resolveApiKey (gaia-judge.ts) was calling secret name "anthropic-api-key" (lowercase) which does not exist in GCP Secret Manager. The secret is stored as "ANTHROPIC_API_KEY" (uppercase). Verified: live e2e smoke now resolves the key and runs 5/5 PASS. Co-Authored-By: RuFlo <ruv@ruv.net>
…orkflow contract - New command: gaia-bench run --level --limit --models --output --concurrency - Lazy-imports gaia-loader/agent/judge from dist/src/benchmarks/ (no src TS files) - JSON output shape matches gaia-benchmark.yml workflow expectations exactly - --smoke-only flag uses 5-question fixture (no HF token required) - Smoke test: 5/5 pass, $0.0016, 1.2 mean turns — CLI end-to-end validated - TypeScript clean (zero errors) Real Level-1 blocked: HF token not yet gate-approved for gaia-benchmark/GAIA. Gate approval required at: https://huggingface.co/datasets/gaia-benchmark/GAIA Co-Authored-By: RuFlo <ruv@ruv.net>
…all L1 questions
Using config=2023_all with length=100 silently dropped 23 of 53 Level-1
questions because the HF API caps responses at 100 rows and L1 questions
are not all in the first 100 rows of the combined dataset.
Switch to config=2023_level{N} which returns all questions for each level
in a single request (L1=53, L2=86, L3=26 — all within the 100-row cap).
Verified live against datasets-server.huggingface.co: all three configs
return the correct num_rows_total matching the Princeton-HAL GAIA paper.
Co-Authored-By: RuFlo <ruv@ruv.net>
Recovers GAIA questions where models return the raw number and expected
is in scaled units (e.g. question asks 'how many thousand hours', model
returns 17000, expected is 17). Stage 1 now tries three comparison
strategies before falling through to LLM-as-judge:
1. Normalised exact-match (existing)
2. Unit-aware scaling: infer multiplier from question text and try
both raw→scaled and scaled→raw directions
3. LLM-as-judge (existing fallback)
Added: unitAwareNumberMatch() — exported for testing.
Fixed: buildJudgeUserMessage() called with question.expected as the
question text arg (copy-paste bug); now passes the real question text.
Updated: judgeAnswer() signature adds optional questionText field
(backward-compatible — all existing callers still type-check).
Updated: gaia-bench.ts + gaia-e2e-smoke.ts pass q.question as
questionText so the unit-aware path activates on real runs.
Validated: 7/7 unit tests pass including Kipchoge e1fc63a2 (17000 vs 17,
question "thousand hours") and correct rejections of false positives.
Refs #2156 ADR-133 iter-15
Co-Authored-By: RuFlo <ruv@ruv.net>
…location Per swarm research (ADR-136 rank 1). Predicts question difficulty from 17 features (embedding NN distance proxy, syntactic, lexical, tool implication) and routes to appropriate compute budget: - easy -> Haiku, 4 turns, 1 attempt - medium -> Sonnet, 8 turns, 1 attempt - hard -> Sonnet, 12 turns, 3-vote (ADR-135 Track A) New files: src/benchmarks/gaia-hardness/features.ts -- 17-dim feature extraction src/benchmarks/gaia-hardness/predictor.ts -- HardnessPredictor (logistic regression, no deps) src/benchmarks/gaia-hardness/train-data-loader.ts -- loads iter-15/23/28 result JSONs src/benchmarks/gaia-hardness/predictor.smoke.ts -- 8/8 smoke tests pass, $0 cost gaia-bench.ts: adds --hardness-routing (opt-in, default off) + --hardness-verbose; overrides model/maxTurns/votingAttempts per question based on HardnessPredictor.predict(); reports hardnessDist in JSON output summary. Cold-start: classifies as medium when untrained (<10 labeled examples). Training: loads historical result JSONs from /tmp/gaia-l1-full.json etc. Standalone lift estimate: +2-4pp. Multiplier on Track A (3-vote only fires on hard questions -> ~75% cost reduction on ensemble runs). TS: 0 new errors. Smoke: 8/8 pass. Refs ADR-136, ADR-135, #2156 Co-Authored-By: RuFlo <ruv@ruv.net>
…ommands, 3 skills, 2 agents Adds a submission-ready GAIA benchmark component to the ruflo-workflows plugin (v0.3.0) that wires the existing gaia-bench CLI backend into user-facing Claude Code slash commands, skills, and agent personas. New artifacts (14 files): - commands/gaia.md — dispatcher for all /gaia subcommands - commands/gaia-run.md — execute benchmark (shells to gaia-bench run) - commands/gaia-submit.md — build Ed25519-signed HAL-compatible package - commands/gaia-leaderboard.md — fetch HAL scores + compare to local runs - commands/gaia-validate.md — pre-submit env/TS/dataset checks - commands/gaia-history.md — tabular view of gaia-runs namespace - commands/gaia-cost.md — cumulative spend + run projection (cost gate at $5) - skills/gaia-submission/SKILL.md — full benchmark→submit walkthrough - skills/gaia-debugging/SKILL.md — failure-mode taxonomy + trace extraction - skills/gaia-architecture-comparison/SKILL.md — ruflo vs HAL gap analysis - agents/gaia-benchmark-runner.md — run/monitor/diagnose agent persona - agents/gaia-submission-coordinator.md — package/sign/submit agent persona - scripts/smoke-gaia.sh — 14-check smoke test (14/14 pass) - .claude-plugin/plugin.json bumped to 0.3.0 with gaia component block Baselines: iter 23 = 20.8% L1 (53 Q), HAL ref = 74.6% (300 Q, Sonnet 4.5). Co-Authored-By: RuFlo <ruv@ruv.net>
This was referenced May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ruflo-workflowsplugin (v0.2.0 → v0.3.0)gaia-benchCLI backend from PR feat(benchmarks): ADR-133 — GAIA loader + tools + agent loop + judge (PR1+PR2+PR3+PR6) #2165; no benchmark logic is re-implemented hereFiles created
commands/gaia.mdcommands/gaia-run.mdcommands/gaia-submit.mdcommands/gaia-leaderboard.mdcommands/gaia-validate.mdcommands/gaia-history.mdcommands/gaia-cost.mdskills/gaia-submission/SKILL.mdskills/gaia-debugging/SKILL.mdskills/gaia-architecture-comparison/SKILL.mdagents/gaia-benchmark-runner.mdagents/gaia-submission-coordinator.mdscripts/smoke-gaia.sh.claude-plugin/plugin.jsonBehavioral requirements met
gaia-run.mdandgaia-submissionskillgaia-submit.mdandgaia-submission-coordinator.mdtask_id,model_answer,tools_used,turns,wall_seconds)gaia-submissionskillgaia-run.mdgaia-benchmark-runneragentgaia-runsused consistently across run, history, and cost commandsBaselines for context
Test plan
bash plugins/ruflo-workflows/scripts/smoke-gaia.sh→ 14/14 pass/gaia validatewith realANTHROPIC_API_KEY+HF_TOKEN/gaia run --smoke-only(5 Q, no HF token required)/gaia submiton a results file from a prior run/gaia historyafter storing at least one run record🤖 Generated with RuFlo