{ "hardware": { "chip": "Apple M5 Max", "memory_gib": 128 }, "api_cost_yen": 0, "method": "6 tasks x 8 models, plus 2 Sente read-only tasks x 8 models; isolated local inference. temperature=0, seed=42, max_tokens=4096 per turn in direct comparison. Native model reasoning defaults except Bonsai, which uses dedicated Prism ML runtime and medium reasoning, server context 65536. One quality trial per task, not a general model ranking. Warmup before quality tasks. Sente uses its own default sampling, 210-second deadline per task. Strict JSON checks include format/type. Personal provider labels/shortcuts were updated separately after benchmarking.", "caveats": [ "Laguna HTTP 500 failures are integration failures, not scored as wrong factual answers.", "Nemotron latest_active and Laguna S merge_ranges hit the 4096-token output limit before a final answer.", "Qwen3.5 122B both code tasks hit 4096 tokens without a final answer. Both direct tool tasks returned correct values inside Markdown fences; Sente tool_read failed for the same formatting reason, while tool_reconcile passed.", "Sente Qwen3 Coder tool_read was factually correct but boolean encoded as a string; tool_reconcile had a wrong amount.", "Sente Nemotron and Laguna S reconciliation reasoning gave correct amounts but did not deliver the requested JSON.", "Sente Laguna S tool_read had correct values inside Markdown fences, failing strict JSON only.", "Task time medians include failed tasks, so compare Qwen3.8 vs Muse (both 6/6) for meaningful full-success timing.", "Native /api/chat pilot is excluded. v2 uses the same OpenAI-compatible endpoint as Sente.", "Warmup timings are not controlled cold-start/load-only measurements and are not used for ranking.", "Bonsai PQ2_0 was downloaded with user approval, official SHA-256 verified. Dedicated Prism ML runtime, not Ollama; merge_ranges exhausted 4096 tokens with no final answer. Runtime and reasoning settings differ.", "Bonsai Sente reconciliation mistyped the bank.json directory, hit the external-directory permission rule repeatedly, then exited 4 at the loop guard without a final answer. This is task execution failure, not a scored arithmetic error.", "The Qwen3.8 UI capture run overlapped the end of the Bonsai Sente reconciliation trial; that timing is not an isolated performance measurement.", "The separate Qwen3.8 UI demo returned correct values with extra prose, failing strict JSON. It uses a different prompt and is not part of the fixed benchmark score.", "Output/context/reasoning limits are not normalized for each model capability; these are bounded practical tasks, not maximum-capability tests.", "status.next_action_ja is automatically checked only for type/length; semantic correctness is not fully graded.", "Model names are local Ollama tags; results apply to the installed artifacts, not a verified upstream lineage." ], "unmeasured": [], "recommendation": { "first_try": "qwen3.8:27b", "coding": "qwen3.8:27b", "second_answer": "muse-glimmer:30b", "fast_draft_with_tests": "qwen3-coder:30b" }, "rows": [ { "model": "laguna-xs-2.1:q4_K_M", "direct_passed": 1, "direct_total": 6, "task_median_seconds": 0.77, "backend": "Ollama 0.35.0", "sente_strict_passed": 0, "sente_total": 2, "sente_seconds": [ 73.962, 70.524 ], "code_cases_passed": 47, "code_cases_available": 94, "tasks": [ { "pass": false, "checks": { "JPY": false, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 7200, "USD": 2000, "refund_c_pending": true }, "model": "laguna-xs-2.1:q4_K_M", "task": "ledger", "seconds": 0.373, "load_seconds": 0.0, "output_tokens": 29, "thinking_chars": 0, "calls": [] }, { "pass": false, "checks": { "refund_stage": false, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "requested", "refund_yen": 0, "deployed": false, "next_action_ja": "カード会社へ確認し、返金処理と本番修正の状況を確認" }, "model": "laguna-xs-2.1:q4_K_M", "task": "status", "seconds": 0.543, "load_seconds": 0.0, "output_tokens": 49, "thinking_chars": 0, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "laguna-xs-2.1:q4_K_M", "task": "merge_ranges", "seconds": 1.486, "load_seconds": 0.0, "output_tokens": 170, "thinking_chars": 0, "calls": [] }, { "pass": false, "code_tests": { "passed": 0, "total": 47, "error": "SyntaxError('invalid syntax', ('', 4, 26, \" eid = event['id']n ts = event['ts']\\n\", 4, 27))" }, "model": "laguna-xs-2.1:q4_K_M", "task": "latest_active", "seconds": 1.113, "load_seconds": 0.0, "output_tokens": 121, "thinking_chars": 0, "calls": [] }, { "pass": false, "error": "RuntimeError('HTTP 500: {\"error\":{\"message\":\"empty Laguna tool call name\",\"type\":\"api_error\",\"param\":null,\"code\":null}}\\n')", "model": "laguna-xs-2.1:q4_K_M", "task": "tool_read", "seconds": 0.556, "load_seconds": 0.0, "output_tokens": 25, "thinking_chars": 0, "calls": [ "release-note.txt" ] }, { "pass": false, "error": "RuntimeError('HTTP 500: {\"error\":{\"message\":\"empty Laguna tool call name\",\"type\":\"api_error\",\"param\":null,\"code\":null}}\\n')", "model": "laguna-xs-2.1:q4_K_M", "task": "tool_reconcile", "seconds": 0.983, "load_seconds": 0.0, "output_tokens": 48, "thinking_chars": 0, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "qwen3-coder:30b", "direct_passed": 4, "direct_total": 6, "task_median_seconds": 0.74, "backend": "Ollama 0.35.0", "sente_strict_passed": 0, "sente_total": 2, "sente_seconds": [ 2.703, 2.608 ], "code_cases_passed": 76, "code_cases_available": 94, "tasks": [ { "pass": false, "checks": { "JPY": false, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 12800, "USD": 2000, "refund_c_pending": true }, "model": "qwen3-coder:30b", "task": "ledger", "seconds": 0.382, "load_seconds": 0.0, "output_tokens": 33, "thinking_chars": 0, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "カード明細に返金反映を確認" }, "model": "qwen3-coder:30b", "task": "status", "seconds": 0.472, "load_seconds": 0.0, "output_tokens": 46, "thinking_chars": 0, "calls": [] }, { "pass": false, "code_tests": { "passed": 29, "total": 47, "failures": [ { "case": 2, "pass": false, "input": [ [ 5, 2 ], [ 1, 2 ], [ 3, 8 ], [ 10, 10 ] ], "expected": [ [ 1, 8 ], [ 10, 10 ] ], "actual": [ [ 1, 2 ], [ 3, 8 ], [ 10, 10 ] ], "mutated": false }, { "case": 7, "pass": false, "input": [ [ 5, 20 ], [ -6, -1 ], [ 20, 12 ], [ 6, -19 ], [ -17, 16 ], [ -3, 9 ], [ 17, 7 ], [ -19, -18 ], [ 11, -15 ], [ -9, 19 ], [ 14, 2 ], [ -16, 4 ], [ 0, -16 ], [ 5, -18 ] ], "expected": [ [ -19, 20 ] ], "actual": [ [ -19, -18 ], [ -17, 20 ] ], "mutated": false }, { "case": 12, "pass": false, "input": [ [ 5, -13 ], [ -16, -13 ], [ -4, 13 ], [ 17, -14 ], [ 6, 11 ], [ -9, 3 ] ], "expected": [ [ -16, 17 ] ], "actual": [ [ -16, -13 ], [ -9, 17 ] ], "mutated": false }, { "case": 16, "pass": false, "input": [ [ -13, -16 ], [ 10, -20 ], [ 10, 3 ], [ 2, -10 ] ], "expected": [ [ -20, 10 ] ], "actual": [ [ -16, -13 ], [ -10, 10 ] ], "mutated": false }, { "case": 18, "pass": false, "input": [ [ 3, 19 ], [ -13, -12 ], [ 16, -12 ], [ 12, 3 ], [ 12, 5 ] ], "expected": [ [ -13, 19 ] ], "actual": [ [ -13, -12 ], [ 3, 19 ] ], "mutated": false }, { "case": 19, "pass": false, "input": [ [ -11, -6 ], [ -13, 20 ], [ 12, -6 ], [ 1, 6 ], [ 0, -5 ], [ 1, -20 ], [ 4, 7 ], [ -14, -1 ] ], "expected": [ [ -20, 20 ] ], "actual": [ [ -14, 20 ] ], "mutated": false }, { "case": 25, "pass": false, "input": [ [ 10, 2 ], [ -14, 19 ], [ 19, 8 ], [ -8, -20 ], [ 18, 18 ], [ -17, -12 ], [ 19, -8 ], [ -4, -9 ], [ 17, 20 ], [ -14, 14 ], [ 18, 3 ], [ 17, -2 ], [ 12, -15 ], [ -8, 0 ], [ 13, -2 ] ], "expected": [ [ -20, 20 ] ], "actual": [ [ -17, 20 ] ], "mutated": false }, { "case": 26, "pass": false, "input": [ [ -1, -20 ], [ -10, 18 ], [ 6, 10 ], [ -1, 15 ] ], "expected": [ [ -20, 18 ] ], "actual": [ [ -10, 18 ] ], "mutated": false }, { "case": 27, "pass": false, "input": [ [ 11, -14 ], [ -9, 16 ] ], "expected": [ [ -14, 16 ] ], "actual": [ [ -9, 16 ] ], "mutated": false }, { "case": 28, "pass": false, "input": [ [ -8, -13 ], [ 18, 15 ], [ -16, -4 ], [ -16, -14 ], [ -1, 8 ], [ 6, 16 ], [ -16, -15 ], [ 16, -10 ], [ -18, 17 ], [ 16, 14 ], [ 9, -17 ], [ -12, 19 ], [ 17, 19 ], [ 10, -20 ] ], "expected": [ [ -20, 19 ] ], "actual": [ [ -18, 19 ] ], "mutated": false }, { "case": 31, "pass": false, "input": [ [ -17, -19 ], [ 9, 14 ], [ -12, 12 ], [ 10, -4 ], [ 12, -5 ], [ 20, -16 ], [ -5, -13 ], [ 0, 0 ], [ 5, 0 ], [ 19, 4 ], [ -3, -14 ], [ -16, -15 ], [ 11, -19 ], [ -10, -4 ], [ -14, 20 ] ], "expected": [ [ -19, 20 ] ], "actual": [ [ -19, -17 ], [ -16, -15 ], [ -14, 20 ] ], "mutated": false }, { "case": 32, "pass": false, "input": [ [ 12, 14 ], [ 17, -16 ], [ 19, 4 ], [ 19, -15 ], [ 6, 8 ] ], "expected": [ [ -16, 19 ] ], "actual": [ [ 6, 8 ], [ 12, 19 ] ], "mutated": false }, { "case": 34, "pass": false, "input": [ [ 16, -4 ], [ 16, 4 ], [ 20, 16 ], [ 15, 8 ], [ -8, -8 ], [ 7, -11 ], [ -10, 17 ], [ 5, -20 ] ], "expected": [ [ -20, 20 ] ], "actual": [ [ -10, 20 ] ], "mutated": false }, { "case": 35, "pass": false, "input": [ [ 14, -13 ], [ 0, 6 ], [ 6, 12 ], [ 15, 11 ], [ -3, 4 ], [ 20, 8 ], [ 12, 8 ], [ -6, 6 ], [ 18, 20 ] ], "expected": [ [ -13, 20 ] ], "actual": [ [ -6, 15 ], [ 18, 20 ] ], "mutated": false }, { "case": 37, "pass": false, "input": [ [ -4, -5 ], [ 6, -16 ], [ -17, -12 ], [ -6, -17 ], [ -14, 2 ], [ 18, -3 ], [ 17, -1 ], [ 19, -11 ], [ 8, 9 ], [ -2, -15 ], [ 20, 6 ], [ -8, -2 ] ], "expected": [ [ -17, 20 ] ], "actual": [ [ -17, 6 ], [ 8, 20 ] ], "mutated": false }, { "case": 38, "pass": false, "input": [ [ -8, -16 ], [ -7, 9 ], [ 5, 4 ], [ 3, -20 ], [ 16, 6 ], [ -7, 12 ], [ -1, -17 ], [ -14, -6 ], [ -9, 6 ], [ -2, -3 ] ], "expected": [ [ -20, 16 ] ], "actual": [ [ -14, 16 ] ], "mutated": false }, { "case": 44, "pass": false, "input": [ [ 17, 11 ], [ 16, 12 ], [ -3, 4 ], [ 13, -4 ], [ -17, -18 ], [ -10, -5 ], [ -3, 0 ] ], "expected": [ [ -18, -17 ], [ -10, -5 ], [ -4, 17 ] ], "actual": [ [ -18, -17 ], [ -10, -5 ], [ -3, 17 ] ], "mutated": false }, { "case": 45, "pass": false, "input": [ [ -16, 3 ], [ -18, 4 ], [ 11, 7 ], [ -12, -8 ], [ 20, -11 ], [ -1, 1 ], [ 9, 9 ], [ 14, 6 ] ], "expected": [ [ -18, 20 ] ], "actual": [ [ -18, 4 ], [ 9, 20 ] ], "mutated": false } ] }, "model": "qwen3-coder:30b", "task": "merge_ranges", "seconds": 2.17, "load_seconds": 0.0, "output_tokens": 261, "thinking_chars": 0, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "qwen3-coder:30b", "task": "latest_active", "seconds": 2.186, "load_seconds": 0.0, "output_tokens": 250, "thinking_chars": 0, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "qwen3-coder:30b", "task": "tool_read", "seconds": 0.577, "load_seconds": 0.0, "output_tokens": 42, "thinking_chars": 0, "calls": [ "release-note.txt" ] }, { "pass": true, "checks": { "invoice_id": true, "settled_yen": true, "outstanding_yen": true, "fully_paid": true, "actually_read": true }, "answer": { "invoice_id": "INV-307", "settled_yen": 14000, "outstanding_yen": 4400, "fully_paid": false }, "model": "qwen3-coder:30b", "task": "tool_reconcile", "seconds": 0.902, "load_seconds": 0.0, "output_tokens": 81, "thinking_chars": 0, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "qwen3.8:27b", "direct_passed": 6, "direct_total": 6, "task_median_seconds": 16.568, "backend": "Ollama 0.35.0", "sente_strict_passed": 2, "sente_total": 2, "sente_seconds": [ 14.113, 17.741 ], "code_cases_passed": 94, "code_cases_available": 94, "tasks": [ { "pass": true, "checks": { "JPY": true, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 16800, "USD": 2000, "refund_c_pending": true }, "model": "qwen3.8:27b", "task": "ledger", "seconds": 10.772, "load_seconds": 0.0, "output_tokens": 479, "thinking_chars": 1028, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "new8の本番デプロイを再実行し、5〜10営業日後にカード明細の返金反映を確認する" }, "model": "qwen3.8:27b", "task": "status", "seconds": 20.938, "load_seconds": 0.0, "output_tokens": 735, "thinking_chars": 1925, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "qwen3.8:27b", "task": "merge_ranges", "seconds": 50.703, "load_seconds": 0.0, "output_tokens": 1641, "thinking_chars": 4787, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "qwen3.8:27b", "task": "latest_active", "seconds": 33.106, "load_seconds": 0.0, "output_tokens": 1024, "thinking_chars": 3126, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "qwen3.8:27b", "task": "tool_read", "seconds": 8.775, "load_seconds": 0.0, "output_tokens": 208, "thinking_chars": 651, "calls": [ "release-note.txt" ] }, { "pass": true, "checks": { "invoice_id": true, "settled_yen": true, "outstanding_yen": true, "fully_paid": true, "actually_read": true }, "answer": { "invoice_id": "INV-307", "settled_yen": 14000, "outstanding_yen": 4400, "fully_paid": false }, "model": "qwen3.8:27b", "task": "tool_reconcile", "seconds": 12.198, "load_seconds": 0.0, "output_tokens": 287, "thinking_chars": 514, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "muse-glimmer:30b", "direct_passed": 6, "direct_total": 6, "task_median_seconds": 88.615, "backend": "Ollama 0.35.0", "sente_strict_passed": 2, "sente_total": 2, "sente_seconds": [ 58.111, 89.501 ], "code_cases_passed": 94, "code_cases_available": 94, "tasks": [ { "pass": true, "checks": { "JPY": true, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 16800, "USD": 2000, "refund_c_pending": true }, "model": "muse-glimmer:30b", "task": "ledger", "seconds": 76.928, "load_seconds": 0.0, "output_tokens": 845, "thinking_chars": 2536, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "カード明細に返金が反映されたか再確認し、デプロイ失敗原因を調査して再デプロイする" }, "model": "muse-glimmer:30b", "task": "status", "seconds": 108.451, "load_seconds": 0.0, "output_tokens": 1278, "thinking_chars": 3054, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "muse-glimmer:30b", "task": "merge_ranges", "seconds": 146.561, "load_seconds": 0.0, "output_tokens": 1918, "thinking_chars": 6325, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "muse-glimmer:30b", "task": "latest_active", "seconds": 100.302, "load_seconds": 0.0, "output_tokens": 1357, "thinking_chars": 4370, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "muse-glimmer:30b", "task": "tool_read", "seconds": 34.882, "load_seconds": 0.0, "output_tokens": 447, "thinking_chars": 1435, "calls": [ "release-note.txt" ] }, { "pass": true, "checks": { "invoice_id": true, "settled_yen": true, "outstanding_yen": true, "fully_paid": true, "actually_read": true }, "answer": { "invoice_id": "INV-307", "settled_yen": 14000, "outstanding_yen": 4400, "fully_paid": false }, "model": "muse-glimmer:30b", "task": "tool_reconcile", "seconds": 54.412, "load_seconds": 0.0, "output_tokens": 686, "thinking_chars": 2008, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "nemotron-3.5-lightning:30b-a3b", "direct_passed": 5, "direct_total": 6, "task_median_seconds": 25.7, "backend": "Ollama 0.35.0", "sente_strict_passed": 1, "sente_total": 2, "sente_seconds": [ 9.044, 63.433 ], "code_cases_passed": 47, "code_cases_available": 94, "tasks": [ { "pass": true, "checks": { "JPY": true, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 16800, "USD": 2000, "refund_c_pending": true }, "model": "nemotron-3.5-lightning:30b-a3b", "task": "ledger", "seconds": 17.657, "load_seconds": 0.0, "output_tokens": 1316, "thinking_chars": 3365, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "カード明細への反映確認（5〜10営業日後）" }, "model": "nemotron-3.5-lightning:30b-a3b", "task": "status", "seconds": 33.744, "load_seconds": 0.0, "output_tokens": 2456, "thinking_chars": 7663, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "nemotron-3.5-lightning:30b-a3b", "task": "merge_ranges", "seconds": 45.8, "load_seconds": 0.0, "output_tokens": 3283, "thinking_chars": 10485, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "nemotron-3.5-lightning:30b-a3b", "task": "latest_active", "seconds": 58.002, "load_seconds": 0.0, "output_tokens": 4096, "thinking_chars": 15939, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "nemotron-3.5-lightning:30b-a3b", "task": "tool_read", "seconds": 4.874, "load_seconds": 0.0, "output_tokens": 329, "thinking_chars": 919, "calls": [ "release-note.txt" ] }, { "pass": true, "checks": { "invoice_id": true, "settled_yen": true, "outstanding_yen": true, "fully_paid": true, "actually_read": true }, "answer": { "invoice_id": "INV-307", "settled_yen": 14000, "outstanding_yen": 4400, "fully_paid": false }, "model": "nemotron-3.5-lightning:30b-a3b", "task": "tool_reconcile", "seconds": 8.962, "load_seconds": 0.0, "output_tokens": 627, "thinking_chars": 1741, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "laguna-s-2.1:nvfp4", "direct_passed": 3, "direct_total": 6, "task_median_seconds": 3.264, "backend": "Ollama 0.35.0", "sente_strict_passed": 0, "sente_total": 2, "sente_seconds": [ 32.189, 66.698 ], "code_cases_passed": 47, "code_cases_available": 94, "tasks": [ { "pass": false, "checks": { "JPY": false, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 17000, "USD": 2000, "refund_c_pending": true }, "model": "laguna-s-2.1:nvfp4", "task": "ledger", "seconds": 3.451, "load_seconds": 0.0, "output_tokens": 25, "thinking_chars": 0, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "カード明細の反映を5〜10営業日待ち、返金・減額行の確認をする" }, "model": "laguna-s-2.1:nvfp4", "task": "status", "seconds": 2.898, "load_seconds": 0.0, "output_tokens": 59, "thinking_chars": 0, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "laguna-s-2.1:nvfp4", "task": "merge_ranges", "seconds": 103.139, "load_seconds": 0.0, "output_tokens": 4096, "thinking_chars": 11404, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "laguna-s-2.1:nvfp4", "task": "latest_active", "seconds": 3.076, "load_seconds": 0.0, "output_tokens": 104, "thinking_chars": 0, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "laguna-s-2.1:nvfp4", "task": "tool_read", "seconds": 2.2, "load_seconds": 0.0, "output_tokens": 49, "thinking_chars": 0, "calls": [ "release-note.txt" ] }, { "pass": false, "error": "RuntimeError('HTTP 500: {\"error\":{\"message\":\"empty Laguna tool call name\",\"type\":\"api_error\",\"param\":null,\"code\":null}}\\n')", "model": "laguna-s-2.1:nvfp4", "task": "tool_reconcile", "seconds": 10.291, "load_seconds": 0.0, "output_tokens": 74, "thinking_chars": 128, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "qwen3.5:122b", "direct_passed": 2, "direct_total": 6, "task_median_seconds": 69.617, "backend": "Ollama 0.35.0", "sente_strict_passed": 1, "sente_total": 2, "sente_seconds": [ 24.045, 29.66 ], "code_cases_passed": 0, "code_cases_available": 94, "tasks": [ { "pass": true, "checks": { "JPY": true, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 16800, "USD": 2000, "refund_c_pending": true }, "model": "qwen3.5:122b", "task": "ledger", "seconds": 67.324, "load_seconds": 0.0, "output_tokens": 3490, "thinking_chars": 10124, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "返金明細の反映待ち（5〜10営業日）、本番デプロイ失敗の原因調査および再デプロイが必要。" }, "model": "qwen3.5:122b", "task": "status", "seconds": 71.91, "load_seconds": 0.0, "output_tokens": 3216, "thinking_chars": 8989, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "qwen3.5:122b", "task": "merge_ranges", "seconds": 141.117, "load_seconds": 0.0, "output_tokens": 4096, "thinking_chars": 15035, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "qwen3.5:122b", "task": "latest_active", "seconds": 137.957, "load_seconds": 0.0, "output_tokens": 4096, "thinking_chars": 15106, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "qwen3.5:122b", "task": "tool_read", "seconds": 7.923, "load_seconds": 0.0, "output_tokens": 213, "thinking_chars": 409, "calls": [ "release-note.txt" ] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "qwen3.5:122b", "task": "tool_reconcile", "seconds": 16.912, "load_seconds": 0.0, "output_tokens": 502, "thinking_chars": 893, "calls": [ "invoice.json", "bank.json" ] } ] }, { "model": "bonsai-2-27b", "direct_passed": 5, "direct_total": 6, "task_median_seconds": 24.627, "backend": "PrismML llama.cpp prism-b10743-adfffbe", "sente_strict_passed": 1, "sente_total": 2, "sente_seconds": [ 52.626, 148.541 ], "code_cases_passed": 47, "code_cases_available": 94, "tasks": [ { "pass": true, "checks": { "JPY": true, "USD": true, "refund_c_pending": true }, "answer": { "JPY": 16800, "USD": 2000, "refund_c_pending": true }, "model": "bonsai-2-27b", "task": "ledger", "seconds": 15.431, "load_seconds": 0.0, "output_tokens": 639, "thinking_chars": 1398, "calls": [] }, { "pass": true, "checks": { "refund_stage": true, "refund_yen": true, "deployed": true, "next_action_format": true }, "answer": { "refund_stage": "merchant_processed", "refund_yen": 0, "deployed": false, "next_action_ja": "カード明細に返金行が反映されるか5〜10営業日後確認し、本番デプロイを再実行する。" }, "model": "bonsai-2-27b", "task": "status", "seconds": 32.699, "load_seconds": 0.0, "output_tokens": 1226, "thinking_chars": 2598, "calls": [] }, { "pass": false, "error": "not strict JSON: Expecting value: line 1 column 1 (char 0)", "model": "bonsai-2-27b", "task": "merge_ranges", "seconds": 121.551, "load_seconds": 0.0, "output_tokens": 4096, "thinking_chars": 14142, "calls": [] }, { "pass": true, "code_tests": { "passed": 47, "total": 47, "failures": [] }, "model": "bonsai-2-27b", "task": "latest_active", "seconds": 80.983, "load_seconds": 0.0, "output_tokens": 2067, "thinking_chars": 8044, "calls": [] }, { "pass": true, "checks": { "release_id": true, "deployed": true, "actually_read": true }, "answer": { "release_id": "R-7429", "deployed": false }, "model": "bonsai-2-27b", "task": "tool_read", "seconds": 10.737, "load_seconds": 0.0, "output_tokens": 191, "thinking_chars": 587, "calls": [ "release-note.txt" ] }, { "pass": true, "checks": { "invoice_id": true, "settled_yen": true, "outstanding_yen": true, "fully_paid": true, "actually_read": true }, "answer": { "invoice_id": "INV-307", "settled_yen": 14000, "outstanding_yen": 4400, "fully_paid": false }, "model": "bonsai-2-27b", "task": "tool_reconcile", "seconds": 16.554, "load_seconds": 0.0, "output_tokens": 344, "thinking_chars": 754, "calls": [ "invoice.json", "bank.json" ] } ] } ] }