{"date": "2026-09-10", "mode": "batch20", "prompt": "以下の20問に、番号順に「番号: 答え」の形式で1行ずつ、できるだけ短く答えてください。前置き・解説は不要です。\n\n1. ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?\n2. バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)\n3. 2026年2月29日は存在する?(する/しない)\n4. 次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"\n5. 「ちょうちょ」は何モーラ?(数字のみ)\n6. サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)\n7. 農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)\n8. 今日が月曜なら100日後は何曜日?\n9. kaleidoscope の7文字目は?(1文字)\n10. 太陽系で2番目に小さい惑星は?\n11. 2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)\n12. 「おはようございます」を1文字ずつ逆順に。\n13. 車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?\n14. 3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)\n15. 父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)\n16. 「わたしはきょうははれです」に「は」は何個?(数字のみ)\n17. \"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?\n18. 「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)\n19. 0.3, 0.25, 1/3, 0.33 を小さい順に。\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)", "questions": [{"id": "q01", "text": "ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?", "pair": "surgeon"}, {"id": "q02", "text": "バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)", "pair": null}, {"id": "q03", "text": "2026年2月29日は存在する?(する/しない)", "pair": null}, {"id": "q04", "text": "次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"", "pair": "longest"}, {"id": "q05", "text": "「ちょうちょ」は何モーラ?(数字のみ)", "pair": "mora"}, {"id": "q06", "text": "サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)", "pair": null}, {"id": "q07", "text": "農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)", "pair": "river"}, {"id": "q08", "text": "今日が月曜なら100日後は何曜日?", "pair": null}, {"id": "q09", "text": "kaleidoscope の7文字目は?(1文字)", "pair": null}, {"id": "q10", "text": "太陽系で2番目に小さい惑星は?", "pair": null}, {"id": "q11", "text": "2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)", "pair": null}, {"id": "q12", "text": "「おはようございます」を1文字ずつ逆順に。", "pair": null}, {"id": "q13", "text": "車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?", "pair": "surgeon"}, {"id": "q14", "text": "3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)", "pair": null}, {"id": "q15", "text": "父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)", "pair": null}, {"id": "q16", "text": "「わたしはきょうははれです」に「は」は何個?(数字のみ)", "pair": null}, {"id": "q17", "text": "\"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?", "pair": "longest"}, {"id": "q18", "text": "「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)", "pair": "mora"}, {"id": "q19", "text": "0.3, 0.25, 1/3, 0.33 を小さい順に。", "pair": null}, {"id": "q20", "text": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)", "pair": "river"}], "results": {"gpt-4.1-nano": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 0, "sec": 3.7, "out_tokens": 132, "price": "$0.1 / $0.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜日", "ok": 0}, "q09": {"a": "5", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ございますおは", "ok": 0}, "q13": {"a": "母と息子の関係", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "3", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "10, 2", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.33, 0.3, 1/3", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 金曜日  \n9: 5  \n10: 金星  \n11: できる  \n12: ございますおは  \n13: 母と息子の関係  \n14: 変える  \n15: 3  \n16: 4  \n17: 10, 2  \n18: 3  \n19: 0.25, 0.33, 0.3, 1/3  \n20: しない"}, "gpt-4o-mini": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 0, "sec": 3.0, "out_tokens": 129, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すではようお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: e  \n10: 火星  \n11: できる  \n12: すではようお  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "gpt-4.1": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 0, "sec": 3.2, "out_tokens": 132, "price": "$2 / $8", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 2  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいざごうよはお  \n13: 母親  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "gpt-4o": {"score": 12, "answered": 20, "first10": 8, "last10": 4, "inconsistent_pairs": 0, "sec": 3.5, "out_tokens": 141, "price": "$2.5 / $10", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "執刀医は父親。", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "gpt-4.1-mini": {"score": 10, "answered": 20, "first10": 5, "last10": 5, "inconsistent_pairs": 0, "sec": 4.1, "out_tokens": 132, "price": "$0.4 / $1.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "claude-opus-4-7": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 1, "sec": 21.8, "out_tokens": 866, "price": "$5 / $25", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ(synchronization)", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ(synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "claude-opus-4-8": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 24.1, "out_tokens": 1433, "price": "", "graded": {"q01": {"a": "父親（母は死亡しているため）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親（父または母）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・1つ（synchronization）", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親（母は死亡しているため）\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親（父または母）\n14: 同じ\n15: 5\n16: 3\n17: 15文字・1つ（synchronization）\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "gemini-2.5-flash": {"score": 15, "answered": 20, "first10": 7, "last10": 8, "inconsistent_pairs": 1, "sec": 30.8, "out_tokens": 5754, "price": "$0.3 / $2.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざうごよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1個", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できない\n12. すまいざうごよはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字, 1個\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "o4-mini": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 35.3, "out_tokens": 4395, "price": "$1.1 / $4.4", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15,2  \n18: 3  \n19: 0.25,0.3,0.33,1/3  \n20: しない"}, "claude-fable-5": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 20.4, "out_tokens": 934, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ(extraordinarily と synchronization)", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "o3-mini": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 16.5, "out_tokens": 3789, "price": "$1.1 / $4.4", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 15,2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "gemini-2.5-flash-lite": {"score": 8, "answered": 20, "first10": 6, "last10": 2, "inconsistent_pairs": 2, "sec": 16.0, "out_tokens": 5908, "price": "$0.1 / $0.4", "graded": {"q01": {"a": "外科医は少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す、い、ま、せ、ん、ご、は、お", "ok": 0}, "q13": {"a": "息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "2", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "する", "ok": 0}}, "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 2\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す、い、ま、せ、ん、ご、は、お\n13: 息子\n14: 変える\n15: 2\n16: 3\n17: 14文字、1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: する"}, "claude-sonnet-4-5-20250929": {"score": 17, "answered": 20, "first10": 9, "last10": 8, "inconsistent_pairs": 1, "sec": 19.4, "out_tokens": 1025, "price": "$3 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまがいさごうょはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまがいさごうょはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字、2つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "meta-llama/llama-4-scout": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 13.5, "out_tokens": 126, "price": "$0.1 / $0.3", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "4", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すうよおはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 4\n7: できる\n8: 木曜\n9: o\n10: 火星\n11: できる\n12: すうよおはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15, 1\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "deepseek-chat": {"score": 13, "answered": 20, "first10": 7, "last10": 6, "inconsistent_pairs": 0, "sec": 2.1, "out_tokens": 128, "price": "$0.257 / $1.029", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "gemini-2.5-pro": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 45.4, "out_tokens": 655, "price": "$1.25 / $10", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "meta-llama/llama-4-maverick": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 1, "sec": 3.3, "out_tokens": 123, "price": "$0.2 / $0.8", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "c", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますございますようお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: c\n10: 水星\n11: できる\n12: ますございますようお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "qwen/qwen3-30b-a3b": {"score": 13, "answered": 20, "first10": 8, "last10": 5, "inconsistent_pairs": 1, "sec": 16.3, "out_tokens": 3216, "price": "$0.13 / $0.52", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "O", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 父  \n2. 10  \n3. しない  \n4. extraordinarily  \n5. 2  \n6. 2  \n7. できない  \n8. 水曜日  \n9. O  \n10. 火星  \n11. できる  \n12. すまいざごうはお  \n13. 母  \n14. 同じ  \n15. 5  \n16. 3  \n17. 16, 1  \n18. 2  \n19. 0.25, 0.3, 0.33, 1/3  \n20. しない"}, "deepseek-reasoner": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 24.5, "out_tokens": 4672, "price": "$0.55 / $2.19", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"}, "google/gemini-3-flash-preview": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 16.1, "out_tokens": 2285, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよは", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できない\n12. すまいざごうよは\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字、2つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "qwen/qwen3-coder": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 1, "sec": 3.5, "out_tokens": 152, "price": "$0.3 / $1", "graded": {"q01": {"a": "外科医は母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すごうよはお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "5", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医は母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 5  \n6: 3  \n7: できない  \n8: 土曜日  \n9: s  \n10: 火星  \n11: できる  \n12: すごうよはお  \n13: 母子  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 5  \n19: 0.25, 1/3, 0.3, 0.33  \n20: 存在する"}, "mistralai/mistral-large": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 3.5, "out_tokens": 149, "price": "$2 / $6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "1", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すはまいざおよおは", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはまいざおよおは\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "mistralai/mistral-small-3.2-24b-instruct": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 0, "sec": 11.8, "out_tokens": 150, "price": "$0.09375 / $0.25", "graded": {"q01": {"a": "少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すごうやおは", "ok": 0}, "q13": {"a": "母と息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "19文字、1単語", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すごうやおは\n13: 母と息子\n14: 変える\n15: 10\n16: 3\n17: 19文字、1単語\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "z-ai/glm-5": {"score": 12, "answered": 20, "first10": 8, "last10": 4, "inconsistent_pairs": 0, "sec": 47.6, "out_tokens": 141, "price": "$0.6 / $1.92", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "執刀医は父親。", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "mistralai/codestral-2508": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 2, "sec": 2.8, "out_tokens": 151, "price": "$0.3 / $0.9", "graded": {"q01": {"a": "外科医は母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ですれはごございよう", "ok": 0}, "q13": {"a": "祖父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 外科医は母親\n2. 10\n3. する\n4. extraordinarily\n5. 3\n6. 3\n7. できない\n8. 水曜日\n9. d\n10. 金星\n11. できる\n12. ですれはごございよう\n13. 祖父\n14. 変える\n15. 10\n16. 3\n17. 16文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "cohere/command-a": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 0, "sec": 2.7, "out_tokens": 142, "price": "$2.5 / $10", "graded": {"q01": {"a": "少年は母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すうざごはよは", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 少年は母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 金曜\n9: s\n10: 水星\n11: できる\n12: すうざごはよは\n13: 母親\n14: 変える\n15: 5\n16: 4\n17: 13文字、1つ\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "cohere/command-r-plus-08-2024": {"score": 3, "answered": 20, "first10": 2, "last10": 1, "inconsistent_pairs": 1, "sec": 4.2, "out_tokens": 140, "price": "$2.5 / $10", "graded": {"q01": {"a": "少年", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "金曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すざぎぞうあはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15、1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "1/3, 0.25, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 少年\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 0\n7: できる\n8: 金曜\n9: s\n10: 水星\n11: できる\n12: すざぎぞうあはお\n13: 父親\n14: 変える\n15: 20\n16: 4\n17: 15、1\n18: 4\n19: 1/3, 0.25, 0.3, 0.33\n20: 存在する"}, "cohere/command-r-08-2024": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 4.2, "out_tokens": 138, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "金曜日", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますぞよあおは", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 父親\n2. 10\n3. する\n4. synchronization\n5. 2\n6. 0\n7. できる\n8. 金曜日\n9. d\n10. 水星\n11. できる\n12. ますぞよあおは\n13. 父親\n14. 同じ\n15. 10\n16. 4\n17. 15, 1\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"}, "google/gemma-3-27b-it": {"score": 4, "answered": 20, "first10": 4, "last10": 0, "inconsistent_pairs": 1, "sec": 8.2, "out_tokens": 139, "price": "$0.08 / $0.45", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜", "ok": 0}, "q09": {"a": "c", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいますうはょうお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "6", "ok": 0}, "q17": {"a": "17, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 金曜\n9: c\n10: 水星\n11: できる\n12: すまいますうはょうお\n13: 母親\n14: 変える\n15: 10\n16: 6\n17: 17, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "moonshotai/kimi-k2.5": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 20.3, "out_tokens": 4490, "price": "$0.45 / $2.25", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2個", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "microsoft/phi-4": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 1, "sec": 4.6, "out_tokens": 249, "price": "$0.07 / $0.14", "graded": {"q01": {"a": "番号: 母親", "ok": 0}, "q02": {"a": "番号: 10", "ok": 1}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: extraordinarily", "ok": 0}, "q05": {"a": "番号: 3", "ok": 1}, "q06": {"a": "番号: 2", "ok": 1}, "q07": {"a": "番号: できない", "ok": 1}, "q08": {"a": "番号: 木曜日", "ok": 0}, "q09": {"a": "番号: o", "ok": 1}, "q10": {"a": "番号: マーキュリー", "ok": 0}, "q11": {"a": "番号: できる", "ok": 0}, "q12": {"a": "番号: suomuogah yadot", "ok": 0}, "q13": {"a": "番号: 母親", "ok": 0}, "q14": {"a": "番号: 変える", "ok": 0}, "q15": {"a": "番号: 10", "ok": 0}, "q16": {"a": "番号: 3", "ok": 1}, "q17": {"a": "番号: 18, 1", "ok": 0}, "q18": {"a": "番号: 3", "ok": 1}, "q19": {"a": "番号: 0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "番号: 存在する", "ok": 0}}, "raw": "1. 番号: 母親\n2. 番号: 10\n3. 番号: しない\n4. 番号: extraordinarily\n5. 番号: 3\n6. 番号: 2\n7. 番号: できない\n8. 番号: 木曜日\n9. 番号: o\n10. 番号: マーキュリー\n11. 番号: できる\n12. 番号: suomuogah yadot\n13. 番号: 母親\n14. 番号: 変える\n15. 番号: 10\n16. 番号: 3\n17. 番号: 18, 1\n18. 番号: 3\n19. 番号: 0.25, 0.3, 0.33, 1/3\n20. 番号: 存在する"}, "openai/gpt-5.6-luna": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 14.1, "out_tokens": 1131, "price": "$0.5 / $3", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25、0.3、0.33、1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25、0.3、0.33、1/3\n20: 存在しない"}, "openai/gpt-5.4": {"score": 13, "answered": 20, "first10": 6, "last10": 7, "inconsistent_pairs": 1, "sec": 3.7, "out_tokens": 135, "price": "$2.5 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.4-mini": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 1, "sec": 1.9, "out_tokens": 140, "price": "$0.75 / $4.5", "graded": {"q01": {"a": "外科医は母親", "ok": 0}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すでまうごうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医は母親  \n2: 10円  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 水星  \n11: できる  \n12: すでまうごうよはお  \n13: 母親  \n14: 変える  \n15: 20  \n16: 3  \n17: 15文字、2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-5.6-sol": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 31.7, "out_tokens": 860, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "openai/gpt-5.4-nano": {"score": 8, "answered": 20, "first10": 4, "last10": 4, "inconsistent_pairs": 3, "sec": 2.6, "out_tokens": 140, "price": "$0.2 / $1.25", "graded": {"q01": {"a": "外科医の自分の息子", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "いすようははお", "ok": 0}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "13文字、1語", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医の自分の息子  \n2: 10  \n3: する  \n4: synchronization  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: o  \n10: 火星  \n11: できない  \n12: いすようははお  \n13: 父と息子  \n14: 変える  \n15: 5  \n16: 2  \n17: 13文字、1語  \n18: 4  \n19: 0.25, 0.3, 1/3, 0.33  \n20: しない"}, "google/gemini-3.1-flash-lite": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 0, "sec": 5.2, "out_tokens": 1184, "price": "$0.125 / $0.75", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまざいさうはよお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまざいさうはよお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.6-luna-pro": {"score": 16, "answered": 20, "first10": 7, "last10": 9, "inconsistent_pairs": 1, "sec": 25.6, "out_tokens": 4128, "price": "$0.5 / $3", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "火曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 火曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"}, "google/gemini-3.5-flash": {"score": 17, "answered": 20, "first10": 9, "last10": 8, "inconsistent_pairs": 2, "sec": 12.6, "out_tokens": 2141, "price": "$0.75 / $4.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいごとうよはお", "ok": 0}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいごとうよはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-3.1-pro-preview": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 11.9, "out_tokens": 1469, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "deepseek/deepseek-v4-pro": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 90.6, "out_tokens": 2071, "price": "$0.66 / $1.98", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "z-ai/glm-5.3": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 0, "sec": 3.9, "out_tokens": 541, "price": "", "graded": {"q01": {"a": "母親(女性外科医)", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいごうはよお", "ok": 0}, "q13": {"a": "母親(女性医師)", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "16文字、1つ(extraordinarily)", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 1/3 < 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親(女性外科医)\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいごうはよお\n13: 母親(女性医師)\n14: 同じ\n15: 5\n16: 4\n17: 16文字、1つ(extraordinarily)\n18: 4\n19: 0.25 < 0.3 < 1/3 < 0.33\n20: しない"}, "deepseek/deepseek-v4-flash-0731": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 35.1, "out_tokens": 4447, "price": "$0.09 / $0.18", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "deepseek/deepseek-v4-pro-cache": {"score": 12, "answered": 20, "first10": 5, "last10": 7, "inconsistent_pairs": 1, "sec": 63.0, "out_tokens": 1045, "price": "$0.022 / $1.98", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できる  \n12: すまいざごうよはお  \n13: 母子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"}, "x-ai/grok-4.3": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 2, "sec": 16.6, "out_tokens": 1759, "price": "$1.25 / $2.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すませいごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すませいごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16文字で1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "deepseek/deepseek-v4-pro-0813": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 0, "sec": 5.0, "out_tokens": 336, "price": "", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す ま い ご ざ よ う は お", "ok": 0}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す ま い ご ざ よ う は お\n13: 親子\n14: 同じ\n15: 5\n16: 2\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "moonshotai/kimi-k3": {"score": 18, "answered": 20, "first10": 8, "last10": 10, "inconsistent_pairs": 1, "sec": 6.0, "out_tokens": 607, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2語  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "deepseek/deepseek-v4-flash": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 52.3, "out_tokens": 4826, "price": "$0.22 / $0.66", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "moonshotai/kimi-k2.7-code": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 0, "sec": 2.2, "out_tokens": 135, "price": "$0.71 / $3.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "i", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまざいごうはよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: i\n10: 金星\n11: できない\n12: すまざいごうはよはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3.8-27b": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 1, "sec": 6.2, "out_tokens": 399, "price": "$0.45 / $3.2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "1", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "r", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ざいみあしやうはお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2個", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できない\n8: 水曜日\n9: r\n10: 水星\n11: できる\n12: ざいみあしやうはお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2個\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"}, "nvidia/nemotron-3-ultra-550b-a55b": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 14.3, "out_tokens": 3438, "price": "$0.6 / $3.6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15文字, 2個", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 2\n17: 15文字, 2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-3.5-flash-lite": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 0, "sec": 11.9, "out_tokens": 3018, "price": "$0.3 / $2.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15文字・1個", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 2\n17: 15文字・1個\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "x-ai/grok-4.20": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 27.2, "out_tokens": 2928, "price": "$1.25 / $2.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16,1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25 0.3 0.33 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16,1\n18: 4\n19: 0.25 0.3 0.33 1/3\n20: しない"}, "qwen/qwen3.6-35b-a3b": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 0, "sec": 22.5, "out_tokens": 5820, "price": "$0.15 / $1", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "す ま ざ い ご う よ は お", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: す ま ざ い ご う よ は お\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "thinkingmachines/inkling": {"score": 12, "answered": 20, "first10": 7, "last10": 5, "inconsistent_pairs": 0, "sec": 6.3, "out_tokens": 1327, "price": "", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいござうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 16文字、1つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "nvidia/Nemotron-3-Nano-30B-A3B": {"score": 13, "answered": 20, "first10": 8, "last10": 5, "inconsistent_pairs": 1, "sec": 17.7, "out_tokens": 4878, "price": "$0.04 / $0.16", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごんるようは", "ok": 0}, "q13": {"a": "5", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15 1", "ok": 0}, "q18": {"a": "5", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごんるようは\n13: 5\n14: 変える\n15: 5\n16: 3\n17: 15 1\n18: 5\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3.8-2.4t-a95b": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 19.0, "out_tokens": 3770, "price": "$2 / $6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "sakana/sakana-namazu": {"score": 18, "answered": 20, "first10": 8, "last10": 10, "inconsistent_pairs": 1, "sec": 17.6, "out_tokens": 3912, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "mistralai/mistral-medium-3-5": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 0, "sec": 34.8, "out_tokens": 3261, "price": "", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますはちおはよう", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字,1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 変える\n15: 10\n16: 3\n17: 15文字,1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "tencent/hy3": {"score": 15, "answered": 20, "first10": 9, "last10": 6, "inconsistent_pairs": 0, "sec": 23.1, "out_tokens": 2029, "price": "$0.132 / $0.528", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すみまそごうようはお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すみまそごうようはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 14文字1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "inception/mercury-2.5": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 2, "sec": 2.5, "out_tokens": 393, "price": "$0.04 / $0.15", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すざごようおお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "17, 2", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 木曜日\n9: d\n10: 水星\n11: できる\n12: すざごようおお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 17, 2\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "openai/gpt-6-astra": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 9.4, "out_tokens": 324, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemma-4-31b-it": {"score": 17, "answered": 20, "first10": 9, "last10": 8, "inconsistent_pairs": 0, "sec": 21.5, "out_tokens": 1866, "price": "$0.1 / $0.34", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すざまごいぶおはよ", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すざまごいぶおはよ\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-3.8-flash": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 13.2, "out_tokens": 2195, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と子（執刀医が父親）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と子（執刀医が父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "anthropic/claude-sonnet-5": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 26.4, "out_tokens": 2060, "price": "", "graded": {"q01": {"a": "父親(母親は死亡しているため)", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親(実の親)", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 父親(母親は死亡しているため)\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(実の親)\n14: 変える\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "tencent/hy-mt2-1.8b": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 2.4, "out_tokens": 114, "price": "$0.044 / $0.177", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "50", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "complicated", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "月曜", "ok": 0}, "q09": {"a": "7", "ok": 0}, "q10": {"a": "エイリアス", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "ございます", "ok": 0}, "q13": {"a": "息子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "1", "ok": 0}, "q17": {"a": "3", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "1/3", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医\n2: 50\n3: しない\n4: complicated\n5: 2\n6: 9\n7: できない\n8: 月曜\n9: 7\n10: エイリアス\n11: できない\n12: ございます\n13: 息子\n14: 同じ\n15: 10\n16: 1\n17: 3\n18: 2\n19: 1/3\n20: しない"}, "z-ai/glm-5.1": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 26.4, "out_tokens": 3207, "price": "$0.966 / $3.036", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "tencent/hy-mt2-30b-a3b": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 1, "sec": 2.8, "out_tokens": 132, "price": "$0.074 / $0.295", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜", "ok": 0}, "q09": {"a": "p", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すござよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "13, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 3\n7: できない\n8: 金曜\n9: p\n10: 火星\n11: できる\n12: すござよはお\n13: 母親\n14: 変える\n15: 20\n16: 4\n17: 13, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "ibm-granite/granite-4.2-8b": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 1, "sec": 2.4, "out_tokens": 153, "price": "$0.06 / $0.25", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "40", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "月曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "土星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "さぁおはようございます", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "1", "ok": 0}, "q17": {"a": "20文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 外科医\n2. 40\n3. しない\n4. extraordinarily\n5. 2\n6. 9\n7. できない\n8. 月曜日\n9. s\n10. 土星\n11. できる\n12. さぁおはようございます\n13. 父\n14. 変える\n15. 20\n16. 1\n17. 20文字、1つ\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "tencent/hy-mt2-7b": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 2, "sec": 2.8, "out_tokens": 151, "price": "$0.074 / $0.295", "graded": {"q01": {"a": "外科医自身", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "6", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "木星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すざおはよう", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "21 2", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25 1/3 0.3 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医自身\n2: 20\n3: しない\n4: extraordinarily\n5: 2\n6: 6\n7: できない\n8: 木曜\n9: o\n10: 木星\n11: できる\n12: すざおはよう\n13: 父親\n14: 同じ\n15: 10\n16: 2\n17: 21 2\n18: 2\n19: 0.25 1/3 0.3 0.33\n20: 存在する"}, "thinkingmachines/inkling-small": {"score": 8, "answered": 20, "first10": 4, "last10": 4, "inconsistent_pairs": 1, "sec": 5.0, "out_tokens": 856, "price": "", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまごろうは", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜\n9: o\n10: 水星\n11: できる\n12: すまごろうは\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"}, "inclusionai/ling-3.0-flash": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 0, "sec": 5.6, "out_tokens": 1100, "price": "$0.021 / $0.063", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまれいこわは", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまれいこわは\n13: 母親\n14: 同じ\n15: 5\n16: 2\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "poolside/laguna-s-2.1": {"score": 4, "answered": 20, "first10": 2, "last10": 2, "inconsistent_pairs": 1, "sec": 2.9, "out_tokens": 168, "price": "$0.09 / $0.18", "graded": {"q01": {"a": "外科医は母親自身である。", "ok": 0}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまぞがあようお", "ok": 0}, "q13": {"a": "父親である", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 外科医は母親自身である。\n2. 10円\n3. する\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 木曜日\n9. o\n10. 水星\n11. できる\n12. すまぞがあようお\n13. 父親である\n14. 変える\n15. 5\n16. 2\n17. 16文字、1つ\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "bytedance-seed/seed-2.0-code": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 21.0, "out_tokens": 1711, "price": "$0.5 / $3", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子（父親と息子）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子（父親と息子）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "poolside/laguna-xs-2.1": {"score": 15, "answered": 20, "first10": 7, "last10": 8, "inconsistent_pairs": 1, "sec": 11.9, "out_tokens": 1857, "price": "$0.06 / $0.12", "graded": {"q01": {"a": "外科医: 父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 外科医: 父\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 2\n7. できる\n8. 水曜\n9. o\n10. 金星\n11. できない\n12. すまいざごうよはお\n13. 父と息子\n14. 変える\n15. 5\n16. 3\n17. 15文字で1つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "moonshotai/kimi-k2.6": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 27.8, "out_tokens": 3900, "price": "$0.5415 / $2.28", "graded": {"q01": {"a": "ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?", "ok": 0}, "q02": {"a": "バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)", "ok": 0}, "q03": {"a": "2026年2月29日は存在する?(する/しない)", "ok": 1}, "q04": {"a": "次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"", "ok": 0}, "q05": {"a": "「ちょうちょ」は何モーラ?(数字のみ)", "ok": 0}, "q06": {"a": "サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)", "ok": 0}, "q07": {"a": "農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)", "ok": 1}, "q08": {"a": "今日が月曜なら100日後は何曜日?", "ok": 0}, "q09": {"a": "kaleidoscope の7文字目は?(1文字)", "ok": 0}, "q10": {"a": "太陽系で2番目に小さい惑星は?", "ok": 0}, "q11": {"a": "2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)", "ok": 1}, "q12": {"a": "「おはようございます」を1文字ずつ逆順に。", "ok": 0}, "q13": {"a": "車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?", "ok": 0}, "q14": {"a": "3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)", "ok": 1}, "q15": {"a": "父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)", "ok": 0}, "q16": {"a": "「わたしはきょうははれです」に「は」は何個?(数字のみ)", "ok": 0}, "q17": {"a": "\"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?", "ok": 0}, "q18": {"a": "「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)", "ok": 0}, "q19": {"a": "0.3, 0.25, 1/3, 0.33 を小さい順に。", "ok": 0}, "q20": {"a": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)", "ok": 1}}, "raw": "ユーザーは20問に対して、番号順に「番号: 答え」の形式で1行ずつ、できるだけ短く答えることを求めています。前置き・解説は不要です。\n\n各問題を確認して回答を考えます。\n\n1. ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?\n   - 母親は死亡したが、外科医は父親です。または、外科医は母親（養子縁組や同性婚の可能性もありますが、古典的ななぞなぞの答えは「父親」）。ただし、現代的な解釈では「父親」が一般的。\n\n2. バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)\n   - バット = x, ボール = y。x + y = 120, x = y + 100。\n   - (y+100) + y = 120 → 2y = 20 → y = 10。\n   - 答え: 10\n\n3. 2026年2月29日は存在する?(する/しない)\n   - 2026年は閏年ではない（2024年が閏年、2028年が閏年）。2026 ÷ 4 = 506.5。閏年ではないので2月29日は存在しない。\n   - 答え: しない\n\n4. 次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"\n   - extraordinarily (16文字), complicated (11文字), synchronization (15文字)。\n   - extraordinarily が最長。\n   - 答え: extraordinarily\n\n5. 「ちょうちょ」は何モーラ?(数字のみ)\n   - ちょ・う・ちょ → 3モーラ？ それとも「ちょうちょ」は「ちょーちょ」？\n   - モーラ（拍）の数え方：「ちょう」は2モーラ（ちょ+う）。「ちょ」は1モーラ、「う」は1モーラ。なので「ちょうちょ」は「ちょ・う・ちょ」で3モーラ。\n   - でも「ちょうちょ」は通常「ちょうちょ」と書いて、モーラ数は4？ いや、長音「う」は1モーラ。なので「ちょ(1)・う(1)・ちょ(1)」で3モーラ。\n   - 待って、18番と重複しているような？ 18番は「拍(モーラ)で区切るといくつに分かれる?」と同じ質問？\n   - 18番と5番は同じ質問に見える。でも違う番号。同じ答えでいいのかな。とりあえず3。\n\n6. サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)\n   - サリーは女の子。兄弟3人。兄弟それぞれに3人の姉妹。サリーもその姉妹の1人。なので姉妹はサリーを含めて3人。サリーの姉妹は3人？ いや、「サリーの姉妹」はサリー以外の姉妹の数。\n   - 兄弟3人がそれぞれ「3人の姉妹」を持つということは、姉妹の総数は3人（サリー+他2人、またはサリー含めて3人）。サリー自身も姉妹の1人なので、サリーの姉妹（サリー以外）は2人。\n   - でも「サリーには3人の兄弟」→サリーは女。「兄弟それぞれに3人の姉妹」→兄弟から見て姉妹が3人。サリーはそのうちの1人。残りは2人。\n   - 答え: 2\n\n7. 農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)\n   - 通常の川渡り問題は往復何度もできるが、ここでは1往復しかできない。つまり行って帰ってくるのが1回だけ。それで全部運べるか？\n   - 農夫+1つしか乗れない。往復1回で壊れる。\n   - 行き1回目：農夫+ヤギを運ぶ（残りはオオカミとキャベツ。オオカミはキャベツを食べない）。対岸にヤギ。\n   - 帰り：農夫だけ戻る。これで1往復完了。ボートが壊れる。\n   - 残りのオオカミとキャベツは運べない。\n   - 行き1回目で農夫+オオカミ、帰りで農夫+ヤギ？ いや、ボートは農夫+1つまで。帰りも農夫+1つまで乗れるが、往復1回で壊れるということは、行きと帰りの合計が1往復。つまり戻ってこれなくなる？\n   - 「1往復で壊れる」→ 行って帰るのが1回まで。2回目の出発はできない。\n   - 行き：農夫+ヤギ。対岸にヤギ。残りはオオカミとキャベツ（安全）。\n   - 帰り：農夫だけ戻る。これで1往復。ボート壊れる。\n   - 残り2つ（オオカミ、キャベツ）を運べない。\n   - 行きで農夫+オオカミ、帰りで農夫+ヤギ？ 帰りにヤギを連れてくると、出発岸にヤギが戻る。対岸にオオカミ。出発岸にヤギとキャベツが残る。ヤギはキャベツを食べる。だめ。\n   - 行きで農夫+キャベツ、帰りで農夫+ヤギ？ 対岸にキャベツ。出発岸にヤギとオオカミ。ヤギが食べられる。だめ。\n   - 行きで農夫+ヤギ、帰りで農夫+何もない？ それなら対岸にヤギだけ。出発岸にオオカミとキャベツ（安全）。でも残り2つを運べない。\n   - 結局、全部運べない。\n   - 答え: できない\n\n8. 今日が月曜なら100日後は何曜日?\n   - 100 ÷ 7 = 14余り2。\n   - 月曜 + 2日 = 水曜日。\n   - 答え: 水曜日\n\n9. kaleidoscope の7文字目は?(1文字)\n   - k(1)a(2)l(3)e(4)i(5)d(6)o(7)s(8)c(9)o(10)p(11)e(12)\n   - 7文字目は o\n   - 答え: o\n\n10. 太陽系で2番目に小さい惑星は?\n    - 惑星の大きさ（直径）順：水星、火星、金星、地球、海王星、天王星、土星、木星。\n    - 最小は水星。2番目に小さいは火星。\n    - 質量順でも火星が2番目に小さい（水星が最も小さい）。\n    - 答え: 火星\n\n11. 2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)\n    - 2Lと4L。4Lに満たす。2Lに移すと4Lに2L残る。2Lを捨てる。4Lの2Lを2Lに移す。4Lにまた満たす。2Lに注ぐと2Lはすでに2Lあるので入らない？ いや、2L容器に2L入っているので満杯。4Lから2L注げない。\n    - 別の方法：4Lに満たす。2Lに注ぎ、4Lに2L残る。2Lを捨てる。4Lの2Lを2Lに移す。4Lに満たす。2Lに注ごうとするが、2Lにはすでに2L入っているので0Lしか入らない。4Lは4Lのまま。\n    - 2Lに満たす。4Lに注ぐ。2Lに満たす。4Lに注ぐと4Lは満杯（4L）。2Lは0L。これで4L。\n    - 2Lと4Lでは3Lは作れない。最大公約数が2Lなので、作れるのは2Lの倍数のみ。3Lは作れない。\n    - 答え: できない\n\n12. 「おはようございます」を1文字ずつ逆順に。\n    - す・ま・い・ざ・い・ご・う・よ・は・お\n    - 答え: すまいざいごうよはお\n\n13. 車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?\n    - 1番と同じパターン。執刀医は父親。\n    - 答え: 父子（または父親）\n\n14. 3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)\n    - 通常のモンティ・ホール問題では司会者は正解を知っているが、ここでは「司会者は正解を知らずランダムにBを開けハズレ」。\n    - これは「無知な司会者」バージョン。条件付き確率を考える。\n    - Aが当たり(1/3)、Bが当たり(1/3)、Cが当たり(1/3)。\n    - 司会者がランダムにBを開けてハズレだった場合：\n      - Aが当たりの確率：1/3\n      - Cが当たりの確率：1/3\n      - Bが当たりの確率：0（開けてハズレだから）\n      - ただし、司会者がBを開けたという条件のもとで、Bが当たりだった場合は開けられない（ハズレを開けたとあるので、Bが当たりならこの事象は起きない）。\n      - 実はこの条件のもとでは、AとCの確率は等しいまま（各1/2）になる。変えても変えなくても同じ。\n    - 答え: 同じ\n\n15. 父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)\n    - x年後：40+x = 3(10+x)\n    - 40+x = 30+3x\n    - 10 = 2x\n    - x = 5\n    - 答え: 5\n\n16. 「わたしはきょうははれです」に「は」は何個?(数字のみ)\n    - わたしは(1) きょうは(2) はれです(3)\n    - 「はれ」の「は」も含めるか？ 問題文は「に『は』は何個」。平仮名の「は」を数える。\n    - わたし「は」、きょう「は」、は「れ」→ 3個。\n    - 待って、「はれです」の「は」は助詞ではなくても文字としての「は」なので数える。3個。\n    - 答え: 3\n\n17. \"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?\n    - The(3) extraordinarily(16) complicated(11) synchronization(15) failed(6).\n    - 最も文字数が多いのは extraordinarily で16文字。\n    - 該当する単語は1つ。\n    - 答えの形式は問われていないが、「16文字、1つ」のように短く。\n    - ユーザーは「できるだけ短く」と言っているので「16文字、1つ」または「16,1」など。でも「番号: 答え」形式。\n    - 文脈的に「16文字で1つ」が適切。\n\n18. 「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)\n    - 5番と同じ。3。\n    - 答え: 3\n\n19. 0.3, 0.25, 1/3, 0.33 を小さい順に。\n    - 0.25 = 1/4 = 0.25\n    - 0.3 = 0.30\n    - 0.33 = 0.33\n    - 1/3 = 0.3333...\n    - 小さい順：0.25, 0.3, 0.33, 1/3\n    - 答え: 0.25, 0.3, 0.33, 1/3\n\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)\n    - 7番と同じ質問？ 7番は「全部運べる?(できる/できない)」、20番は「手順は存在する?(存在する/しない)」。\n    - 7番ですでに「できない」と判断。20番も同じく「しない」。\n    - 答え: しない\n\n再確認：\n- 1: 父親\n- 2: 10\n- 3: しない\n- 4: extraordinarily\n- 5: 3\n- 6: 2\n- 7: できない\n- 8: 水曜日\n- 9: o\n- 10: 火星\n- 11: できない\n- 12: すまいざいごうよはお\n- 13: 父親（父子）\n- 14: 同じ\n- 15: 5\n- 16: 3\n- 17: 16文字、1つ\n- 18: 3\n- 19: 0.25, 0.3, 0.33, 1/3\n- 20: しない\n\n17番の答え方：「16文字で1つ」が短くていいかも。\n19番：カンマ区切りで。\n\nすべて1行ずつ、番号: 答え の形式。\n\n最終確認：\n- 5番と18番は同じ答え（3）で問題なし。\n- 7番と20番は同じ結論（できない/しない）で問題なし。\n- 1番と13番は似ているが、13番は「執刀医と息子の関係は?」なので「父子」または「父と息子」。1番は「外科医は誰?」なので「父親」。\n\n出力します。1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざいごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "meta/muse-glimmer-30b": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 0, "sec": 15.1, "out_tokens": 1611, "price": "$0.3 / $1.1", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごようはお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字 1個", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごようはお\n13: 母\n14: 同じ\n15: 5\n16: 3\n17: 16文字 1個\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "aion-labs/aion-3.0-mini": {"score": 11, "answered": 20, "first10": 6, "last10": 5, "inconsistent_pairs": 0, "sec": 4.3, "out_tokens": 192, "price": "$0.7 / $1.4", "graded": {"q01": {"a": "外科医は少年の父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまざうございよはうお", "ok": 0}, "q13": {"a": "執刀医は父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字, 1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は少年の父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 3\n7. できない\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. すまざうございよはうお\n13. 執刀医は父親\n14. 変える\n15. 5\n16. 3\n17. 16文字, 1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在しない"}, "anthropic/claude-opus-4.7": {"score": 18, "answered": 20, "first10": 8, "last10": 10, "inconsistent_pairs": 2, "sec": 27.9, "out_tokens": 1360, "price": "$5 / $25", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2個", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "tencent/hy3-preview": {"score": 12, "answered": 20, "first10": 7, "last10": 5, "inconsistent_pairs": 0, "sec": 25.0, "out_tokens": 2399, "price": "$0.18 / $0.6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すんいざごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すんいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "arcee-ai/trinity-large-thinking": {"score": 11, "answered": 20, "first10": 6, "last10": 5, "inconsistent_pairs": 0, "sec": 8.6, "out_tokens": 2047, "price": "$0.25 / $0.8", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまざごようこはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 2  \n6: 2  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できない  \n12: すまざごようこはお  \n13: 母親  \n14: 同じ  \n15: 10  \n16: 3  \n17: 14文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"}, "rekaai/reka-edge": {"score": 2, "answered": 20, "first10": 0, "last10": 2, "inconsistent_pairs": 0, "sec": 2.6, "out_tokens": 244, "price": "$0.1 / $0.1", "graded": {"q01": {"a": "答え: 1", "ok": 0}, "q02": {"a": "答え: 2", "ok": 0}, "q03": {"a": "答え: 29", "ok": 0}, "q04": {"a": "答え: 101", "ok": 0}, "q05": {"a": "答え: 7", "ok": 0}, "q06": {"a": "答え: 6", "ok": 0}, "q07": {"a": "答え: 2", "ok": 0}, "q08": {"a": "答え: 101", "ok": 0}, "q09": {"a": "答え: 7", "ok": 0}, "q10": {"a": "答え: Mercury", "ok": 0}, "q11": {"a": "答え: できない", "ok": 1}, "q12": {"a": "答え: おはようございます", "ok": 0}, "q13": {"a": "答え: 執刀医と息子の関係は不明", "ok": 0}, "q14": {"a": "答え: 変える", "ok": 0}, "q15": {"a": "答え: 15年", "ok": 0}, "q16": {"a": "答え: 3", "ok": 1}, "q17": {"a": "答え: 8", "ok": 0}, "q18": {"a": "答え: 8", "ok": 0}, "q19": {"a": "小さい順に: 0.3、0.25、1/3、0.33", "ok": 0}, "q20": {"a": "答え: 存在する", "ok": 0}}, "raw": "1. 答え: 1\n2. 答え: 2\n3. 答え: 29\n4. 答え: 101\n5. 答え: 7\n6. 答え: 6\n7. 答え: 2\n8. 答え: 101\n9. 答え: 7\n10. 答え: Mercury\n11. 答え: できない\n12. 答え: おはようございます\n13. 答え: 執刀医と息子の関係は不明\n14. 答え: 変える\n15. 答え: 15年\n16. 答え: 3\n17. 答え: 8\n18. 答え: 8\n19. 小さい順に: 0.3、0.25、1/3、0.33\n20. 答え: 存在する"}, "anthropic/claude-opus-4.8": {"score": 17, "answered": 20, "first10": 9, "last10": 8, "inconsistent_pairs": 1, "sec": 26.9, "out_tokens": 1425, "price": "$5 / $25", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子（親）", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子（親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "inception/mercury-2": {"score": 16, "answered": 20, "first10": 9, "last10": 7, "inconsistent_pairs": 0, "sec": 4.0, "out_tokens": 1254, "price": "$0.25 / $0.75", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいごはお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できる  \n12: すまいごはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "mistralai/mistral-small-2603": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 2, "sec": 10.3, "out_tokens": 1246, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "はちにんまよ", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: はちにんまよ\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字, 1\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "google/gemini-3.1-flash-lite-preview": {"score": 14, "answered": 20, "first10": 8, "last10": 6, "inconsistent_pairs": 1, "sec": 4.1, "out_tokens": 1092, "price": "$0.25 / $1.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまざいごうはお", "ok": 0}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまざいごうはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "x-ai/grok-build-0.1": {"score": 14, "answered": 20, "first10": 6, "last10": 8, "inconsistent_pairs": 2, "sec": 21.2, "out_tokens": 3409, "price": "$1 / $2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-coder-next": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 3.1, "out_tokens": 153, "price": "$0.12 / $0.8", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "月曜", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すめでれはははょうわし", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 母親  \n2. 10  \n3. しない  \n4. synchronization  \n5. 4  \n6. 3  \n7. できる  \n8. 月曜  \n9. d  \n10. 水星  \n11. できる  \n12. すめでれはははょうわし  \n13. 母親  \n14. 同じ  \n15. 5  \n16. 4  \n17. 14文字、1つ  \n18. 4  \n19. 0.25, 0.3, 0.33, 1/3  \n20. 存在する"}, "openai/gpt-5.3-codex": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 16.3, "out_tokens": 1297, "price": "$1.75 / $14", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "bytedance-seed/seed-2.0-lite": {"score": 13, "answered": 20, "first10": 6, "last10": 7, "inconsistent_pairs": 0, "sec": 24.1, "out_tokens": 1719, "price": "$0.25 / $2", "graded": {"q01": {"a": "少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "upstage/solar-pro-3": {"score": 11, "answered": 20, "first10": 6, "last10": 5, "inconsistent_pairs": 2, "sec": 12.1, "out_tokens": 1734, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すんいざごうはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15文字で2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できる  \n12: すんいざごうはお  \n13: 父親  \n14: 変える  \n15: 5  \n16: 2  \n17: 15文字で2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "writer/palmyra-x5": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 1, "sec": 3.0, "out_tokens": 149, "price": "$0.6 / $6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すますようはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字,1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,1/3,0.3,0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: すますようはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 14文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"}, "openai/gpt-5.2-codex": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 33.6, "out_tokens": 1511, "price": "$1.75 / $14", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字・2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "bytedance-seed/seed-1.6-flash": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 0, "sec": 8.3, "out_tokens": 887, "price": "$0.075 / $0.3", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまございようはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "13文字、2つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまございようはお\n13: 母親\n14: 変える\n15: 5\n16: 2\n17: 13文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "mistralai/devstral-2512": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 0, "sec": 2.8, "out_tokens": 145, "price": "$0.4 / $2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "p", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ます。はちおお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ます。はちおお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "openai/gpt-5.2": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 0, "sec": 3.9, "out_tokens": 134, "price": "$1.75 / $14", "graded": {"q01": {"a": "少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、2つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、2つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "mistralai/ministral-14b-2512": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 0, "sec": 3.3, "out_tokens": 146, "price": "$0.2 / $0.2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "火曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますごうはうお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 火曜\n9: s\n10: 水星\n11: できる\n12: ますごうはうお\n13: 母\n14: 変える\n15: 10\n16: 3\n17: 13文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "mistralai/ministral-8b-2512": {"score": 4, "answered": 20, "first10": 3, "last10": 1, "inconsistent_pairs": 3, "sec": 3.3, "out_tokens": 142, "price": "$0.15 / $0.15", "graded": {"q01": {"a": "少年", "ok": 0}, "q02": {"a": "1", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "火曜", "ok": 0}, "q09": {"a": "i", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "まつごうよ", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "19文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 少年\n2. 1\n3. しない\n4. extraordinarily\n5. 3\n6. 0\n7. できない\n8. 火曜\n9. i\n10. 水星\n11. できる\n12. まつごうよ\n13. 父\n14. 変える\n15. 20\n16. 2\n17. 19文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "mistralai/ministral-3b-2512": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 2, "sec": 3.4, "out_tokens": 259, "price": "$0.1 / $0.1", "graded": {"q01": {"a": "番号: 少年が外科医の息子", "ok": 0}, "q02": {"a": "番号: 20", "ok": 0}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: synchronization (11文字)", "ok": 0}, "q05": {"a": "番号: 2", "ok": 0}, "q06": {"a": "番号: 12", "ok": 0}, "q07": {"a": "番号: できる", "ok": 0}, "q08": {"a": "番号: 火曜日", "ok": 0}, "q09": {"a": "番号: e", "ok": 0}, "q10": {"a": "番号: 火星", "ok": 1}, "q11": {"a": "番号: できる", "ok": 0}, "q12": {"a": "番号: すごはお", "ok": 0}, "q13": {"a": "番号: 父親", "ok": 1}, "q14": {"a": "番号: 同じ", "ok": 1}, "q15": {"a": "番号: 10", "ok": 0}, "q16": {"a": "番号: 2", "ok": 0}, "q17": {"a": "番号: 19文字、1つ(\"extraordinarily\")", "ok": 0}, "q18": {"a": "番号: 3", "ok": 1}, "q19": {"a": "番号: 0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "番号: 存在する（手順：農夫+キャベツ→オオカミ→農夫+ヤギ→キャベツ→オオカミ→農夫+キャベツ）", "ok": 0}}, "raw": "1. 番号: 少年が外科医の息子\n2. 番号: 20\n3. 番号: しない\n4. 番号: synchronization (11文字)\n5. 番号: 2\n6. 番号: 12\n7. 番号: できる\n8. 番号: 火曜日\n9. 番号: e\n10. 番号: 火星\n11. 番号: できる\n12. 番号: すごはお\n13. 番号: 父親\n14. 番号: 同じ\n15. 番号: 10\n16. 番号: 2\n17. 番号: 19文字、1つ(\"extraordinarily\")\n18. 番号: 3\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する（手順：農夫+キャベツ→オオカミ→農夫+ヤギ→キャベツ→オオカミ→農夫+キャベツ）"}, "mistralai/mistral-large-2512": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 3.1, "out_tokens": 146, "price": "$0.5 / $1.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "1", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すはようごおは", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはようごおは\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "bytedance-seed/seed-1.6": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 28.6, "out_tokens": 1496, "price": "$0.25 / $2", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1個", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 14文字、1個\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "openai/gpt-5.1": {"score": 12, "answered": 20, "first10": 7, "last10": 5, "inconsistent_pairs": 1, "sec": 2.4, "out_tokens": 145, "price": "$1.25 / $10", "graded": {"q01": {"a": "父親または母親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親または母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 親子\n14: 同じ\n15: 10\n16: 3\n17: 14文字で1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "amazon/nova-2-lite-v1": {"score": 12, "answered": 20, "first10": 7, "last10": 5, "inconsistent_pairs": 0, "sec": 19.8, "out_tokens": 2872, "price": "$0.3 / $2.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "16,1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.33, 0.3", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 2\n17: 16,1\n18: 4\n19: 0.25, 1/3, 0.33, 0.3\n20: しない"}, "amazon/nova-premier-v1": {"score": 10, "answered": 20, "first10": 7, "last10": 3, "inconsistent_pairs": 0, "sec": 2.9, "out_tokens": 149, "price": "$2.5 / $12.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "c", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すございますおはよう", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10年", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 父親\n2. 10円\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できる\n8. 水曜日\n9. c\n10. 火星\n11. できる\n12. すございますおはよう\n13. 父親\n14. 変える\n15. 10年\n16. 3\n17. 14文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "qwen/qwen3-vl-32b-instruct": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 0, "sec": 3.9, "out_tokens": 159, "price": "$0.104 / $0.416", "graded": {"q01": {"a": "外科医は母親の夫（父親）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すいごうあおは", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医は母親の夫（父親）  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 3  \n7: できる  \n8: 土曜日  \n9: o  \n10: 水星  \n11: できる  \n12: すいごうあおは  \n13: 父親  \n14: 同じ  \n15: 10  \n16: 4  \n17: 14文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "ibm-granite/granite-4.0-h-micro": {"score": 6, "answered": 20, "first10": 3, "last10": 3, "inconsistent_pairs": 2, "sec": 5.0, "out_tokens": 128, "price": "$0.017 / $0.112", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "6", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "Wednesday", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "Earth", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ですございます", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "1", "ok": 0}, "q17": {"a": "complicated", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医\n2: 20\n3: しない\n4: extraordinarily\n5: 2\n6: 6\n7: できない\n8: Wednesday\n9: o\n10: Earth\n11: できる\n12: ですございます\n13: 父\n14: 同じ\n15: 10\n16: 1\n17: complicated\n18: 2\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "qwen/qwen3-vl-8b-instruct": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 1, "sec": 2.2, "out_tokens": 147, "price": "$0.117 / $0.455", "graded": {"q01": {"a": "母親の夫", "ok": 0}, "q02": {"a": "5", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "月曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すございますおはよう", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親の夫  \n2: 5  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 月曜  \n9: o  \n10: 金星  \n11: できる  \n12: すございますおはよう  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "openai/gpt-5.1-codex": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 34.9, "out_tokens": 1851, "price": "$1.25 / $10", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・1語", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字・1語\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"}, "anthropic/claude-haiku-4.5": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 0, "sec": 28.1, "out_tokens": 3783, "price": "$1 / $5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1個", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 16文字、1個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-vl-30b-a3b-instruct": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 0, "sec": 2.5, "out_tokens": 156, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "外科医は母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまそなようはお", "ok": 0}, "q13": {"a": "執刀医は母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医は母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 2  \n6: 3  \n7: できる  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できる  \n12: すまそなようはお  \n13: 執刀医は母親  \n14: 変える  \n15: 5  \n16: 2  \n17: 16文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "thedrummer/cydonia-24b-v4.1": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 1, "sec": 4.7, "out_tokens": 151, "price": "$0.3 / $0.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す ご う よ う は お", "ok": 0}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "17文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: s\n10: 水星\n11: できる\n12: す ご う よ う は お\n13: 父と息子\n14: 変える\n15: 10\n16: 3\n17: 17文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "qwen/qwen3-coder-plus": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 0, "sec": 4.1, "out_tokens": 147, "price": "$0.65 / $3.25", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すにがもはお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "17文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できない  \n12: すにがもはお  \n13: 母  \n14: 変える  \n15: 5  \n16: 3  \n17: 17文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "anthropic/claude-sonnet-4.5": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 1, "sec": 14.5, "out_tokens": 835, "price": "$3 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいござうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-vl-235b-a22b-instruct": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 0, "sec": 3.8, "out_tokens": 145, "price": "$0.21 / $1.9", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "ございますようはお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 0\n7: できない\n8: 土曜日\n9: s\n10: 水星\n11: できない\n12: ございますようはお\n13: 母\n14: 同じ\n15: 5\n16: 4\n17: 13文字、1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"}, "qwen/qwen3-coder-flash": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 1, "sec": 3.0, "out_tokens": 153, "price": "$0.195 / $0.975", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すみどうはおはようございまし", "ok": 0}, "q13": {"a": "母亲", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 金星  \n11: できる  \n12: すみどうはおはようございまし  \n13: 母亲  \n14: 変える  \n15: 5  \n16: 3  \n17: 13文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "qwen/qwen3-next-80b-a3b-instruct": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 1.6, "out_tokens": 147, "price": "$0.09 / $1.1", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "マーズ", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すですははようお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できる  \n8: 木曜  \n9: d  \n10: マーズ  \n11: できる  \n12: すですははようお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 4  \n17: 15文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "qwen/qwen-plus-2025-07-28": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 1, "sec": 2.8, "out_tokens": 156, "price": "$0.26 / $0.78", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す　は　ご　う　よ　は　お", "ok": 0}, "q13": {"a": "息子の母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字,1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,1/3,0.3,0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: す　は　ご　う　よ　は　お  \n13: 息子の母  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"}, "mistralai/mistral-medium-3.1": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 0, "sec": 2.4, "out_tokens": 145, "price": "$0.4 / $2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "p", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますはちおはよう", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "moonshotai/kimi-k2-0905": {"score": 8, "answered": 20, "first10": 4, "last10": 4, "inconsistent_pairs": 0, "sec": 5.4, "out_tokens": 136, "price": "$0.6 / $2.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざいうごうやはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: すまいざいうごうやはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-vl-30b-a3b-thinking": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 0, "sec": 25.9, "out_tokens": 3241, "price": "$0.2 / $2.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "O", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: O\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15, 1\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "openai/gpt-5-mini": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 75.1, "out_tokens": 4807, "price": "$0.25 / $2", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15,2\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "openai/gpt-5": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 1, "sec": 78.7, "out_tokens": 123, "price": "$1.25 / $10", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "c", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますございますようお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: c\n10: 水星\n11: できる\n12: ますございますようお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "qwen/qwen3-30b-a3b-thinking-2507": {"score": 13, "answered": 20, "first10": 7, "last10": 6, "inconsistent_pairs": 0, "sec": 26.0, "out_tokens": 4366, "price": "$0.2 / $2.4", "graded": {"q01": {"a": "母", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざうごうよはお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざうごうよはお\n13: 母\n14: 同じ\n15: 5\n16: 3\n17: 15 1\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "moonshotai/kimi-k2-thinking": {"score": 15, "answered": 20, "first10": 7, "last10": 8, "inconsistent_pairs": 1, "sec": 13.8, "out_tokens": 2845, "price": "$0.6 / $2.5", "graded": {"q01": {"a": "母", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざいごうよはお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字,1個", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できない\n12: すまいざいごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 14文字,1個\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "openai/gpt-oss-120b": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 2, "sec": 20.9, "out_tokens": 1130, "price": "$0.037 / $0.17", "graded": {"q01": {"a": "母", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 2", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できる  \n12: すまいざごうはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15, 2  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "qwen/qwen3-30b-a3b-instruct-2507": {"score": 6, "answered": 20, "first10": 3, "last10": 3, "inconsistent_pairs": 0, "sec": 3.8, "out_tokens": 154, "price": "$0.04815 / $0.19305", "graded": {"q01": {"a": "女性の外科医", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すいませんはよおうお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 女性の外科医  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜日  \n9: s  \n10: 水星  \n11: できる  \n12: すいませんはよおうお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 16文字、1つ  \n18: 3  \n19: 0.25, 1/3, 0.3, 0.33  \n20: 存在する"}, "qwen/qwen3-235b-a22b-2507": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 1, "sec": 2.7, "out_tokens": 156, "price": "$0.22 / $0.88", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す　は　ご　う　よ　は　お", "ok": 0}, "q13": {"a": "息子の母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字,1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,1/3,0.3,0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: す　は　ご　う　よ　は　お  \n13: 息子の母  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"}, "google/gemini-2.5-flash-lite": {"score": 8, "answered": 20, "first10": 6, "last10": 2, "inconsistent_pairs": 2, "sec": 15.7, "out_tokens": 5908, "price": "$0.1 / $0.4", "graded": {"q01": {"a": "外科医は少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す、い、ま、せ、ん、ご、は、お", "ok": 0}, "q13": {"a": "息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "2", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "する", "ok": 0}}, "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 2\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す、い、ま、せ、ん、ご、は、お\n13: 息子\n14: 変える\n15: 2\n16: 3\n17: 14文字、1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: する"}, "qwen/qwen3-vl-8b-thinking": {"score": 11, "answered": 20, "first10": 7, "last10": 4, "inconsistent_pairs": 0, "sec": 39.1, "out_tokens": 5161, "price": "$0.18 / $2.1", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいまぞうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15,1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 2\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまいまぞうよはお\n13: 父親\n14: 変える\n15: 5\n16: 2\n17: 15,1\n18: 2\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "openai/gpt-oss-20b": {"score": 13, "answered": 20, "first10": 6, "last10": 7, "inconsistent_pairs": 0, "sec": 12.7, "out_tokens": 1485, "price": "$0.03 / $0.13", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "14", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父  \n2: 10  \n3: しない  \n4: 14  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できない  \n12: すまいざごうよはお  \n13: 父  \n14: 変える  \n15: 5  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "cognitivecomputations/dolphin-mistral-24b-venice-edition": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 0, "sec": 2.8, "out_tokens": 155, "price": "$0.2 / $0.9", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "6", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すごむ よあう ごはお", "ok": 0}, "q13": {"a": "母と息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "19文字、1単語", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 2\n6: 6\n7: できる\n8: 土曜日\n9: o\n10: 水星\n11: できない\n12: すごむ よあう ごはお\n13: 母と息子\n14: 変える\n15: 10\n16: 3\n17: 19文字、1単語\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "moonshotai/kimi-k2": {"score": 9, "answered": 20, "first10": 5, "last10": 4, "inconsistent_pairs": 1, "sec": 9.2, "out_tokens": 135, "price": "$0.57 / $2.3", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざうよごはうお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: すまいざうよごはうお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-2.5-flash": {"score": 15, "answered": 20, "first10": 7, "last10": 8, "inconsistent_pairs": 1, "sec": 30.2, "out_tokens": 5754, "price": "$0.3 / $2.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざうごよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1個", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できない\n12. すまいざうごよはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字, 1個\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"}, "anthropic/claude-opus-4.1": {"score": 14, "answered": 20, "first10": 6, "last10": 8, "inconsistent_pairs": 1, "sec": 30.4, "out_tokens": 967, "price": "$15 / $75", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2単語", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2単語\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-next-80b-a3b-thinking": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 1, "sec": 41.0, "out_tokens": 10613, "price": "$0.15 / $1.2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "5", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 5  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15,2  \n18: 5  \n19: 0.25,0.3,0.33,1/3  \n20: しない"}, "google/gemini-2.5-pro": {"score": 11, "answered": 20, "first10": 8, "last10": 3, "inconsistent_pairs": 1, "sec": 32.5, "out_tokens": 137, "price": "$1.25 / $10", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "mistralai/mistral-medium-3": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 0, "sec": 3.1, "out_tokens": 145, "price": "$0.4 / $2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "0", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "p", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ますはちおはよう", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "baidu/ernie-4.5-vl-424b-a47b": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 1, "sec": 11.4, "out_tokens": 360, "price": "$0.42 / $1.25", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまざうごはお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "17文字、1単語", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できる  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまざうごはお  \n13: 父  \n14: 変える  \n15: 5  \n16: 3  \n17: 17文字、1単語  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "anthropic/claude-sonnet-4": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 18.6, "out_tokens": 1730, "price": "$3 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "anthropic/claude-opus-4": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 25.0, "out_tokens": 860, "price": "$15 / $75", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3-14b": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 1, "sec": 40.4, "out_tokens": 1967, "price": "$0.2275 / $0.91", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "せいざごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できる  \n12: せいざごうよはお  \n13: 父親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字, 1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-4.1": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 0, "sec": 2.3, "out_tokens": 131, "price": "$2 / $8", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 2  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいざごうはお  \n13: 母親  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-4.1-mini": {"score": 10, "answered": 20, "first10": 5, "last10": 5, "inconsistent_pairs": 0, "sec": 25.3, "out_tokens": 132, "price": "$0.4 / $1.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-4.1-nano": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 0, "sec": 22.6, "out_tokens": 134, "price": "$0.1 / $0.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜日", "ok": 0}, "q09": {"a": "5", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいございますおは", "ok": 0}, "q13": {"a": "母と息子の関係", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "6", "ok": 0}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "11、2", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.33, 0.3, 1/3", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 金曜日  \n9: 5  \n10: 金星  \n11: できる  \n12: すまいございますおは  \n13: 母と息子の関係  \n14: 変える  \n15: 6  \n16: 4  \n17: 11、2  \n18: 3  \n19: 0.25, 0.33, 0.3, 1/3  \n20: しない"}, "google/gemma-3-4b-it": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 1, "sec": 17.5, "out_tokens": 144, "price": "$0.05 / $0.1", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土", "ok": 0}, "q09": {"a": "p", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ょうめしがさうも", "ok": 0}, "q13": {"a": "息子と父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "30", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "18, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1.  父\n2.  20\n3.  しない\n4.  synchronization\n5.  2\n6.  3\n7.  できない\n8.  土\n9.  p\n10. 火星\n11. できる\n12. ょうめしがさうも\n13. 息子と父\n14. 変える\n15. 30\n16. 3\n17. 18, 1\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "deepseek/deepseek-chat-v3-0324": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 18.0, "out_tokens": 131, "price": "$0.29 / $1.14", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまざよごうやはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "5", "ok": 0}, "q17": {"a": "14文字, 1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: s\n10: 火星\n11: できる\n12: すまざよごうやはお\n13: 母親\n14: 変える\n15: 5\n16: 5\n17: 14文字, 1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "google/gemma-3-12b-it": {"score": 4, "answered": 20, "first10": 3, "last10": 1, "inconsistent_pairs": 1, "sec": 14.2, "out_tokens": 134, "price": "$0.05 / $0.15", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "1", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "月曜日", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すございますおはよう", "ok": 0}, "q13": {"a": "医者", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 1\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 月曜日\n9: e\n10: 金星\n11: できる\n12: すございますおはよう\n13: 医者\n14: 変える\n15: 10\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "openai/o4-mini": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 2, "sec": 36.4, "out_tokens": 3753, "price": "$1.1 / $4.4", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15,2  \n18: 3  \n19: 0.25,0.3,0.33,1/3  \n20: しない"}, "openai/o3": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 30.0, "out_tokens": 4194, "price": "$2 / $8", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15,2", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15,2\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "thedrummer/skyfall-36b-v2": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 3, "sec": 3.3, "out_tokens": 148, "price": "$0.55 / $0.8", "graded": {"q01": {"a": "その少年", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "6", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すますごようは", "ok": 0}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. その少年\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 6\n7. できない\n8. 土曜日\n9. d\n10. 水星\n11. できない\n12. すますごようは\n13. 父と息子\n14. 変える\n15. 10\n16. 3\n17. 16文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "mistralai/mistral-saba": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 1, "sec": 2.3, "out_tokens": 145, "price": "$0.2 / $0.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すもよはごう", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "12, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すもよはごう\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 12, 1\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "qwen/qwen2.5-vl-72b-instruct": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 1, "sec": 4.7, "out_tokens": 150, "price": "$0.8 / $1", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すみませんごようはお", "ok": 0}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 土曜日\n9: o\n10: マーキュリー\n11: できる\n12: すみませんごようはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "aion-labs/aion-rp-llama-3.1-8b": {"score": 6, "answered": 20, "first10": 3, "last10": 3, "inconsistent_pairs": 0, "sec": 14.3, "out_tokens": 537, "price": "$0.8 / $1.6", "graded": {"q01": {"a": "The surgeon who claimed to be the boy's father is actually the mother's husband, the boy's stepfather.", "ok": 1}, "q02": {"a": "The bat is 30 yen and the ball is 10 yen.", "ok": 0}, "q03": {"a": "2026年2月29日は存在しない。leap yearの条件を満たさないため。", "ok": 1}, "q04": {"a": "\"complicated\"", "ok": 0}, "q05": {"a": "ちょうちょは5モーラ。", "ok": 0}, "q06": {"a": "サリーの姉妹は3×3=9人。", "ok": 0}, "q07": {"a": "農夫・オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶことはできない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じる。", "ok": 1}, "q08": {"a": "今日が月曜なら、100日後は火曜日。", "ok": 0}, "q09": {"a": "kaleidoscope の7文字目は s。", "ok": 0}, "q10": {"a": "水星。", "ok": 0}, "q11": {"a": "2Lと4Lの容器だけで、ちょうど3Lを測ることはできない。3Lの倍数以外の容量を測るため。", "ok": 1}, "q12": {"a": "\"ございますおはよう\"。", "ok": 0}, "q13": {"a": "父は執刀医で、子が10歳の時点で亡くなっている。", "ok": 1}, "q14": {"a": "3つの扉のうち、Aが当たり扉で、BとCはハズレ。司会者がランダムにBを開けても、最終的にはAを開けてもらうのが最善の戦略。", "ok": 0}, "q15": {"a": "父が子のちょうど3倍になるのは10年後。", "ok": 0}, "q16": {"a": "「わたしはきょうははれです」に「は」は2個ある。", "ok": 0}, "q17": {"a": "\"complicated\"が13文字で、該当する単語は1つ。", "ok": 0}, "q18": {"a": "「ちょうちょ」を拍(モーラ)で区切ると2+3+3+3+2=13に分かれる。", "ok": 0}, "q19": {"a": "0.3 < 0.25 < 1/3 < 0.33", "ok": 0}, "q20": {"a": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在しない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じ、全員を無事に運ぶことができない。", "ok": 1}}, "raw": "1. The surgeon who claimed to be the boy's father is actually the mother's husband, the boy's stepfather.\n2. The bat is 30 yen and the ball is 10 yen.\n3. 2026年2月29日は存在しない。leap yearの条件を満たさないため。\n4. \"complicated\"\n5. ちょうちょは5モーラ。\n6. サリーの姉妹は3×3=9人。\n7. 農夫・オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶことはできない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じる。\n8. 今日が月曜なら、100日後は火曜日。\n9. kaleidoscope の7文字目は s。\n10. 水星。\n11. 2Lと4Lの容器だけで、ちょうど3Lを測ることはできない。3Lの倍数以外の容量を測るため。\n12. \"ございますおはよう\"。\n13. 父は執刀医で、子が10歳の時点で亡くなっている。\n14. 3つの扉のうち、Aが当たり扉で、BとCはハズレ。司会者がランダムにBを開けても、最終的にはAを開けてもらうのが最善の戦略。\n15. 父が子のちょうど3倍になるのは10年後。\n16. 「わたしはきょうははれです」に「は」は2個ある。\n17. \"complicated\"が13文字で、該当する単語は1つ。\n18. 「ちょうちょ」を拍(モーラ)で区切ると2+3+3+3+2=13に分かれる。\n19. 0.3 < 0.25 < 1/3 < 0.33\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在しない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じ、全員を無事に運ぶことができない。"}, "mistralai/mistral-small-24b-instruct-2501": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 1, "sec": 6.7, "out_tokens": 146, "price": "$0.05 / $0.08", "graded": {"q01": {"a": "その母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "6", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すうごはよ", "ok": 0}, "q13": {"a": "同性愛者", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "18", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "19, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: その母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 6\n7: できない\n8: 木曜\n9: d\n10: 水星\n11: できない\n12: すうごはよ\n13: 同性愛者\n14: 変える\n15: 18\n16: 3\n17: 19, 1\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "qwen/qwen-plus": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 1, "sec": 3.7, "out_tokens": 148, "price": "$0.26 / $0.78", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ございますようはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "16文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 2  \n7: できない  \n8: 木曜日  \n9: o  \n10: 火星  \n11: できる  \n12: ございますようはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 4  \n17: 16文字, 1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "cohere/command-r7b-12-2024": {"score": 5, "answered": 20, "first10": 2, "last10": 3, "inconsistent_pairs": 0, "sec": 4.0, "out_tokens": 184, "price": "$0.0375 / $0.15", "graded": {"q01": {"a": "外科医は「この子は私の息子だ」と言った。外科医は親。", "ok": 0}, "q02": {"a": "50円。", "ok": 0}, "q03": {"a": "する。", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2。", "ok": 0}, "q06": {"a": "9人。", "ok": 0}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日。", "ok": 1}, "q09": {"a": "p。", "ok": 0}, "q10": {"a": "水星。", "ok": 0}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すおようございます。", "ok": 0}, "q13": {"a": "執刀医は親。", "ok": 0}, "q14": {"a": "変える。", "ok": 0}, "q15": {"a": "10年後。", "ok": 0}, "q16": {"a": "2個。", "ok": 0}, "q17": {"a": "12文字、4個。", "ok": 0}, "q18": {"a": "2。", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3。", "ok": 1}, "q20": {"a": "しない。", "ok": 1}}, "raw": "1. 外科医は「この子は私の息子だ」と言った。外科医は親。\n2. 50円。\n3. する。\n4.  synchronization\n5. 2。\n6. 9人。\n7. できない。\n8. 水曜日。\n9. p。\n10. 水星。\n11. できない。\n12. すおようございます。\n13. 執刀医は親。\n14. 変える。\n15. 10年後。\n16. 2個。\n17. 12文字、4個。\n18. 2。\n19. 0.25, 0.3, 0.33, 1/3。\n20. しない。"}, "sao10k/l3.3-euryale-70b": {"score": 7, "answered": 20, "first10": 5, "last10": 2, "inconsistent_pairs": 1, "sec": 11.3, "out_tokens": 190, "price": "$0.65 / $0.75", "graded": {"q01": {"a": "1. 母親", "ok": 0}, "q02": {"a": "2. 10", "ok": 1}, "q03": {"a": "3. しない", "ok": 1}, "q04": {"a": "4. synchronization", "ok": 0}, "q05": {"a": "5. 3", "ok": 1}, "q06": {"a": "6. 2", "ok": 1}, "q07": {"a": "7. できる", "ok": 0}, "q08": {"a": "8. 金曜日", "ok": 0}, "q09": {"a": "9. o", "ok": 1}, "q10": {"a": "10. 金星", "ok": 0}, "q11": {"a": "11. できる", "ok": 0}, "q12": {"a": "12. すむざいおはよ", "ok": 0}, "q13": {"a": "13. 母", "ok": 0}, "q14": {"a": "14. 変える", "ok": 0}, "q15": {"a": "15. 10", "ok": 0}, "q16": {"a": "16. 2", "ok": 0}, "q17": {"a": "17. 17文字、1", "ok": 0}, "q18": {"a": "18. 3", "ok": 1}, "q19": {"a": "19. 0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "20. しない", "ok": 1}}, "raw": "1: 1. 母親\n2: 2. 10\n3: 3. しない\n4: 4. synchronization\n5: 5. 3\n6: 6. 2\n7: 7. できる\n8: 8. 金曜日\n9: 9. o\n10: 10. 金星\n11: 11. できる\n12: 12. すむざいおはよ\n13: 13. 母\n14: 14. 変える\n15: 15. 10\n16: 16. 2\n17: 17. 17文字、1\n18: 18. 3\n19: 19. 0.25, 0.3, 1/3, 0.33\n20: 20. しない"}, "meta-llama/llama-3.3-70b-instruct": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 0, "sec": 3.5, "out_tokens": 134, "price": "$0.1 / $0.32", "graded": {"q01": {"a": "外科医は少年の母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すすおはよ", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "17文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できる\n8: 木曜日\n9: e\n10: 水星\n11: できる\n12: すすおはよ\n13: 母\n14: 変える\n15: 10\n16: 3\n17: 17文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "amazon/nova-lite-v1": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 1, "sec": 2.9, "out_tokens": 201, "price": "$0.06 / $0.24", "graded": {"q01": {"a": "番号: 外科医", "ok": 0}, "q02": {"a": "番号: 10", "ok": 1}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: synchronization", "ok": 0}, "q05": {"a": "番号: 3", "ok": 1}, "q06": {"a": "番号: 3", "ok": 0}, "q07": {"a": "番号: できる", "ok": 0}, "q08": {"a": "番号: 日", "ok": 0}, "q09": {"a": "番号: o", "ok": 1}, "q10": {"a": "番号: 火星", "ok": 1}, "q11": {"a": "番号: できる", "ok": 0}, "q12": {"a": "番号: すみようお", "ok": 0}, "q13": {"a": "番号: 父", "ok": 1}, "q14": {"a": "番号: 変える", "ok": 0}, "q15": {"a": "番号: 20", "ok": 0}, "q16": {"a": "番号: 3", "ok": 1}, "q17": {"a": "番号: 11, synchronization", "ok": 0}, "q18": {"a": "番号: 3", "ok": 1}, "q19": {"a": "番号: 0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "番号: 存在する", "ok": 0}}, "raw": "1. 番号: 外科医\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 3\n6. 番号: 3\n7. 番号: できる\n8. 番号: 日\n9. 番号: o\n10. 番号: 火星\n11. 番号: できる\n12. 番号: すみようお\n13. 番号: 父\n14. 番号: 変える\n15. 番号: 20\n16. 番号: 3\n17. 番号: 11, synchronization\n18. 番号: 3\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する"}, "amazon/nova-micro-v1": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 1, "sec": 2.4, "out_tokens": 204, "price": "$0.035 / $0.14", "graded": {"q01": {"a": "番号: 母親", "ok": 0}, "q02": {"a": "番号: 20円", "ok": 0}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: complicated", "ok": 0}, "q05": {"a": "番号: 3", "ok": 1}, "q06": {"a": "番号: 9", "ok": 0}, "q07": {"a": "番号: できない", "ok": 1}, "q08": {"a": "番号: 火曜日", "ok": 0}, "q09": {"a": "番号: o", "ok": 1}, "q10": {"a": "番号: 水星", "ok": 0}, "q11": {"a": "番号: できない", "ok": 1}, "q12": {"a": "番号: ますおはよう", "ok": 0}, "q13": {"a": "番号: 父親", "ok": 1}, "q14": {"a": "番号: 同じ", "ok": 1}, "q15": {"a": "番号: 20年", "ok": 0}, "q16": {"a": "番号: 2", "ok": 0}, "q17": {"a": "番号: 15文字、1", "ok": 0}, "q18": {"a": "番号: 3", "ok": 1}, "q19": {"a": "番号: 0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "番号: しない", "ok": 1}}, "raw": "1. 番号: 母親\n2. 番号: 20円\n3. 番号: しない\n4. 番号: complicated\n5. 番号: 3\n6. 番号: 9\n7. 番号: できない\n8. 番号: 火曜日\n9. 番号: o\n10. 番号: 水星\n11. 番号: できない\n12. 番号: ますおはよう\n13. 番号: 父親\n14. 番号: 同じ\n15. 番号: 20年\n16. 番号: 2\n17. 番号: 15文字、1\n18. 番号: 3\n19. 番号: 0.25, 1/3, 0.3, 0.33\n20. 番号: しない"}, "deepseek/deepseek-chat": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 10.8, "out_tokens": 129, "price": "$0.32 / $0.89", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまざうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "5", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 母親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 3\n7. できる\n8. 水曜日\n9. e\n10. 火星\n11. できる\n12. すまざうよはお\n13. 母親\n14. 変える\n15. 5\n16. 5\n17. 15文字、1つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"}, "openai/o3-mini": {"score": 15, "answered": 20, "first10": 8, "last10": 7, "inconsistent_pairs": 0, "sec": 13.8, "out_tokens": 2764, "price": "$1.1 / $4.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16,1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16,1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "amazon/nova-pro-v1": {"score": 4, "answered": 20, "first10": 3, "last10": 1, "inconsistent_pairs": 0, "sec": 1.9, "out_tokens": 209, "price": "$0.8 / $3.2", "graded": {"q01": {"a": "番号: 外科医は母親", "ok": 0}, "q02": {"a": "番号: 10", "ok": 1}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: synchronization", "ok": 0}, "q05": {"a": "番号: 4", "ok": 0}, "q06": {"a": "番号: 3", "ok": 0}, "q07": {"a": "番号: できない", "ok": 1}, "q08": {"a": "番号: 木曜日", "ok": 0}, "q09": {"a": "番号: p", "ok": 0}, "q10": {"a": "番号: 水星", "ok": 0}, "q11": {"a": "番号: できる", "ok": 0}, "q12": {"a": "番号: すうございますはようお", "ok": 0}, "q13": {"a": "番号: 母親", "ok": 0}, "q14": {"a": "番号: 変える", "ok": 0}, "q15": {"a": "番号: 20", "ok": 0}, "q16": {"a": "番号: 4", "ok": 0}, "q17": {"a": "番号: 19, 1", "ok": 0}, "q18": {"a": "番号: 4", "ok": 0}, "q19": {"a": "番号: 0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "番号: しない", "ok": 1}}, "raw": "1. 番号: 外科医は母親\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 4\n6. 番号: 3\n7. 番号: できない\n8. 番号: 木曜日\n9. 番号: p\n10. 番号: 水星\n11. 番号: できる\n12. 番号: すうございますはようお\n13. 番号: 母親\n14. 番号: 変える\n15. 番号: 20\n16. 番号: 4\n17. 番号: 19, 1\n18. 番号: 4\n19. 番号: 0.25, 1/3, 0.3, 0.33\n20. 番号: しない"}, "openai/o1": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 1, "sec": 43.9, "out_tokens": 5777, "price": "$15 / $60", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で2つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 14文字で2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"}, "thedrummer/unslopnemo-12b": {"score": 3, "answered": 20, "first10": 2, "last10": 1, "inconsistent_pairs": 1, "sec": 5.1, "out_tokens": 147, "price": "$0.4 / $0.4", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "火曜日", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "土星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ます、ごう、よう、お", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "25文字、1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 父\n2: 20\n3: する\n4: synchronization\n5: 2\n6: 3\n7: できない\n8: 火曜日\n9: e\n10: 土星\n11: できる\n12: ます、ごう、よう、お\n13: 父\n14: 変える\n15: 20\n16: 2\n17: 25文字、1\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "mistralai/mistral-large-2407": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 4.8, "out_tokens": 150, "price": "$2 / $6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "1", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すはまいざおごうよお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはまいざおごうよお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "meta-llama/llama-3.2-1b-instruct": {"score": 3, "answered": 20, "first10": 1, "last10": 2, "inconsistent_pairs": 0, "sec": 2.3, "out_tokens": 116, "price": "$0.027 / $0.201", "graded": {"q01": {"a": "医師", "ok": 0}, "q02": {"a": "120", "ok": 0}, "q03": {"a": "あり", "ok": 0}, "q04": {"a": "あり", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "1", "ok": 0}, "q09": {"a": "1", "ok": 0}, "q10": {"a": "さくら", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "え", "ok": 0}, "q13": {"a": "あり", "ok": 0}, "q14": {"a": "あり", "ok": 0}, "q15": {"a": "3", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "3", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "1/3、1/2、1/3、1/4", "ok": 0}, "q20": {"a": "あり", "ok": 0}}, "raw": "1.  医師\n2. 120\n3. あり\n4. あり\n5. 3\n6. 3\n7. できる\n8. 1\n9. 1\n10. さくら\n11. できる\n12. え\n13. あり\n14. あり\n15. 3\n16. 3\n17. 3\n18. 3\n19. 1/3、1/2、1/3、1/4\n20. あり"}, "meta-llama/llama-3.2-3b-instruct": {"score": 4, "answered": 20, "first10": 3, "last10": 1, "inconsistent_pairs": 2, "sec": 2.0, "out_tokens": 106, "price": "$0.05 / $0.33", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ごようごようございます", "ok": 0}, "q13": {"a": "兄弟", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "30", "ok": 0}, "q16": {"a": "1", "ok": 0}, "q17": {"a": "synchronization", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25", "ok": 0}, "q20": {"a": "あり", "ok": 0}}, "raw": "1. 外科医\n2. 20\n3. する\n4. synchronization\n5. 5\n6. 9\n7. できない\n8. 水曜日\n9. o\n10. 水星\n11. できる\n12. ごようごようございます\n13. 兄弟\n14. 変える\n15. 30\n16. 1\n17. synchronization\n18. 3\n19. 0.25\n20. あり"}, "openai/gpt-4o-2024-11-20": {"score": 8, "answered": 20, "first10": 6, "last10": 2, "inconsistent_pairs": 1, "sec": 2.8, "out_tokens": 146, "price": "$2.5 / $10", "graded": {"q01": {"a": "外科医は父親以外の親、例えば母親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親以外の親、例えば母親。  \n2. 10  \n3. する  \n4. extraordinarily  \n5. 4  \n6. 2  \n7. できない  \n8. 水曜日  \n9. s  \n10. 火星  \n11. できる  \n12. すまいざごうはお  \n13. 母親  \n14. 変える  \n15. 10  \n16. 3  \n17. 14文字、1つ  \n18. 4  \n19. 0.25, 0.3, 1/3, 0.33  \n20. 存在しない"}, "qwen/qwen-2.5-7b-instruct": {"score": 3, "answered": 20, "first10": 2, "last10": 1, "inconsistent_pairs": 1, "sec": 4.6, "out_tokens": 157, "price": "$0.1 / $0.2", "graded": {"q01": {"a": "外科医は母親の夫", "ok": 0}, "q02": {"a": "5", "ok": 0}, "q03": {"a": "不する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "日曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "地球", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すもももももももものうち", "ok": 0}, "q13": {"a": "母親の息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "10, 2", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医は母親の夫\n2: 5\n3: 不する\n4: synchronization\n5: 3\n6: 9\n7: できる\n8: 日曜\n9: o\n10: 地球\n11: できる\n12: すもももももももものうち\n13: 母親の息子\n14: 変える\n15: 20\n16: 2\n17: 10, 2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "sao10k/l3.1-euryale-70b": {"score": 9, "answered": 20, "first10": 7, "last10": 2, "inconsistent_pairs": 1, "sec": 4.2, "out_tokens": 130, "price": "$0.85 / $0.85", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すもようはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "17文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 2\n7. できる\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すもようはお\n13. 母親\n14. 変える\n15. 10\n16. 2\n17. 17文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. しない"}, "sao10k/l3-lunaris-8b": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 0, "sec": 3.7, "out_tokens": 139, "price": "$0.04 / $0.05", "graded": {"q01": {"a": "外科医は少年の父親。", "ok": 1}, "q02": {"a": "ボールは20円。", "ok": 0}, "q03": {"a": "しない。", "ok": 1}, "q04": {"a": "complicated", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "土曜日", "ok": 0}, "q09": {"a": "i", "ok": 0}, "q10": {"a": "ベスタ", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ございますおは", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "30", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "complicated (14文字) × 2 = 28文字", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "できない", "ok": 1}}, "raw": "1. 外科医は少年の父親。\n2. ボールは20円。\n3. しない。\n4. complicated\n5. 4\n6. 9\n7. できない\n8. 土曜日\n9. i\n10. ベスタ\n11. できる\n12. ございますおは\n13. 父親\n14. 変える\n15. 30\n16. 2\n17. complicated (14文字) × 2 = 28文字\n18. 4\n19. 0.25, 1/3, 0.3, 0.33\n20. できない"}, "meta-llama/llama-3.1-8b-instruct": {"score": 4, "answered": 20, "first10": 3, "last10": 1, "inconsistent_pairs": 0, "sec": 2.0, "out_tokens": 130, "price": "$0.05 / $0.08", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "complicated", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ようござい", "ok": 0}, "q13": {"a": "兄弟", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "1", "ok": 0}, "q17": {"a": "complicated (12文字) 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "できない", "ok": 1}}, "raw": "1. 母親\n2. 20\n3. しない\n4. complicated\n5. 4\n6. 9\n7. できない\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. ようござい\n13. 兄弟\n14. 変える\n15. 10\n16. 1\n17. complicated (12文字) 1つ\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. できない"}, "anthracite-org/magnum-v4-72b": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 1, "sec": 10.4, "out_tokens": 141, "price": "$2.5 / $5", "graded": {"q01": {"a": "母", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13, 2", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 9\n7: できない\n8: 木曜\n9: s\n10: マーキュリー\n11: できる\n12: す\n13: 父\n14: 変える\n15: 5\n16: 3\n17: 13, 2\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"}, "openai/gpt-4o-mini": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 0, "sec": 3.2, "out_tokens": 130, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "a", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すではよお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: a  \n10: 火星  \n11: できる  \n12: すではよお  \n13: 母子  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "openai/gpt-4o-2024-08-06": {"score": 11, "answered": 20, "first10": 8, "last10": 3, "inconsistent_pairs": 1, "sec": 3.0, "out_tokens": 137, "price": "$2.5 / $10", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "openai/gpt-4o-mini-2024-07-18": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 0, "sec": 3.1, "out_tokens": 129, "price": "$0.15 / $0.6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "a", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すではよお", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: a  \n10: 火星  \n11: できる  \n12: すではよお  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "meta-llama/llama-3.1-70b-instruct": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 1, "sec": 3.5, "out_tokens": 130, "price": "$0.4 / $0.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すもおはよ", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "20文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できる\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すもおはよ\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 20文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. しない"}, "qwen/qwen-2.5-72b-instruct": {"score": 10, "answered": 20, "first10": 4, "last10": 6, "inconsistent_pairs": 2, "sec": 8.0, "out_tokens": 143, "price": "$0.36 / $0.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15, 2", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 9\n7: できない\n8: 水曜\n9: d\n10: マーキュリー\n11: できる\n12: す\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15, 2\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"}, "openai/gpt-4o-2024-05-13": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 0, "sec": 4.3, "out_tokens": 134, "price": "$5 / $15", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 母親\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 15文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"}, "google/gemma-2-27b-it": {"score": 2, "answered": 20, "first10": 2, "last10": 0, "inconsistent_pairs": 0, "sec": 4.4, "out_tokens": 139, "price": "$0.65 / $0.65", "graded": {"q01": {"a": "母", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "1", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "金星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す", "ok": 0}, "q13": {"a": "息子と執刀医は親子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "18, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母\n2: 10\n3: する\n4: extraordinarily\n5: 2\n6: 1\n7: できる\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: す\n13: 息子と執刀医は親子\n14: 変える\n15: 20\n16: 2\n17: 18, 1\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "mistralai/mixtral-8x22b-instruct": {"score": 10, "answered": 20, "first10": 6, "last10": 4, "inconsistent_pairs": 1, "sec": 2.4, "out_tokens": 145, "price": "$2 / $6", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すもよはごう", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "12, 1", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 1/3, 0.3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すもよはごう\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 12, 1\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"}, "anthropic/claude-3-haiku": {"score": 2, "answered": 20, "first10": 2, "last10": 0, "inconsistent_pairs": 1, "sec": 5.2, "out_tokens": 153, "price": "$0.25 / $1.25", "graded": {"q01": {"a": "外科医は息子", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "月曜", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "す・ま・い・う・ご・は・お", "ok": 0}, "q13": {"a": "外科医は母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 外科医は息子\n2. 20\n3. する\n4. extraordinarily\n5. 3\n6. 9\n7. できる\n8. 月曜\n9. o\n10. 水星\n11. できる\n12. す・ま・い・う・ご・は・お\n13. 外科医は母親\n14. 変える\n15. 10\n16. 2\n17. 13文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "openai/gpt-3.5-turbo-0613": {"score": 11, "answered": 20, "first10": 6, "last10": 5, "inconsistent_pairs": 0, "sec": 4.2, "out_tokens": 133, "price": "$1 / $2", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-4o": {"score": 12, "answered": 20, "first10": 8, "last10": 4, "inconsistent_pairs": 0, "sec": 3.0, "out_tokens": 141, "price": "$2.5 / $10", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "執刀医は父親。", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "openai/gpt-3.5-turbo-16k": {"score": 12, "answered": 20, "first10": 6, "last10": 6, "inconsistent_pairs": 0, "sec": 4.5, "out_tokens": 132, "price": "$3 / $4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいござうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "openai/gpt-4-turbo": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 0, "sec": 5.7, "out_tokens": 152, "price": "$10 / $30", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまざいごうはよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "17文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: o\n10: マーキュリー\n11: できる\n12: すまざいごうはよはお\n13: 母親\n14: 変える\n15: 5\n16: 4\n17: 17文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-3.5-turbo-instruct": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 1, "sec": 3.5, "out_tokens": 145, "price": "$1.5 / $2", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "プルート", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すいようざごお", "ok": 0}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "20", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "extraordinarily, synchronization, complicated, failed", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 9\n7: できない\n8: 木曜\n9: s\n10: プルート\n11: できる\n12: すいようざごお\n13: 父子\n14: 変える\n15: 20\n16: 3\n17: extraordinarily, synchronization, complicated, failed\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "mistralai/mistral-nemo": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 1, "sec": 5.1, "out_tokens": 140, "price": "$0.019 / $0.03", "graded": {"q01": {"a": "Father", "ok": 1}, "q02": {"a": "50", "ok": 0}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "complicated", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "火曜日", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "Mars", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "ますがおよう", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "30", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "25, 1", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: Father\n2: 50\n3: する\n4: complicated\n5: 2\n6: 3\n7: できない\n8: 火曜日\n9: e\n10: Mars\n11: できない\n12: ますがおよう\n13: 父\n14: 変える\n15: 30\n16: 2\n17: 25, 1\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "microsoft/wizardlm-2-8x22b": {"score": 10, "answered": 20, "first10": 5, "last10": 5, "inconsistent_pairs": 1, "sec": 8.6, "out_tokens": 132, "price": "$0.62 / $0.62", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまざごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "5", "ok": 0}, "q17": {"a": "18文字, 1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n\n2: 10\n\n3: する\n\n4: synchronization\n\n5: 3\n\n6: 3\n\n7: できる\n\n8: 水曜日\n\n9: s\n\n10: 火星\n\n11: できる\n\n12: すまざごうよはお\n\n13: 父親\n\n14: 変える\n\n15: 5\n\n16: 5\n\n17: 18文字, 1つ\n\n18: 3\n\n19: 0.25, 0.3, 0.33, 1/3\n\n20: 存在しない"}, "openai/gpt-3.5-turbo": {"score": 8, "answered": 20, "first10": 4, "last10": 4, "inconsistent_pairs": 1, "sec": 2.7, "out_tokens": 156, "price": "$0.5 / $1.5", "graded": {"q01": {"a": "外科医の父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3人", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すますざようはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5年後", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "13文字、1つ", "ok": 0}, "q18": {"a": "4つ", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 外科医の父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3人  \n7: できる  \n8: 木曜日  \n9: d  \n10: マーキュリー  \n11: できない  \n12: すますざようはお  \n13: 父親  \n14: 変える  \n15: 5年後  \n16: 4  \n17: 13文字、1つ  \n18: 4つ  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "undi95/remm-slerp-l2-13b": {"score": 6, "answered": 20, "first10": 2, "last10": 4, "inconsistent_pairs": 0, "sec": 12.4, "out_tokens": 301, "price": "$0.35 / $0.65", "graded": {"q01": {"a": "外科医は私。", "ok": 0}, "q02": {"a": "ボールは50円。", "ok": 0}, "q03": {"a": "存在する。", "ok": 0}, "q04": {"a": "単語は「synchronization」で、10文字。", "ok": 0}, "q05": {"a": "ちょうちょは3モーラ。", "ok": 1}, "q06": {"a": "サリーの姉妹は6人。", "ok": 0}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "月曜は110日後。", "ok": 0}, "q09": {"a": "kaleidoscope の7文字目は O。", "ok": 0}, "q10": {"a": "カロン。", "ok": 0}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "\"ogJmamus\"。", "ok": 0}, "q13": {"a": "息子は執刀医の息子である。", "ok": 0}, "q14": {"a": "変える。", "ok": 0}, "q15": {"a": "子が10歳のとき。", "ok": 0}, "q16": {"a": "はれですは2個。", "ok": 0}, "q17": {"a": "\"synchronization\" で、1文字。", "ok": 0}, "q18": {"a": "3に分かれる。", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3 を小さい順に。", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}, "raw": "1. 外科医は私。\n2. ボールは50円。\n3. 存在する。\n4. 単語は「synchronization」で、10文字。\n5. ちょうちょは3モーラ。\n6. サリーの姉妹は6人。\n7. できない。\n8. 月曜は110日後。\n9. kaleidoscope の7文字目は O。\n10. カロン。\n11. できない。\n12. \"ogJmamus\"。\n13. 息子は執刀医の息子である。\n14. 変える。\n15. 子が10歳のとき。\n16. はれですは2個。\n17. \"synchronization\" で、1文字。\n18. 3に分かれる。\n19. 0.25, 0.3, 0.33, 1/3 を小さい順に。\n20. 存在しない。"}, "openai/gpt-4": {"score": 9, "answered": 20, "first10": 3, "last10": 6, "inconsistent_pairs": 1, "sec": 4.8, "out_tokens": 154, "price": "$30 / $60", "graded": {"q01": {"a": "少年の父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "22文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 少年の父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 木曜日\n9: s\n10: マーキュリー\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 22文字, 1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "mancer/weaver": {"score": 7, "answered": 20, "first10": 3, "last10": 4, "inconsistent_pairs": 2, "sec": 3.5, "out_tokens": 193, "price": "$0.4 / $0.75", "graded": {"q01": {"a": "1", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "はい", "ok": 0}, "q04": {"a": "複雑", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "月曜日は変わらない", "ok": 0}, "q09": {"a": "k", "ok": 0}, "q10": {"a": "マーキュリー", "ok": 0}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "おはようございます", "ok": 0}, "q13": {"a": "息子と執刀医は父親同士", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10年後", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "10", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "はい", "ok": 0}}, "raw": "1. 1\n2. 10\n3. はい\n4. 複雑\n5. 3\n6. 9\n7. できない\n8. 月曜日は変わらない\n9. k\n10. マーキュリー\n11. できない\n12. おはようございます\n13. 息子と執刀医は父親同士\n14. 同じ\n15. 10年後\n16. 2\n17. 10\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. はい"}, "minimax/minimax-m3": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 0, "sec": 17.5, "out_tokens": 3455, "price": "", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で1個", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 変える\n15: 5\n16: 3\n17: 15文字で1個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "teai/auto": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 6.7, "out_tokens": 579, "price": "", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字・1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 16文字・1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "~anthropic/claude-fable-latest": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 25.3, "out_tokens": 1018, "price": "$10 / $50", "graded": {"q01": {"a": "父親（もう一人の親）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily（synchronizationも同じ15字）", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親（親子）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ（extraordinarily, synchronization）", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親（もう一人の親）\n2: 10\n3: しない\n4: extraordinarily（synchronizationも同じ15字）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親（親子）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ（extraordinarily, synchronization）\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~anthropic/claude-opus-latest": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 17.3, "out_tokens": 875, "price": "$5 / $25", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily（synchronizationも同じ15文字で最長）", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily（synchronizationも同じ15文字で最長）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 執刀医は息子の父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"}, "~google/gemini-flash-latest": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 9.9, "out_tokens": 1138, "price": "$0.75 / $3.75", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すいまざごうよはお", "ok": 0}, "q13": {"a": "父と子（執刀医は父親）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すいまざごうよはお\n13: 父と子（執刀医は父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~google/gemini-pro-latest": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 10.8, "out_tokens": 1403, "price": "$2 / $12", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-chat-latest": {"score": 13, "answered": 20, "first10": 6, "last10": 7, "inconsistent_pairs": 1, "sec": 2.7, "out_tokens": 136, "price": "$5 / $30", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、2語\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/orchestrator:verify": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 41.2, "out_tokens": 6502, "price": "$0.78 / $2.4", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまうごはよお", "ok": 0}, "q13": {"a": "母子(執刀医は母親)", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまうごはよお\n13: 母子(執刀医は母親)\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"}, "teai/lux": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 19.2, "out_tokens": 903, "price": "$10 / $50", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親(執刀医は息子の父)", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ(extraordinarily と synchronization)", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(執刀医は息子の父)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~moonshotai/kimi-latest": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 2, "sec": 23.3, "out_tokens": 720, "price": "$2.4 / $12", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 親子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"}, "~openai/gpt-latest": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 19.5, "out_tokens": 948, "price": "$2 / $10", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~openai/gpt-mini-latest": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 0, "sec": 18.7, "out_tokens": 2723, "price": "$0.75 / $4.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "bytedance/ui-tars-1.5-7b": {"score": 6, "answered": 20, "first10": 4, "last10": 2, "inconsistent_pairs": 2, "sec": 3.8, "out_tokens": 158, "price": "$0.1 / $0.2", "graded": {"q01": {"a": "外科医", "ok": 0}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "2", "ok": 0}, "q06": {"a": "9", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "日曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すきょうはおはようございま", "ok": 0}, "q13": {"a": "息子の父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10年後", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "10文字、2つ", "ok": 0}, "q18": {"a": "2", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 外科医\n2. 10円\n3. しない\n4. synchronization\n5. 2\n6. 9\n7. できない\n8. 日曜日\n9. o\n10. 水星\n11. できる\n12. すきょうはおはようございま\n13. 息子の父親\n14. 変える\n15. 10年後\n16. 2\n17. 10文字、2つ\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"}, "gryphe/mythomax-l2-13b": {"score": 6, "answered": 20, "first10": 2, "last10": 4, "inconsistent_pairs": 3, "sec": 5.9, "out_tokens": 321, "price": "$0.06 / $0.06", "graded": {"q01": {"a": "外科医は病院の医師。", "ok": 0}, "q02": {"a": "ボールは10円。", "ok": 1}, "q03": {"a": "存在する。", "ok": 0}, "q04": {"a": "単語は \"The\"。", "ok": 0}, "q05": {"a": "ちょうちょは1モーラ。", "ok": 0}, "q06": {"a": "サリーの姉妹は9人。", "ok": 0}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "月曜は100日後は月曜。", "ok": 0}, "q09": {"a": "kaleidoscope の7文字目は \"ale\"。", "ok": 0}, "q10": {"a": "カロン。", "ok": 0}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "おはようございますぞ。", "ok": 0}, "q13": {"a": "息子と執刀医は父子関係。", "ok": 1}, "q14": {"a": "同じ。", "ok": 1}, "q15": {"a": "10年後。", "ok": 0}, "q16": {"a": "は2個。", "ok": 0}, "q17": {"a": "最も文字数が多い単語は \"extraordinarily\" で、それは1文字のみ。", "ok": 0}, "q18": {"a": "3つに分かれる。", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33 の順。", "ok": 0}, "q20": {"a": "存在する。", "ok": 0}}, "raw": "1. 外科医は病院の医師。\n2. ボールは10円。\n3. 存在する。\n4. 単語は \"The\"。\n5. ちょうちょは1モーラ。\n6. サリーの姉妹は9人。\n7. できない。\n8. 月曜は100日後は月曜。\n9. kaleidoscope の7文字目は \"ale\"。\n10. カロン。\n11. できない。\n12. おはようございますぞ。\n13. 息子と執刀医は父子関係。\n14. 同じ。\n15. 10年後。\n16. は2個。\n17. 最も文字数が多い単語は \"extraordinarily\" で、それは1文字のみ。\n18. 3つに分かれる。\n19. 0.25, 0.3, 1/3, 0.33 の順。\n20. 存在する。"}, "minimax/minimax-01": {"score": 6, "answered": 20, "first10": 3, "last10": 3, "inconsistent_pairs": 0, "sec": 12.0, "out_tokens": 195, "price": "$0.2 / $1.1", "graded": {"q01": {"a": "番号: 外科医は少年の父親", "ok": 1}, "q02": {"a": "番号: 10", "ok": 1}, "q03": {"a": "番号: しない", "ok": 1}, "q04": {"a": "番号: synchronization", "ok": 0}, "q05": {"a": "番号: 4", "ok": 0}, "q06": {"a": "番号: 3", "ok": 0}, "q07": {"a": "番号: できる", "ok": 0}, "q08": {"a": "番号: 金曜日", "ok": 0}, "q09": {"a": "番号: c", "ok": 0}, "q10": {"a": "番号: 金星", "ok": 0}, "q11": {"a": "番号: できる", "ok": 0}, "q12": {"a": "番号: すまいよはうきょうは", "ok": 0}, "q13": {"a": "番号: 執刀医は少年の父親", "ok": 1}, "q14": {"a": "番号: 変える", "ok": 0}, "q15": {"a": "番号: 5", "ok": 1}, "q16": {"a": "番号: 3", "ok": 1}, "q17": {"a": "番号: 14, 1つ", "ok": 0}, "q18": {"a": "番号: 4", "ok": 0}, "q19": {"a": "番号: 0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "番号: 存在する", "ok": 0}}, "raw": "1. 番号: 外科医は少年の父親\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 4\n6. 番号: 3\n7. 番号: できる\n8. 番号: 金曜日\n9. 番号: c\n10. 番号: 金星\n11. 番号: できる\n12. 番号: すまいよはうきょうは\n13. 番号: 執刀医は少年の父親\n14. 番号: 変える\n15. 番号: 5\n16. 番号: 3\n17. 番号: 14, 1つ\n18. 番号: 4\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する"}, "nousresearch/hermes-3-llama-3.1-405b": {"score": 6, "answered": 20, "first10": 3, "last10": 3, "inconsistent_pairs": 1, "sec": 5.9, "out_tokens": 131, "price": "$1 / $1", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "ましてすごはようお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "17文字, 1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜\n9: d\n10: 水星\n11: できる\n12: ましてすごはようお\n13: 父親\n14: 変える\n15: 5\n16: 4\n17: 17文字, 1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"}, "nousresearch/hermes-3-llama-3.1-70b": {"score": 2, "answered": 20, "first10": 1, "last10": 1, "inconsistent_pairs": 0, "sec": 4.7, "out_tokens": 130, "price": "$0.7 / $0.7", "graded": {"q01": {"a": "1: 母親", "ok": 0}, "q02": {"a": "20", "ok": 0}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "火曜日", "ok": 0}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すいおはよ", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "20, 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1. 1: 母親\n2. 20\n3. する\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 火曜日\n9. o\n10. 水星\n11. できる\n12. すいおはよ\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 20, 1\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"}, "perceptron/perceptron-mk1": {"score": 4, "answered": 20, "first10": 2, "last10": 2, "inconsistent_pairs": 0, "sec": 8.0, "out_tokens": 298, "price": "$0.15 / $1.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すいませんごうよはお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "5", "ok": 0}, "q17": {"a": "17文字, 2個", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 1/3, 0.33, 0.3", "ok": 0}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 木曜日\n9: s\n10: 水星\n11: できる\n12: すいませんごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 5\n17: 17文字, 2個\n18: 4\n19: 0.25, 1/3, 0.33, 0.3\n20: 存在しない"}, "qwen/qwen3-max": {"score": 10, "answered": 20, "first10": 5, "last10": 5, "inconsistent_pairs": 0, "sec": 4.2, "out_tokens": 152, "price": "$0.78 / $3.9", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいさごよあはお", "ok": 0}, "q13": {"a": "母子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "13文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 2  \n7: できる  \n8: 水曜日  \n9: s  \n10: 火星  \n11: できない  \n12: すまいさごよあはお  \n13: 母子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 13文字で1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"}, "shitate/license": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 3.0, "out_tokens": 128, "price": "$0.28 / $0.42", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/orchestrator:sure": {"score": 13, "answered": 20, "first10": 7, "last10": 6, "inconsistent_pairs": 0, "sec": 61.9, "out_tokens": 9459, "price": "$0.57 / $1.99", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母と息子", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25,0.3,0.33,1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "⚠️ 要確認: 独立した複数モデルとコードの答えが割れました。この回答はそのまま使わず、必ず裏取りしてください。\n\n1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母と息子\n14: 変える\n15: 5\n16: 3\n17: 15文字で1つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"}, "teai/default": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 0, "sec": 6.6, "out_tokens": 650, "price": "$3 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~z-ai/glm-latest": {"score": 9, "answered": 20, "first10": 6, "last10": 3, "inconsistent_pairs": 0, "sec": 2.9, "out_tokens": 461, "price": "$1.113 / $3.498", "graded": {"q01": {"a": "母親(女性外科医)", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいごううはお", "ok": 0}, "q13": {"a": "母親(または両親)", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "16文字、1つ(extraordinarily)", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親(女性外科医)\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいごううはお\n13: 母親(または両親)\n14: 同じ\n15: 5\n16: 4\n17: 16文字、1つ(extraordinarily)\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "minimax/minimax-m2-her": {"score": 5, "answered": 20, "first10": 3, "last10": 2, "inconsistent_pairs": 2, "sec": 2.9, "out_tokens": 125, "price": "$0.3 / $1.2", "graded": {"q01": {"a": "父", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "する", "ok": 0}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "日曜日", "ok": 0}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すはごよ はなはき おは", "ok": 0}, "q13": {"a": "母", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "16 & 1", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "する", "ok": 0}}, "raw": "1. 父  \n2. 10  \n3. する  \n4. extraordinarily  \n5. 4  \n6. 3  \n7. できない  \n8. 日曜日  \n9. e  \n10. 水星  \n11. できる  \n12. すはごよ はなはき おは  \n13. 母  \n14. 変える  \n15. 5  \n16. 4  \n17. 16 & 1  \n18. 4  \n19. 0.25, 0.3, 0.33, 1/3  \n20. する"}, "perplexity/sonar": {"score": 9, "answered": 20, "first10": 4, "last10": 5, "inconsistent_pairs": 1, "sec": 6.4, "out_tokens": 136, "price": "$1 / $1", "graded": {"q01": {"a": "外科医の母親（女性の外科医）", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "金曜", "ok": 0}, "q09": {"a": "c", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すうまうごはちお", "ok": 0}, "q13": {"a": "父", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25、0.3、0.33、1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医の母親（女性の外科医）\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 金曜\n9: c\n10: 火星\n11: できる\n12: すうまうごはちお\n13: 父\n14: 同じ\n15: 10\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25、0.3、0.33、1/3\n20: しない"}, "perplexity/sonar-pro": {"score": 9, "answered": 20, "first10": 5, "last10": 4, "inconsistent_pairs": 2, "sec": 2.7, "out_tokens": 141, "price": "$3 / $15", "graded": {"q01": {"a": "外科医の父親（母親ではなく父親）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "うすごときうおは", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15, 2", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 外科医の父親（母親ではなく父親）\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜\n9: d\n10: 水星\n11: できる\n12: うすごときうおは\n13: 父親\n14: 変える\n15: 10\n16: 2\n17: 15, 2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/freelance": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 2.3, "out_tokens": 128, "price": "$0.28 / $0.42", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/infra": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 1.9, "out_tokens": 128, "price": "$0.28 / $0.42", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/legal": {"score": 13, "answered": 20, "first10": 7, "last10": 6, "inconsistent_pairs": 0, "sec": 4.1, "out_tokens": 128, "price": "$0.28 / $0.42", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"}, "shitate/security": {"score": 14, "answered": 20, "first10": 7, "last10": 7, "inconsistent_pairs": 0, "sec": 2.1, "out_tokens": 128, "price": "$0.28 / $0.42", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~deepseek/deepseek-v4-flash-latest": {"score": 8, "answered": 20, "first10": 5, "last10": 3, "inconsistent_pairs": 1, "sec": 6.9, "out_tokens": 149, "price": "$0.05 / $0.16", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "e", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざうよこはうお", "ok": 0}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1個", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: e\n10: 火星\n11: できる\n12: すまいざうよこはうお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1個\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "~z-ai/glm-flash-latest": {"score": 11, "answered": 20, "first10": 7, "last10": 4, "inconsistent_pairs": 1, "sec": 15.8, "out_tokens": 311, "price": "$0.075 / $0.25", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すでいすがよはうお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 4\n6. 2\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できる\n12. すでいすがよはうお\n13. 父親\n14. 変える\n15. 5\n16. 4\n17. 16文字、1つ\n18. 4\n19. 0.25 < 0.3 < 0.33 < 1/3\n20. しない"}, "deepseek/deepseek-v4-flash-vision-exp": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 56.9, "out_tokens": 5855, "price": "$0.22 / $0.66", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "teai/fast": {"score": 18, "answered": 20, "first10": 8, "last10": 10, "inconsistent_pairs": 2, "sec": 47.9, "out_tokens": 3575, "price": "$0.22 / $0.66", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字・2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と子\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "minimax/minimax-m2.1": {"score": 12, "answered": 20, "first10": 7, "last10": 5, "inconsistent_pairs": 0, "sec": 25.3, "out_tokens": 1250, "price": "$0.3 / $1.2", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざいうようは お", "ok": 0}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざいうようは お\n13: 父子\n14: 変える\n15: 5\n16: 3\n17: 14\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~anthropic/claude-haiku-latest": {"score": 11, "answered": 20, "first10": 6, "last10": 5, "inconsistent_pairs": 0, "sec": 20.7, "out_tokens": 2455, "price": "$1 / $5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "5", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "水星", "ok": 0}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうよあはお", "ok": 0}, "q13": {"a": "母と子", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "16文字、1つ", "ok": 0}, "q18": {"a": "5", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すまいざごうよあはお\n13: 母と子\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 5\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "~anthropic/claude-sonnet-latest": {"score": 15, "answered": 20, "first10": 7, "last10": 8, "inconsistent_pairs": 2, "sec": 24.0, "out_tokens": 1779, "price": "$2 / $10", "graded": {"q01": {"a": "母親(外科医は女性で、少年の母)", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ(extraordinarily, synchronization)", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 母親(外科医は女性で、少年の母)\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily, synchronization)\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: 存在する"}, "nousresearch/hermes-4-70b": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 0, "sec": 4.4, "out_tokens": 485, "price": "$0.13 / $0.4", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.1-codex-max": {"score": 17, "answered": 20, "first10": 8, "last10": 9, "inconsistent_pairs": 0, "sec": 25.8, "out_tokens": 1457, "price": "$1.25 / $10", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字、2つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "anthropic/claude-fable-5": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 20.5, "out_tokens": 870, "price": "$10 / $50", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily(synchronizationも同じ15文字)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親(執刀医は父)", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(執刀医は父)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "anthropic/claude-fable-5.1": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 23.5, "out_tokens": 869, "price": "$10 / $50", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily(synchronizationも同じ15文字)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "anthropic/claude-opus-5": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 19.3, "out_tokens": 881, "price": "$5 / $25", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "同じ15文字で2語（extraordinarily / synchronization）", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: 同じ15文字で2語（extraordinarily / synchronization）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "anthropic/claude-opus-5-fast": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 12.1, "out_tokens": 967, "price": "$10 / $50", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily(15文字、synchronizationも同数)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父と息子(執刀医は父親)", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(15文字、synchronizationも同数)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子(執刀医は父親)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "google/gemini-2.5-pro-preview-05-06": {"score": 12, "answered": 20, "first10": 8, "last10": 4, "inconsistent_pairs": 0, "sec": 34.7, "out_tokens": 141, "price": "$1.25 / $10", "graded": {"q01": {"a": "外科医は父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すまいざごうはお", "ok": 0}, "q13": {"a": "執刀医は父親。", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "14文字で1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"}, "google/gemini-3.1-pro-preview-customtools": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 16.2, "out_tokens": 1417, "price": "$2 / $12", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-3.6-flash": {"score": 13, "answered": 20, "first10": 6, "last10": 7, "inconsistent_pairs": 1, "sec": 3.6, "out_tokens": 142, "price": "$0.75 / $3.75", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "s", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうようはお", "ok": 0}, "q13": {"a": "親子（父親）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうようはお\n13: 親子（父親）\n14: 同じ\n15: 5\n16: 4\n17: 15文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "google/gemini-3.7-flash": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 6.1, "out_tokens": 513, "price": "$0.375 / $1.875", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "親子（父親）", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13. 親子（父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "minimax/minimax-m2.7": {"score": 7, "answered": 20, "first10": 4, "last10": 3, "inconsistent_pairs": 1, "sec": 17.1, "out_tokens": 836, "price": "$0.3 / $1.2", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "できる", "ok": 0}, "q08": {"a": "木曜日", "ok": 0}, "q09": {"a": "d", "ok": 0}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "まらせょうはうお", "ok": 0}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "2", "ok": 0}, "q17": {"a": "15文字が2語", "ok": 1}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 1/3, 0.33", "ok": 0}, "q20": {"a": "存在する", "ok": 0}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 木曜日\n9: d\n10: 火星\n11: できる\n12: まらせょうはうお\n13: 父子\n14: 変える\n15: 5\n16: 2\n17: 15文字が2語\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"}, "openai/gpt-5.5": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 19.5, "out_tokens": 1174, "price": "$5 / $30", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.6-sol-pro": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 29.4, "out_tokens": 2785, "price": "$2.5 / $15", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.6-terra": {"score": 16, "answered": 20, "first10": 8, "last10": 8, "inconsistent_pairs": 1, "sec": 11.8, "out_tokens": 593, "price": "$1.25 / $7.5", "graded": {"q01": {"a": "母親", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "母親", "ok": 0}, "q14": {"a": "変える", "ok": 0}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-5.6-terra-pro": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 0, "sec": 18.4, "out_tokens": 2264, "price": "$1.25 / $7.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openai/gpt-6-astra-pro": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 12.3, "out_tokens": 632, "price": "$10 / $50", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2語", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "openrouter/auto": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 18.1, "out_tokens": 837, "price": "$1 / $3", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily(synchronizationも同じ15文字)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"}, "shitate/orchestrator": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 50.9, "out_tokens": 7444, "price": "$0.27 / $1.33", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/orchestrator-max": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 32.1, "out_tokens": 727, "price": "$5.95 / $30.1", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいいざごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/orchestrator:max": {"score": 20, "answered": 20, "first10": 10, "last10": 10, "inconsistent_pairs": 0, "sec": 39.4, "out_tokens": 5365, "price": "$10.5 / $52.5", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily(synchronizationも同じ15文字)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親と息子", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ(extraordinarily と synchronization)", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "shitate/orchestrator:solid": {"score": 19, "answered": 20, "first10": 9, "last10": 10, "inconsistent_pairs": 1, "sec": 34.6, "out_tokens": 3339, "price": "$6.04 / $28.89", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で2つ(extraordinarily, synchronization)", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ(extraordinarily, synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "teai/max": {"score": 18, "answered": 20, "first10": 9, "last10": 9, "inconsistent_pairs": 1, "sec": 32.3, "out_tokens": 701, "price": "$5.95 / $30.1", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいいざごうよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字、2つ", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "z-ai/glm-5.3-flash": {"score": 13, "answered": 20, "first10": 8, "last10": 5, "inconsistent_pairs": 0, "sec": 5.4, "out_tokens": 331, "price": "$0.07 / $0.25", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "synchronization", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できる", "ok": 0}, "q12": {"a": "すでいざうがよはお", "ok": 0}, "q13": {"a": "父親", "ok": 1}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "4", "ok": 0}, "q17": {"a": "15文字、1つ", "ok": 0}, "q18": {"a": "4", "ok": 0}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すでいざうがよはお\n13: 父親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"}, "qwen/qwen3.5-flash-02-23": {"score": 9, "answered": 12, "first10": 8, "last10": 1, "inconsistent_pairs": 0, "sec": 28.4, "out_tokens": 6005, "price": "$0.065 / $0.26", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいさご", "ok": 0}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいさご"}, "deepseek/deepseek-v4-flash-cache": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 30.4, "out_tokens": 6000, "price": "$0.007 / $0.66", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "z-ai/glm-5.2": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 30.3, "out_tokens": 6000, "price": "", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "qwen/qwen3.6-flash": {"score": 10, "answered": 12, "first10": 9, "last10": 1, "inconsistent_pairs": 0, "sec": 22.7, "out_tokens": 4157, "price": "$0.1875 / $1.125", "graded": {"q01": {"a": "父親", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない", "ok": 1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごう", "ok": 0}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごう"}, "inclusionai/ling-3.0-flash-fin": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 19.9, "out_tokens": 1616, "price": "$0.06 / $0.18", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "stepfun/step-3.7-flash": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 29.9, "out_tokens": 6000, "price": "$0.2 / $1.15", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "deepseek/deepseek-v3.1-terminus": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 7.6, "out_tokens": 129, "price": "$0.27 / $1", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "qwen/qwen3-coder-30b-a3b-instruct": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 3.0, "out_tokens": 152, "price": "$0.07 / $0.28", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "deepseek/deepseek-chat-v3.1": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 7.9, "out_tokens": 129, "price": "$0.25 / $0.95", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}, "google/gemini-2.5-pro-preview": {"score": 0, "answered": 0, "first10": 0, "last10": 0, "inconsistent_pairs": 0, "sec": 41.7, "out_tokens": 2048, "price": "$1.25 / $10", "graded": {"q01": {"a": null, "ok": -1}, "q02": {"a": null, "ok": -1}, "q03": {"a": null, "ok": -1}, "q04": {"a": null, "ok": -1}, "q05": {"a": null, "ok": -1}, "q06": {"a": null, "ok": -1}, "q07": {"a": null, "ok": -1}, "q08": {"a": null, "ok": -1}, "q09": {"a": null, "ok": -1}, "q10": {"a": null, "ok": -1}, "q11": {"a": null, "ok": -1}, "q12": {"a": null, "ok": -1}, "q13": {"a": null, "ok": -1}, "q14": {"a": null, "ok": -1}, "q15": {"a": null, "ok": -1}, "q16": {"a": null, "ok": -1}, "q17": {"a": null, "ok": -1}, "q18": {"a": null, "ok": -1}, "q19": {"a": null, "ok": -1}, "q20": {"a": null, "ok": -1}}, "raw": ""}}, "timeout_502": ["aion-labs/aion-2.0", "aion-labs/aion-3.0", "anthropic/claude-opus-4.5", "anthropic/claude-opus-4.6", "anthropic/claude-sonnet-4.6", "bytedance-seed/seed-2-1-turbo", "bytedance-seed/seed-2.0-mini", "cerebras/gemma-4-31b", "cerebras/gpt-oss-120b", "cerebras/qwen-3-32b", "claude-haiku-4-5-20251001", "claude-opus-4-5", "claude-opus-4-6", "claude-sonnet-4-6", "claude-sonnet-5", "deepseek/deepseek-r1", "deepseek/deepseek-r1-0528", "deepseek/deepseek-r1-distill-llama-70b", "deepseek/deepseek-v3.2", "deepseek/deepseek-v3.2-exp", "gemini-2.0-flash", "gemma2-9b-it", "google/gemma-4-26b-a4b-it", "inclusionai/ling-2.6-flash", "inclusionai/ring-2.6-1t", "kimi-k2-0711", "kwaipilot/kat-coder-pro-v2", "kwaipilot/kat-coder-pro-v2.5", "llama-3.3-70b-specdec", "llama-4-maverick-17b-128e", "llama-4-scout-17b-16e", "meituan/longcat-2.0", "meta/muse-spark-1.1", "meta/muse-spark-1.2", "meta/muse-spark-1.2-contributor", "meta/muse-spark-1.3", "meta/muse-spark-1.3-contributor", "minimax/minimax-m1", "minimax/minimax-m2", "minimax/minimax-m2.5", "mistral-saba-24b", "mistralai/mistral-small-3.1-24b-instruct", "mistralai/voxtral-small-24b-2507", "morph/morph-v3-fast", "morph/morph-v3-large", "nousresearch/hermes-4-405b", "nvidia/Llama-3.3-Nemotron-Super-49B-v1.5", "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B", "nvidia/nemotron-3-super-120b-a12b", "nvidia/nemotron-3.5-lightning", "openai/gpt-4-turbo-preview", "openai/gpt-5-nano", "openai/gpt-5-pro", "openai/gpt-5.1-codex-mini", "openai/gpt-5.2-chat", "openai/gpt-5.2-pro", "openai/gpt-5.4-pro", "openai/gpt-5.5-pro", "openai/o1-pro", "openai/o3-mini-high", "openai/o3-pro", "openai/o4-mini-high", "perplexity/sonar-reasoning-pro", "qwen-qwq-32b", "qwen/qwen-2.5-coder-32b-instruct", "qwen/qwen3-235b-a22b", "qwen/qwen3-235b-a22b-thinking-2507", "qwen/qwen3-32b", "qwen/qwen3-8b", "qwen/qwen3-max-thinking", "qwen/qwen3-vl-235b-a22b-thinking", "qwen/qwen3.5-122b-a10b", "qwen/qwen3.5-27b", "qwen/qwen3.5-35b-a3b", "qwen/qwen3.5-397b-a17b", "qwen/qwen3.5-9b", "qwen/qwen3.5-plus-02-15", "qwen/qwen3.5-plus-20260420", "qwen/qwen3.6-27b", "qwen/qwen3.6-max-preview", "qwen/qwen3.6-plus", "qwen/qwen3.7-flash", "qwen/qwen3.7-max", "qwen/qwen3.7-plus", "qwen/qwen3.8-flash", "qwen/qwen3.8-max", "qwen/qwen3.8-max-0902", "rekaai/reka-flash-3", "sakana/fugu-ultra", "stepfun/step-3.5-flash", "tencent/hunyuan-a13b-instruct", "tencent/hy4-preview", "upstage/solar-pro4", "x-ai/grok-4.20-multi-agent", "x-ai/grok-4.5", "x-ai/grok-4.6", "xiaomi/mimo-v2.5", "xiaomi/mimo-v2.5-pro", "z-ai/glm-4.5", "z-ai/glm-4.5-air", "z-ai/glm-4.5v", "z-ai/glm-4.6", "z-ai/glm-4.6v", "z-ai/glm-4.7", "z-ai/glm-4.7-flash", "z-ai/glm-5-turbo", "z-ai/glm-5v-turbo", "~x-ai/grok-latest"], "single_vs_batch_top23": {"anthropic/claude-fable-5": {"single_score": 20, "single_sec": 155, "single_out_tokens": 2063, "batch_score": 20, "batch_sec": 20.5, "batch_out_tokens": 870, "single_graded": {"q01": {"a": "外科医は少年の**父親**です。\n\n（このなぞなぞの元の版では「父親が死亡し、外科医は母親」という展開ですが、この設定では母親が亡くなっているので、外科医は父親となります。）", "ok": 1}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない。2026年はうるう年ではないため、2月は28日までです。", "ok": 1}, "q04": {"a": "「extraordinarily」（15文字）です。ちなみに「synchronization」も同じ15文字で並んでいます。", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない。**\n\n通常この問題は7回の川渡り(最低でも複数往復)が必要ですが、ボートが1往復しか持たないなら、運べるのは1つだけ。残りをすべて運ぶ手段がなくなります。", "ok": 1}, "q08": {"a": "水曜日です。（100 ÷ 7 = 14 余り 2 なので、月曜の2日後）", "ok": 1}, "q09": {"a": "**o** です。（k-a-l-e-i-d-**o**）", "ok": 1}, "q10": {"a": "**火星**です。\n\n（最小は水星、2番目に小さいのが火星です。）", "ok": 1}, "q11": {"a": "**できない。**\n\n2Lと4Lの組み合わせでは2の倍数(2L、4L、6L…)しか測れないため、3Lは不可能です。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。\n\n（このなぞなぞは通常「執刀医は母親」という答えで性別の思い込みを問うものですが、この場合は母親が亡くなっているため、父親が答えになります。）", "ok": 1}, "q14": {"a": "**同じ**です。\n\n司会者が正解を知らずにランダムに開けてたまたまハズレだった場合、AもCも当たる確率は **1/2ずつ** になります(通常のモンティ・ホール問題と違い、変えても有利になりません)。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最長は **15文字** で、該当する単語は **2つ**(extraordinarily, synchronization)です。", "ok": 1}, "q18": {"a": "**3**\n\n（ちょ・う・ちょ）", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）", "ok": 1}, "q20": {"a": "**存在しない。**\n\n古典的な川渡り問題では最低7回の渡河(3.5往復)が必要です。1往復(2回の渡河)では、ボートは元の岸に戻ってしまう上、一度に運べるのは1つだけなので、3つ全てを対岸に無事に運ぶことは不可能です。", "ok": 1}}}, "anthropic/claude-fable-5.1": {"single_score": 18, "single_sec": 169, "single_out_tokens": 1783, "batch_score": 20, "batch_sec": 23.5, "batch_out_tokens": 869, "single_graded": {"q01": {"a": "外科医は少年の**父親**です。（母親は亡くなっているので、「息子だ」と言えるのは父親です。）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。2026年は閏年ではないため、2月は28日までです。", "ok": 1}, "q04": {"a": "「extraordinarily」（15文字）です。ただし「synchronization」も同じ15文字で同率最長です。", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない**\n\n全部運ぶには最低7回の渡河(3往復半)が必要ですが、ボートは1往復しか使えないため、途中でボートが壊れて残りを運べません。", "ok": 1}, "q08": {"a": "100 ÷ 7 = 14 余り 2 なので、月曜から2日進んで **水曜日** です。", "ok": 1}, "q09": {"a": "k-a-l-e-i-d-o-s-c-o-p-e → 7文字目は「**o**」です。", "ok": 0}, "q10": {"a": "火星（マーズ）です。最小は水星で、その次に小さいのが火星（直径約6,779km）です。", "ok": 1}, "q11": {"a": "**できない**\n\n2Lと4Lはどちらも2の倍数なので、これらの組み合わせでは偶数の量（0, 2, 4L）しか作れず、奇数の3Lは測れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。（母親が亡くなった後も、もう一人の親である父が医師として手術に立ち会った、という筋書きです。）", "ok": 1}, "q14": {"a": "**どちらでも同じ**（1/2ずつ）。\n\n司会者が正解を知らずランダムに開けてハズレだった場合、AもCも当たりの確率は1/2になり、変える利点はありません。（司会者が正解を知っていて意図的にハズレを開ける通常のモンティ・ホール問題なら、変えると2/3で有利です。）", "ok": 1}, "q15": {"a": "40+x = 3(10+x) → 40+x = 30+3x → x = 5\n\n**5**", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最長は15文字で、該当する単語は2つ（extraordinarily、synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3（≈0.3333…）", "ok": 1}, "q20": {"a": "**存在しない。**\n\nボートには農夫のほかに1つしか載せられないため、1往復では1つしか運べません。3つ全てを安全に運ぶには最低7回の渡河（3往復半）が必要です。", "ok": 1}}}, "anthropic/claude-opus-5": {"single_score": 20, "single_sec": 148, "single_out_tokens": 2837, "batch_score": 20, "batch_sec": 19.3, "batch_out_tokens": 881, "single_graded": {"q01": {"a": "少年の**父親**です。（母親が亡くなっているので、もう一人の親＝父親。同性カップルなら「もう一人の母親」の可能性も）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。\n\n2026年はうるう年ではないため、2月は28日までです。（直近のうるう年は2024年と2028年）", "ok": 1}, "q04": {"a": "どちらも15文字で同着ですが、1つ挙げるなら **extraordinarily**", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない。**\n\n1往復（渡って戻る）だけでは、運べるのは1つだけ。仮に多めに数えても最大2つで、残り1つは向こう岸", "ok": 1}, "q08": {"a": "水曜日です。（100 ÷ 7 = 14 余り 2 → 月曜＋2日）", "ok": 1}, "q09": {"a": "**o**", "ok": 1}, "q10": {"a": "**火星**です。（最小は水星）", "ok": 1}, "q11": {"a": "**できない。**\n\n2Lと4Lはどちらも偶数(最大公約数2)なので、作れる量は必ず2の倍数になり、奇数の3Lは測れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。\n\n（元の有名ななぞなぞは「父親が死亡し、執刀医は母親」という形で性別の思い込みを突くもの。この版では母が亡くなっているので、単純に父親が答えになります。もちろん同性カップルの「もう一人の母」という可能性もあります。）", "ok": 1}, "q14": {"a": "**同じ**(どちらでも1/2）。\n\n司会者が正解を知らずランダムに開けた場合、ハズレが出た時点でAとCは各1/2。通常のモンティ・ホール（司会者が知っている）と違い、変える利点はありません。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "**15文字**で、該当する単語は**2つ**（extraordinarily と synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）", "ok": 1}, "q20": {"a": "**存在しない。**\n\n理由：ボートには農夫＋1つしか乗せられないため、全員を運ぶには最低7回の渡河（往復を繰り返す）が必要。1往復では出発地点に戻ってしまい、しかも残した組み合わせでオオカミがヤギを、ヤギがキャベツを食べてしまう。", "ok": 1}}}, "anthropic/claude-opus-5-fast": {"single_score": 20, "single_sec": 99, "single_out_tokens": 2803, "batch_score": 20, "batch_sec": 12.1, "batch_out_tokens": 967, "single_graded": {"q01": {"a": "父親です。\n\nこの問題は「外科医＝母親」という有名ななぞなぞをひねったもので、母親が亡くなっている以上、答えは父親（あるいは同性カップルのもう一人の親）になります。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない（2026年は閏年ではないため、2月は28日まで）。", "ok": 1}, "q04": {"a": "**extraordinarily**（15文字）\n※ synchronization も同じ15文", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない。**\n\n1往復（向こう岸へ1回＋戻り1回）で壊れるなら、荷物を1つ運んで戻った時点で終わり。残り2つと農夫が元の岸に取り残されます。本来この問題は7回の渡河が必要です。", "ok": 1}, "q08": {"a": "**水曜日**です。\n\n100 ÷ 7 = 14 あまり 2 なので、月曜から2日進んで水曜日になります。", "ok": 1}, "q09": {"a": "**o**", "ok": 1}, "q10": {"a": "火星です。（最小は水星）", "ok": 1}, "q11": {"a": "**できない**\n\n2Lと4Lはどちらも2の倍数なので、量れる量は必ず偶数(0・2・4・6L…)になり、奇数の3Lは作れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は**息子の父親**です。\n\n（有名な「父が亡くなり、執刀医が母親だった」という引っかけ問題の逆パターンで、この場合は素直に父親と考えるのが自然です。もちろん、同性カップルのもう一人の母親、という可能性もあります）", "ok": 1}, "q14": {"a": "**同じ**(どちらも1/2)。\n\n司会者が正解を知らずランダムに開けた場合、たまたまハズレが出た時点でAとCは対等になり、通常のモンティ・ホール問題(2/3で変える方が得)とは異なります。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "**15文字**で、該当は**2語**（extraordinarily / synchronization）。", "ok": 1}, "q18": {"a": "**3**（ちょ／う／ちょ）", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3\n\n(1/3 = 0.333… なので 0.33 より大きい)", "ok": 1}, "q20": {"a": "**存在しない。**\n\n1往復（向こう岸へ1回＋戻り1回）では、運べるのは最大1つだけ。残されたオオカミ・ヤギ・キャベツのどれかの組み合わせで必ず食べられてしまう（オオカミとヤギ、またはヤギとキャベツが残る）ため、全員無事に渡すことは不可能です。最低でも「3往復＋片道1回（計7回の渡河）」が必要です。", "ok": 1}}}, "google/gemini-2.5-pro-preview-05-06": {"single_score": 15, "single_sec": 180, "single_out_tokens": 3101, "batch_score": 12, "batch_sec": 34.7, "batch_out_tokens": 141, "single_graded": {"q01": {"a": "外科医は少年の母親です。", "ok": 0}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の母親です。", "ok": 0}, "q14": {"a": "同じ", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、1つです (extraordinarily)。", "ok": 0}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "しない", "ok": 1}}}, "google/gemini-3.1-pro-preview-customtools": {"single_score": 19, "single_sec": 115, "single_out_tokens": 3113, "batch_score": 19, "batch_sec": 16.2, "batch_out_tokens": 1417, "single_graded": {"q01": {"a": "少年の父親です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "太陽系で2番目に小さい惑星は**火星**です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の「父親」です。", "ok": 1}, "q14": {"a": "同じ（確率はどちらも1/2です）", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多いのは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない。\n\n（通常のルールではボートに1つしか乗せられないため、全てを運ぶには複数回の往復が必要です。）", "ok": 1}}}, "google/gemini-3.6-flash": {"single_score": 19, "single_sec": 67, "single_out_tokens": 2184, "batch_score": 13, "batch_sec": 3.6, "batch_out_tokens": 142, "single_graded": {"q01": {"a": "外科医は**少年の父親**です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない**", "ok": 1}, "q08": {"a": "**水曜日**です。（100 ÷ 7 = 14 余り 2 なので、月曜日から2日後になります）", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。（一番小さいのは水星）", "ok": 1}, "q11": {"a": "**できない**", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**（またはもう一人の母親）です。", "ok": 1}, "q14": {"a": "**「同じ」（どちらでも確率は1/2で同じ）**", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多い単語は **15文字** で、該当する単語は **2つ**（\"extraordinarily\" と \"synchronization\"）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "小さい順（昇順）に並べると以下の通りです。\n\n**0.25, 0.3, 0.33, 1/3**", "ok": 1}, "q20": {"a": "存在しない", "ok": 1}}}, "google/gemini-3.7-flash": {"single_score": 20, "single_sec": 82, "single_out_tokens": 2870, "batch_score": 19, "batch_sec": 6.1, "batch_out_tokens": 513, "single_graded": {"q01": {"a": "**少年の父親**です（または、同性", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "存在しません（しない）。", "ok": 1}, "q04": {"a": "**extraordinarily**（または **synchronization**）\n\n※どちらも同じ15文字で最長です。", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない**\n\n（全員を無事に運ぶには最低でも3往復半＝片道7回が必要なため、1往復では不可能です。）", "ok": 1}, "q08": {"a": "**水曜日**です。\n\n（100 ÷ 7 ＝ 14 余り 2 なので、月曜の2日後になります）", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "**火星**です。", "ok": 1}, "q11": {"a": "**できない**。\n\n（どちらも偶数リットルのため、作れる水の量は常に偶数になり、奇数の3Lは測れません）", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。", "ok": 1}, "q14": {"a": "**同じ**\n\n得られる結果（当たる確率）の期待値は", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多いものは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "**0.25, 0.3, 0.33, 1/3**\n\n（※ 1/3 ≒ 0.333... のため）", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "minimax/minimax-m2.7": {"single_score": 11, "single_sec": 125, "single_out_tokens": 3791, "batch_score": 7, "batch_sec": 17.1, "batch_out_tokens": 836, "single_graded": {"q01": {"a": "外科医はその少年的**父親**です。\n\n（「私の息子だ」と言えるのは、その外科医は男性＝父親だからです。）", "ok": 1}, "q02": {"a": "**10**", "ok": 1}, "q03": {"a": "**しない**\n\n2026は4で割り切れないため、うるう年ではありません。", "ok": 1}, "q04": {"a": "", "ok": -1}, "q05": {"a": "6", "ok": 0}, "q06": {"a": "**3**", "ok": 0}, "q07": {"a": "", "ok": -1}, "q08": {"a": "水曜日", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "", "ok": -1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。", "ok": 1}, "q14": {"a": "", "ok": -1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、1つ（「extraordinarily」）。", "ok": 0}, "q18": {"a": "", "ok": -1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "", "ok": -1}}}, "openai/gpt-5.4-pro": {"single_score": 17, "single_sec": 234, "single_out_tokens": 3015, "batch_score": null, "batch_sec": null, "batch_out_tokens": null, "single_graded": {"q01": {"a": "父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "", "ok": -1}, "q08": {"a": "水曜日。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。2Lと4Lだけでは偶数Lしか測れないから。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "父親です。", "ok": 1}, "q14": {"a": "", "ok": -1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最長は15文字で、該当する単語は2つです（**extraordinarily**, **synchronization**）。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-5.5": {"single_score": 18, "single_sec": 93, "single_out_tokens": 2142, "batch_score": 19, "batch_sec": 19.5, "batch_out_tokens": 1174, "single_graded": {"q01": {"a": "父親です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親。", "ok": 1}, "q14": {"a": "", "ok": -1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、2語（extraordinarily / synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-5.5-pro": {"single_score": 17, "single_sec": 178, "single_out_tokens": 3382, "batch_score": null, "batch_sec": null, "batch_out_tokens": null, "single_graded": {"q01": {"a": "父親。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "", "ok": -1}, "q14": {"a": "同じ。  \n司会者が正解を知らずラン", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "", "ok": -1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-5.6-sol-pro": {"single_score": 19, "single_sec": 118, "single_out_tokens": 3463, "batch_score": 19, "batch_sec": 29.4, "batch_out_tokens": 2785, "single_graded": {"q01": {"a": "少年の父親。", "ok": 1}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない。", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。ボートが1往復で壊れるため、全てを運べない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親。", "ok": 1}, "q14": {"a": "同じ。AもCも当たり確率は1/2です。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最長は15文字で、該当する単語は2つです。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-5.6-terra": {"single_score": 18, "single_sec": 66, "single_out_tokens": 840, "batch_score": 16, "batch_sec": 11.8, "batch_out_tokens": 593, "single_graded": {"q01": {"a": "外科医は少年の母親です。", "ok": 0}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily（synchronization と同じ長さ）", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。1往復で壊れると、全員・全品を対岸へ運ぶ前にボートが使えなくなる。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できる。", "ok": 0}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親です。", "ok": 1}, "q14": {"a": "同じ（確率はAもCも1/2）。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、該当する単語は2つです（“extraordinarily”, “synchronization”）。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-5.6-terra-pro": {"single_score": 18, "single_sec": 104, "single_out_tokens": 3606, "batch_score": 18, "batch_sec": 18.4, "batch_out_tokens": 2264, "single_graded": {"q01": {"a": "少年のもう一人の親、つまり父親です。", "ok": 1}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親です。", "ok": 1}, "q14": {"a": "同じ。司会者は正解を知らずランダムなので、Bがハズレだった条件ではAとCは各50%です。", "ok": 1}, "q15": {"a": "10", "ok": 0}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、該当する単語は2つです（“extraordinarily”, “synchronization”）。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openai/gpt-6-astra-pro": {"single_score": 19, "single_sec": 124, "single_out_tokens": 1881, "batch_score": 19, "batch_sec": 12.3, "batch_out_tokens": 632, "single_graded": {"q01": {"a": "少年の父親です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親です。", "ok": 1}, "q14": {"a": "同じ。AもCも当たる確率は1/2。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、該当する単語は2つです。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "openrouter/auto": {"single_score": 19, "single_sec": 151, "single_out_tokens": 2715, "batch_score": 20, "batch_sec": 18.1, "batch_out_tokens": 837, "single_graded": {"q01": {"a": "父親です。（この設定では母親が亡くなっているので、「女性医師＝母」という有名ななぞなぞの答えは当てはまりません。同性カップルならもう一人の母親、という可能性もあります）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。\n\n2026年は閏年ではない(4で割り切れない)ため、2月は28日までです。", "ok": 1}, "q04": {"a": "", "ok": -1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない。**\n\n1往復(往路+復路)で壊れるので、運べるのは実質1つだけ。残った2つのどちらかの組み合わせ(オオカミ+ヤギ、またはヤギ+キャベツ)が必ず岸に残り、食べられてしまう。全部運ぶには最低7回の渡り(3往復+片道1回)が必要。", "ok": 1}, "q08": {"a": "**水曜日**です。\n\n100 ÷ 7 = 14 あまり 2 → 月曜から2日進んで水曜日。", "ok": 1}, "q09": {"a": "**o**", "ok": 1}, "q10": {"a": "**火星（マーズ）**です。\n\n最小は水星（直径約4,879 km）、2番目に小さいのが火星（直径約6,779 km）です。", "ok": 1}, "q11": {"a": "**できない。**\n\n2Lと4Lはどちらも偶数（最大公約数が2）なので、注ぐ・移す操作でつくれる量は必ず2の倍数（0L・2L・4L・6L）になり、奇数の3Lは作れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親です。", "ok": 1}, "q14": {"a": "**同じ（変えても変えなくても1/2）**\n\n司会者が正解を知らずランダムに開けた場合、「ハズレが出た」という情報でAが当たりの確率も1/3→1/2に上がります。C も 1/2 なので有利不利はありません。\n\n（通常のモンティ・ホール問題は司会者が正解を知っていて必ずハズレを開けるため、変えると2/3になります。）", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "**15文字**で、該当は**2語**(extraordinarily と synchronization)。", "ok": 1}, "q18": {"a": "**3**\n\n（ちょ・う・ちょ）", "ok": 1}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）", "ok": 1}, "q20": {"a": "**存在しない。**\n\nボートには自分＋1つしか乗せられないため、", "ok": 1}}}, "shitate/orchestrator": {"single_score": 19, "single_sec": 237, "single_out_tokens": 7542, "batch_score": 19, "batch_sec": 50.9, "batch_out_tokens": 7444, "single_graded": {"q01": {"a": "父親（少年の父親）です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。", "ok": 1}, "q08": {"a": "水曜日です。", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の「父親」です。", "ok": 1}, "q14": {"a": "同じ（どちらでも確率は1/", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多いのは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3", "ok": 1}, "q20": {"a": "存在しない。", "ok": 1}}}, "shitate/orchestrator-max": {"single_score": 20, "single_sec": 383, "single_out_tokens": 16892, "batch_score": 18, "batch_sec": 32.1, "batch_out_tokens": 727, "single_graded": {"q01": {"a": "父親です。（母親は事故で亡くなっているため、外科医は少年の父親です。）", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない(2026年は平年のため、2月は28日までです)", "ok": 1}, "q04": {"a": "extraordinarily(15文字。synchronization も同じ15文字ですが、先に出現する語として前者)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない\n\nボートが1往復(2回の川渡り)しか使えないと、農夫と一緒に運べる荷物は最大1つだけです。全部運ぶには通常7回の川渡り(片道7回)が必要なため、条件を満たす方法は存在しません。", "ok": 1}, "q08": {"a": "水曜日です。(100 ÷ 7 = 14週あまり2日のため、月曜の2日後)", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。2Lと4Lの組み合わせでは偶数の量(2L・4L・6L)しか測れず、3Lを測ることはできません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の**父親**です。\n\n（母親は事故で亡くなっているため、もう一人の親である父親が執刀医だった、という答えになります。）", "ok": 1}, "q14": {"a": "どちらでも同じです（変える必要はありません）。\n\n司会者が正解を知らずにランダムに開けた場合、Bがハズレだったという事実により条件付き確率が更新され、AとCの当たり確率はともに1/2になります。通常のモンティ・ホール問題（司会者が正解を知っていて必ずハズレを開ける場合）とは異なり、変えても有利にはなりません。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多い単語は**15文字**で、該当する単語は**2つ**です（extraordinarily、synchronization）。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3\n\n(1/3 ≈ 0.333… のため、0.33 より大きくなります)", "ok": 1}, "q20": {"a": "存在しない。\n\nボートには農夫のほか1つしか乗せられないため、1往復(行き1回・帰り1回)では最大でも1つしか対岸に残せず、3つ全てを運ぶことは不可能です。通常このパズルの解決には最低7回の渡航が必要とされています。", "ok": 1}}}, "shitate/orchestrator:max": {"single_score": 19, "single_sec": 252, "single_out_tokens": 13440, "batch_score": 20, "batch_sec": 39.4, "batch_out_tokens": 5365, "single_graded": {"q01": {"a": "外科医は少年の**父親**です。\n\n（元のなぞなぞは「父親が死亡し、外科医は母親」という性別の思い込みを問うものですが、この設問では母親が死亡しているため、外科医はもう一人の親である父親と考えるのが自然です。なお、同性カップルの家庭であれば「もう一人の母親」という可能性もあります。）", "ok": 1}, "q02": {"a": "10円", "ok": 1}, "q03": {"a": "しない。2026年はうるう年ではないため、2月は28日までです。", "ok": 1}, "q04": {"a": "**extraordinarily**（15文字）\n\n※ \"synchronization\" も同じ15文字で同率最長ですが、文中で先に現れる extraordinarily を回答とします。", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "**できない。**\n\nボートは1往復（片道2回）で壊れるため、農夫が運べる荷物は1つだけ。全部を安全に渡すには最低7回の片道移動が必要なので、不可能です。", "ok": 1}, "q08": {"a": "**水曜日**です。（100 ÷ 7 = 14 余り 2 → 月曜の2日後）", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "**火星**です。（最も小さいのは水星、2番目が火星）", "ok": 1}, "q11": {"a": "**できない。**\n\n2Lと4Lの最大公約数は2なので、作れる量はすべて2の倍数(2L、4L、6L…)になります。奇数の3Lは測れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "**執刀医は息子の父親**です。\n\n（元のなぞなぞは「父親が事故死し、執刀医が『私の子だ』と言う」設定で、答えは「母親」——「外科医＝男性」という思い込みを突くものです。この改変版では母親が亡くなっているため、素直に父親が答えになります。なお、同性カップルのもう一人の母親という別解もあり得ます。）", "ok": 1}, "q14": {"a": "**どちらでも同じ**（AもCも各1/2）。\n\n司会者が正解を知らずランダムに開けてたまたまハズレだった場合、条件付き確率はAもCも1/2になり、変えても有利になりません。切り替えが有利（2/3）になるのは、司会者が正解を知っていて必ずハズレを開ける通常のモンティ・ホール問題の場合だけです。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、該当する単語は2つです（\"extraordinarily\" と \"synchronization\"）。", "ok": 1}, "q18": {"a": "3\n\n「ちょ・う・ちょ」の3拍です。拗音「ちょ」は1拍、長音「う」も1拍と数えます。", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3（1/3 ≈ 0.333… なので 0.33 より大きい）", "ok": 1}, "q20": {"a": "**存在しない。**\n\nボートには農夫のほかに1つしか乗せられないため、1往復（片道2回の移動）では最大1つしか対岸に運べません。オオカミとヤギ、ヤギとキャベツを一緒に残せない制約を守って全てを運ぶには、最低でも片道7回（3往復半）の渡河が必要です。", "ok": 1}}}, "shitate/orchestrator:solid": {"single_score": 18, "single_sec": 276, "single_out_tokens": 11782, "batch_score": 19, "batch_sec": 34.6, "batch_out_tokens": 3339, "single_graded": {"q01": {"a": "外科医は少年の父親です。\n\n元のなぞなぞは「父親が死亡し、外科医が母親」という、\"外科医＝男性\"という思い込みを突く問題ですが、この設問では死亡したのが母親なので、外科医は父親と考えるのが自然です。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない\n\n2026年は閏年ではないため、2月は28日までです（次の閏年は2028年）。", "ok": 1}, "q04": {"a": "**extraordinarily**(15文字)\n\n※ synchronization も同じく15文字で同率最長です。", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2\n\n（兄弟から見た3人の姉妹にはサリー自身が含まれるため、サリーの姉妹は2人です）", "ok": 0}, "q07": {"a": "できない。\n\nこの問題の標準解は片道7回(3往復半)の渡河が必要です。ボートが1往復(行き帰り2回)で壊れるなら、最初にヤギを運んで戻った時点でボートが使えなくなり、オオカミとキャベツを運ぶ手段がなくなります。どの組み合わせで始めても、1往復では最大1つしか対岸に運べず、全部を運ぶことは不可能です。", "ok": 1}, "q08": {"a": "水曜日です。（100 ÷ 7 = 14 あまり 2 → 月曜の2日後）", "ok": 1}, "q09": {"a": "o\n\n（k-a-l-e-i-d-**o**-s-c-o-p-e）", "ok": 1}, "q10": {"a": "火星です。（最小は水星、2番目に小さいのが火星です）", "ok": 1}, "q11": {"a": "**できない。**\n\n2Lと4Lはどちらも2の倍数(最大公約数が2)のため、移し替えや注ぎ足しで作れる量はすべて2の倍数(2L、4L、6L…)になります。奇数である3Lは測れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "**執刀医は息子の父親**です。\n\nこのなぞなぞの元の形は「父親が事故死し、執刀医が『私の息子だ』と言う」もので、答えは「執刀医は母親」——「外科医＝男性」という思い込みに気づかせる問題です。今回は母親が亡くなっているため、執刀医は父親と考えるのが自然です（同性カップルのもう一人の母親という答えも成り立ちます）。", "ok": 1}, "q14": {"a": "**どちらでも同じ**(勝率は各1/2)。\n\nこれは通常のモンティ・ホール問題と異なり、司会者が正解を知らずランダムに開けてたまたまハズレだった場合、その情報によりAとCの確率は両方1/2に更新されます。切り替えの優位性は消えます。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "15文字で、該当する単語は2つです（extraordinarily と synchronization）。", "ok": 1}, "q18": {"a": "**3**\n\n「ちょ・う・ちょ」と3拍(モーラ)に分かれます。拗音「ちょ」は1拍、長音「う」も独立した1拍として数えます。", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3(≈0.333…)", "ok": 1}, "q20": {"a": "**存在しない。**\n\n1往復では最大1つしか対岸に運べず、残る2つのうち「オオカミとヤギ」または「ヤギとキャベツ」の危険な組み合わせが必ず岸に残るため、全てを無事に運ぶ手順は成立しません(通常は7回の川渡りが必要です)。", "ok": 1}}}, "teai/max": {"single_score": 20, "single_sec": 363, "single_out_tokens": 17269, "batch_score": 18, "batch_sec": 32.3, "batch_out_tokens": 701, "single_graded": {"q01": {"a": "少年の父親です。\n\n一緒にいた母親が亡くなっているため、「この子は私の息子だ」と言った外科医は父親と考えるのが自然です(なお、同性カップルの場合はもう一人の母親という可能性もあります)。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない(2026年は閏年ではないため、2月は28日までです)", "ok": 1}, "q04": {"a": "extraordinarily(15文字)", "ok": 1}, "q05": {"a": "3", "ok": 1}, "q06": {"a": "2", "ok": 1}, "q07": {"a": "できない。ボートが1往復しか使えないため、一度に運べるのは1つだけで、全部を対岸に移すことは不可能です。", "ok": 1}, "q08": {"a": "水曜日です。(100 ÷ 7 = 14 余り 2 のため、月曜の2日後)", "ok": 1}, "q09": {"a": "o", "ok": 1}, "q10": {"a": "火星です。", "ok": 1}, "q11": {"a": "できない。2Lと4Lで測れる量はすべて2の倍数(最大公約数2の倍数)となるため、奇数の3Lは測れません。", "ok": 1}, "q12": {"a": "すまいざごうよはお", "ok": 1}, "q13": {"a": "執刀医は息子の父親です。\n\n(母親は事故で亡くなっているため、もう一人の親である父親が執刀医だったという答えになります。)", "ok": 1}, "q14": {"a": "どちらでも同じです。司会者が正解を知らずに偶然ハズレを開けた場合、Aが当たりの確率もCが当たりの確率も1/2となり、変えても有利にはなりません(標準的なモンティ・ホール問題とは異なります)。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最も文字数が多いのは15文字で、該当する単語は2つ（extraordinarily、synchronization）です。", "ok": 1}, "q18": {"a": "3", "ok": 1}, "q19": {"a": "0.25, 0.3, 0.33, 1/3(1/3 ≒ 0.333…)", "ok": 1}, "q20": {"a": "存在しない。ボートには一度に1つしか載せられないため、1往復(片道2回)では3つ全てを対岸へ運ぶことは不可能です。", "ok": 1}}}, "z-ai/glm-5.3-flash": {"single_score": 13, "single_sec": 72, "single_out_tokens": 1051, "batch_score": 13, "batch_sec": 5.4, "batch_out_tokens": 331, "single_graded": {"q01": {"a": "外科医は少年の父親です(または母親のパートナー)。\n\nこの古典的ななぞなぞは、外科医を男性と思い込むバイアスを試すものです。外科医が女性なら「母親」と答えたくなりますが、母親は死亡しています。つまり、外科医は少年の父親というのが答えです。", "ok": 1}, "q02": {"a": "10", "ok": 1}, "q03": {"a": "しない。2026年は平年（うるう年は4で割り切れる年だが、2026は割り切れない）なので、2月29日は存在しません。", "ok": 1}, "q04": {"a": "extraordinarily", "ok": 0}, "q05": {"a": "4", "ok": 0}, "q06": {"a": "3", "ok": 0}, "q07": {"a": "**できない。**\n\n理由：ボートが1往復で壊れる=農夫は片道1回しか渡れない。農夫+1つしか乗れないので、3つの荷物(オオカミ・ヤギ・キャベツ)を1回の片道では運びきれません。", "ok": 1}, "q08": {"a": "水曜日です。(100 ÷ 7 = 14余り2 → 月曜+2日)", "ok": 1}, "q09": {"a": "**e**（k-a-l-e-i-d-o-**s**-c-o-p-e… 7文字目は s です）\n\n訂正：kaleidoscope は k(1)a(2)l(3)e(4)i(5)d(6)o(7) なので、7文字目は **o** です。", "ok": 0}, "q10": {"a": "火星です。(水星が最も小さく、火星が2番目に小さい惑星です)", "ok": 1}, "q11": {"a": "**できる。**\n\n手順:\n1. 4L容器に満タン入れる(4L)\n2. それで2L容器を満タンにする → 4L容器に残り **2L**\n3. 2L容器を空にして、残り2Lを2L容器に移す\n4. もう一度4Lを満タンにし、2L容器に2L移す → 4L容器に **3L** 残る", "ok": 0}, "q12": {"a": "すみまんがお「すみまんがお」です。", "ok": 0}, "q13": {"a": "**父親**です。母は亡くなりましたが、執刀医は息子の父(母の夫)なので「私の子だ」と言えます。", "ok": 1}, "q14": {"a": "**同じ（変える必要なし）** です。\n\n司会者が正解を知らずにランダムに開け、たまたまハズレだった場合、残るAとCの確率は各1/2で同等。通常のモンティ・ホール問題（司会者が知っていて意図的にハズレを開ける）と違い、変えても有利になりません。", "ok": 1}, "q15": {"a": "5", "ok": 1}, "q16": {"a": "3", "ok": 1}, "q17": {"a": "最長は **15文字** で、該当する単語は **2つ**（extraordinarily と synchronization）です。", "ok": 1}, "q18": {"a": "**2**（ちょ・ちょ）", "ok": 0}, "q19": {"a": "0.25 < 0.3 < 0.33 < 1/3（≈0.333…）", "ok": 1}, "q20": {"a": "**存在しない。**\n\n1往復（＝行き1回＋帰り1回、計2回の渡河）では、最大でも2個しか対岸へ運べません。オオカミ・ヤギ・キャベツの3つを運ぶには最低7回の渡河が必要なため、1往復では不可能です。", "ok": 1}}}}}