NanoClassify · Banking77

Submissions

Receipt-backed Banking77 development results. 400 fixed examples; higher is better.

Start a submission ↗

Current record · Accuracy

87.75%

Tinker SFT + CISPO v5by Synth

Tinker SFT + CISPO v5

GPT-OSS-20B · 64 rollouts · 50 updates · Not recorded train time · Not recorded cost

87.75% · Evidence ↗

Stepped line = best result so far · dots = evaluated submissions · zoomed scale

Leaderboard

Existing entries predate the PR submission flow, so their source link points to the public-release commit. New entries link their submission PR.

Rank	Submission	Accuracy	Macro-F1	Train time	Cost	Submitter	Source
1	Tinker SFT + CISPO v5GPT-OSS-20B · 64 rollouts · 50 updates	87.75%	86.78%	Not recorded	Not recorded	Synth	Commit ↗
2	Tinker SFT v1GPT-OSS-20B · rank-16 LoRA · 100 updates	87.50%	86.29%	7m 32s	Not recorded	Synth	Commit ↗

Start here

Explore with Workshop

Download Workshop and give its coding agent the submission prompt. It reads the contract, chooses a parent, prepares one bounded Banking77 experiment, and stops before publishing or heldout evaluation.

Download Workshop →GitHub repository ↗Submission template ↗

Using your own provider key is billed by that provider and does not draw from a Synth allowance. Workshop asks separately before Synth-backed paid compute.

Agent submission prompt

Credential-safe and ready to paste

Help me create a new NanoClassify submission.

First read README.md, CONTRIBUTING.md, submissions/README.md, records/leaderboard.json, and the closest existing submission recipe. Then propose and implement one bounded, interpretable intervention for the Banking77 development benchmark. Start from an existing public submission when possible and clearly name its parent.

Work only with the published development interface. Never request, reconstruct, inspect, or tune against heldout membership, heldout gold labels, or per-example heldout predictions. Do not run a sealed-heldout evaluation. Keep provider credentials in a project-local .env and use only credentials already configured there.

Create a complete submission directory from submissions/template. Record the model, parent, split digest, random seeds, decoding settings, training settings, aggregate cost and usage, accuracy, macro-F1, valid-label rate, failures, and artifact paths. Preserve the development artifacts needed to reproduce the aggregate result. Run the relevant tests and scripts/validate_public_release.py before finishing.


…

Leaderboard

Existing entries predate the PR submission flow, so their source link points to the public-release commit. New entries link their submission PR.

Rank	Submission	Accuracy	Macro-F1	Train time	Cost	Submitter	Source
1	Tinker SFT + CISPO v5GPT-OSS-20B · 64 rollouts · 50 updates	87.75%	86.78%	Not recorded	Not recorded	Synth	Commit ↗
2	Tinker SFT v1GPT-OSS-20B · rank-16 LoRA · 100 updates	87.50%	86.29%	7m 32s	Not recorded	Synth	Commit ↗

Explore with Workshop

Using your own provider key is billed by that provider and does not draw from a Synth allowance. Workshop asks separately before Synth-backed paid compute.

Help me create a new NanoClassify submission. First read README.md, CONTRIBUTING.md, submissions/README.md, records/leaderboard.json, and the closest existing submission recipe. Then propose and implement one bounded, interpretable intervention for the Banking77 development benchmark. Start from an existing public submission when possible and clearly name its parent. Work only with the published development interface. Never request, reconstruct, inspect, or tune against heldout membership, heldout gold labels, or per-example heldout predictions. Do not run a sealed-heldout evaluation. Keep provider credentials in a project-local .env and use only credentials already configured there. Create a complete submission directory from submissions/template. Record the model, parent, split digest, random seeds, decoding settings, training settings, aggregate cost and usage, accuracy, macro-F1, valid-label rate, failures, and artifact paths. Preserve the development artifacts needed to reproduce the aggregate result. Run the relevant tests and scripts/validate_public_release.py before finishing. …