Sutro · MNIST competition · follow-up

New D3 and D5 records; three candidates stay unmerged

Two of five new pull requests qualified and merged. The spherical direction/radius kernel in #117 and tangent-jitter ensemble in #118 each passed all three fresh Modal A100 reviewer runs, setting D3 and D5 records at 51.772 ms and 3,335.395 ms per call. #115 missed accuracy, #116 failed timing dispersion, and #119 had two incomplete warmup failures. Those three remain unmerged. D1 and D2 keep their earlier qualified records, with every earlier source and failed run retained below.

The October 12 checks cover #115–#119, each with one initial, energy-enabled three-run reviewer set and a frozen source. #115–#118 each completed all 45 timed calls. #119 saved 15, seven and seven calls before two later warmup timeouts; those incomplete runs cannot qualify. No failed outcome was replaced or retried. Saved complete accuracy, timing and energy were independently reconstructed using the unchanged canonical scorer.

#117 merged in 49aa4bd; amended evidence head a787542. #118 merged in 36aeb22; amended evidence head f9acf50. Both qualified learners remain byte-for-byte unchanged.

D5 follow-up: Sung Jae’s 2.58 s author result is supported. Why #119 remains absent from the leaderboard: cold warmup timeouts and the additional maintainer review bar.

Latest review · October 12

Review has finished for all five initial sets. #117 and #118 merged; #115, #116 and #119 remain open and unmerged. Each source and its full or partial outcomes stays separate.

Merged · 3 / 3 reviewer passes

#117 · spherical kernel · D3

All three runs passed accuracy, timing and energy checks. Reviewer medians: 51.772184 ms and 8,839.065985 mJ above idle. Aggregate MNIST error: 2.613939% over 330,000 predictions.

Merged · 3 / 3 reviewer passes

#118 · tangent-jitter ensemble · D5

All three runs passed accuracy, timing and energy checks. Reviewer medians: 3,335.394886 ms and 638,234.154568 mJ above idle. Aggregate MNIST error: 1.824848% over 330,000 predictions; individual accuracy margins were 30, 153 and 65 labels.

Unmerged · 2 / 3 reviewer passes

#115 · native depth ensemble · D5

Run 2 scored 107,873 / 110,000: 1.933636% error against the 1.90% limit, a shortfall of 37 correct predictions. All 45 timed calls completed. No qualifying time or energy median.

Unmerged · 2 / 3 reviewer passes

#116 · upstream RBF kernel · D3

Run 2’s ninth call took 258.320374 ms, above the 99.658882 ms allowed maximum. Its MNIST median was 48.829441 ms. All 45 calls completed, but this timing failure prevents a qualifying time or energy median.

Unmerged · 1 / 3 complete reviewer passes

#119 · tangent refresh400 · D5

Run 1 passed all 15 calls. Runs 2 and 3 each saved seven timed calls, then exceeded the 90-second untimed warmup limit before call 8: KMNIST in run 2, Fashion-MNIST in run 3. They are incomplete qualification failures, with no ranked score or energy. No qualifying three-run median.

These decisions use the fresh reviewer sets. A complete qualifying run requires 11 MNIST and four foreign holdout calls. #115’s reviewer errors were 1.790000%, 1.933636% and 1.839091%; only two passed D5. The descriptive pooled error, 1.854242%, cannot override run 2’s accuracy failure. #116 passed accuracy in all three runs, but its timing outlier was also observed by the scorer’s parent clock; the record gives no independent infrastructure explanation that would justify replacing it.

Author records remain separate. All five candidates’ primary author energy records reconstruct as 3 / 3 passes: #115’s author median is 7,022.197998 ms / 1,978,525.562446 mJ; #116’s is 51.593931 ms / 8,499.141927 mJ; #117’s is 51.627899 ms / 8,706.872618 mJ; #118’s is 3,359.445468 ms / 677,766.483098 mJ; #119’s is 2,580.845581 ms / 536,878.610466 mJ. These author results do not substitute for any failed reviewer run. Accepted #117 and #118 leaderboard entries use their new reviewer sets above.

#119’s only complete reviewer run: 108,033 / 110,000 correct predictions (1.788182% error), 2,571.362149 ms and 517,731.021470 mJ above idle. These are one passing run’s measurements, not medians. The two other runs each stopped after seven saved calls and cannot be judged for qualifying accuracy. Their partial observed accuracy is descriptive only. The cause of the warmup stalls was not determined; no controller/setup exception or replacement run was reported.

Separate #119 variants remain retained. The 2,500-update, every-400-refresh comparison has 3 / 3 author passes at 3,385.826782 ms / 674,307.053343 mJ. The earlier 2,000-update one-refresh candidate has only 2 / 3 author passes; its failed run missed accuracy by 60 labels. Neither variant’s successful runs are pooled into the submitted periodic-refresh source, and neither replaces the fresh reviewer warmup failures.

Separate failed #117 development source: the one-update Newton–Schulz candidate remains preserved with 1 / 3 canonical passes. Its other two runs missed accuracy by 22 and 28 correct predictions. Those source bytes and outcomes are separate from the spherical two-update source accepted here.

#117 explicitly credits #116 and @islamborghini. The methods were evaluated on separate hidden draws; these records do not establish a speed or accuracy advantage over #116. The D3 comparison below is against the previously accepted native MLP record.

Four current qualified records

Merged · #111 · 3 / 3 passes

Difficulty 1 · error limit 5.40%

6.706 ms / call

D1 ranked time: previous 61.7 ms, Bae 6.706 msBars start at zero. Bae’s median ranked time is approximately 9.20 times faster than the previously listed 61.7 ms record. Prior recordBae · D1 61.7 6.706 9.20× faster · ranked time in milliseconds
Compared with the preceding D1 leaderboard record, a CUDA graph MLP at 61.7 ms. Bars use a zero baseline.
529.6 mJMedian GPU energy above idle / call
4.8445%Aggregate MNIST error · 330,000 labels

Two 60–256–256–10 MLPs, 30 updates and three-neighbor voting. Three distinct A100 80 GB GPUs; one PCIe and two SXM4 devices.

Merged · #112 · 3 / 3 passes

Difficulty 2 · error limit 3.40%

14.600 ms / call

D2 ranked time: previous 38.3 ms, Bae 14.600 msBars start at zero. Bae’s median ranked time is approximately 2.62 times faster than the previously listed 38.3 ms record. Prior recordBae · D2 38.3 14.600 2.62× faster · ranked time in milliseconds
Compared with the preceding D2 leaderboard record, Islam’s kernel PCG at 38.3 ms. Bars use a zero baseline.
1,332.6 mJMedian GPU energy above idle / call
3.2836%Aggregate MNIST error · 330,000 labels

Two 60–256–256–10 MLPs, 120 updates and three-neighbor voting. Three distinct A100 SXM4 80 GB GPUs.

Merged · #117 · 3 / 3 reviewer passes

Difficulty 3 · error limit 2.70%

51.772 ms / call

D3 ranked time: prior accepted native MLP 416.135 ms, spherical kernel 51.772 msBars start at zero. The spherical kernel reviewer median is about 8.04 times faster than the previous accepted native MLP record. This is a comparison of published results, not a paired rerun. Prior accepted · native MLPCurrent · spherical kernel 416.135 51.772 8.04× faster · ranked time in milliseconds
Against repaired native D3 #113’s previously accepted 416.135 ms. Its source and successful qualification history remain below.

8.04× faster than the preceding accepted D3 record. Earlier entries were not rerun on the same devices. This does not claim superiority over the separate upstream #116 candidate.

8,839.1 mJReviewer median GPU energy above idle / call
2.6139%Aggregate MNIST error · 330,000 labels

A spherical RBF kernel separates normalized input direction from standardized log radius and learns two analytic metric updates on direction. The solver and local class-geometry correction are derived from #116. Every call refits from supplied data; no pretrained state or query truth is used. Three distinct A100 SXM4 80 GB GPUs.

Reviewer errors: 2.643636%, 2.663636% and 2.534545%; each independently passes 2.70%. All 45 calls, exact source identities and complete energy windows were verified.

Merged · #118 · 3 / 3 reviewer passes

Difficulty 5 · error limit 1.90%

3,335.395 ms / call

D5 ranked time: prior established Ladder 14,130.6 ms, tangent-jitter ensemble 3,335.395 msBars start at zero. The tangent-jitter ensemble reviewer median is about 4.24 times faster than the prior established Ladder record. The separate 13,264.3 ms one-run entry was provisional and is not this baseline. This compares published results, not paired reruns. Prior established · LadderCurrent · tangent-jitter MLP 14,130.6 3,335.395 4.24× faster · ranked time in milliseconds
Against Seth’s established D5 Ladder record at 14,130.6 ms. The separate 13,264.3 ms one-run leaderboard entry was provisional.

4.24× faster than the preceding established D5 record. Earlier entries were not rerun on the same devices. The comparison uses each submission’s published time.

638,234.2 mJReviewer median GPU energy above idle / call
1.8248%Aggregate MNIST error · 330,000 labels

Laura’s eight RMSNorm/GELU MLPs use Muon/AdamW, 2,500 updates, tangent jitter fitted from the current labelled rows and cross-teaching on disjoint query pools. The joint train/query graph readout credits @sjbaebae. Inputs, weights, optimizer, EMA, pseudo-labels and pools are reset each call. Three distinct A100 SXM4 80 GB boards, each with a 500 W power limit.

Reviewer errors: 1.872727%, 1.760909% and 1.840909%; each independently passes 1.90%, with margins of 30, 153 and 65 correct predictions. All 45 calls, source identities and full energy windows were verified.

Comparisons use published leaderboard times, rather than reruns of the prior submissions on the same devices. Ranked time includes the canonical scorer’s clock floor. Energy is idle-adjusted GPU board energy, not whole-system energy. Aggregate error describes all 33 MNIST calls for each qualified source; qualification is checked separately for each run.

Earlier accepted D3 result · retained history

The repaired native MLP in #113 still has a valid three-run qualification. Its October 8–9 record and earlier K5-only failure remain distinct from the new spherical kernel source in #117.

Merged · repaired #113 · 3 / 3 reviewer passes

Difficulty 3 · error limit 2.70%

416.135 ms / call

2.78× faster than the preceding D3 leaderboard record, Islam’s kernel-tuned sampled Ladder at 1,157.1 ms. This comparison uses its published time; the prior submission was not rerun on the same devices.

98,226.0 mJMedian GPU energy above idle / call
2.5730%Aggregate MNIST error · 330,000 labels

Eight width-256 MLPs with three RMSNorm/ReLU hidden layers, 1,250 updates and K5 voting, followed by three-step query-graph propagation. The revised learner was frozen for all three initial reviewer runs.

October 8–9 submission history

The original October 8 review and its follow-ups covered seven pull requests. Five were merged: #108, #109, #111, #112 and the repaired #113. No additional PR was merged in this D4 recheck. D3 and D4 were each rechecked after the author changed their sources. Results below come from saved canonical reviewer records, independently checked against the unchanged scorer.

D3 repair follow-up merged . D4 repair rechecked (UTC). Earlier failed source versions remain documented below.

Current #114 status: open and unmerged. It became non-draft on October 9 at 20:16 UTC. Its reviewed head 74dc7ad… and learner bytes are unchanged, so the retained 1 / 3 repaired-source qualification still applies.

SubmissionDecisionInitial reviewer passesEvidence and remaining issue
D1 · #111Merged3 / 3Median 6.706 ms, 529.6 mJ. Per-run MNIST errors: 4.8982%, 4.8773%, 4.7582%; all below 5.40%.
D2 · #112Merged3 / 3Median 14.600 ms, 1,332.6 mJ. Per-run MNIST errors: 3.2491%, 3.3127%, 3.2891%; all below 3.40%.
D3 · repaired #113Merged3 / 3 repair
2 / 3 earlier source
Repair median 416.135 ms, 98,226.0 mJ. Per-run reviewer MNIST errors: 2.5736%, 2.4791%, 2.6664%; all below 2.70%. The historical K5-only failure is retained separately below.
D4 · revised #114Unmerged
Draft at the October 9 review
1 / 3 repair
1 / 3 earlier source
Repair reviewer run 1 scored 107,406 / 110,000: 2.358182% error, 64 correct predictions short. Run 2 scored 107,466 / 110,000: 2.303636% error, four short. Both completed all 15 timed calls but exceeded the 2.30% error limit. Run 3: passed at 2.217273% error (107,561 / 110,000), with 983.497 ms and 307,125.793 mJ above idle; these are one passing run’s measurements, not a three-run median. No qualifying median.

Every initial run is included, with source versions kept separate. The unchanged canonical scorer and Modal runner were used throughout. There were no qualification reruns to replace failures. D3’s graph-readout repair and D4’s cold-compilation repair each change the submitted source, so each received its own initial three-run reviewer qualification set. Earlier failures are retained below. D4’s revised source has two complete accuracy failures and no qualifying three-run median; passing author records do not substitute for these reviewer outcomes.

Historical D3 outcome · earlier K5-only source: two of three initial reviewer runs passed. Run 3 made 106,998 / 110,000 correct predictions: 2.72909% error against the 2.70% limit, a shortfall of 32 correct predictions. That source has no qualifying median. The repaired source adds three steps of query-graph propagation; it has a different SHA-256 identity and is assessed separately.

D4 follow-up · revised cold-compilation source: all three fresh reviewer runs completed their warmups and all 45 timed calls: 33 MNIST and 12 holdout calls. Runs 1 and 2 failed accuracy against the required 107,470 correct predictions per run. Their ranked scores are null and they have no energy result. Run 3: passed at 2.217273% error (107,561 / 110,000), with 983.497 ms and 307,125.793 mJ above idle; these are one passing run’s measurements, not a three-run median. Aggregate MNIST error across all 330,000 reviewer MNIST predictions is 2.2930%; this descriptive aggregate cannot override either individual run’s failure. The revised source has no qualifying time or energy median.

Historical D4 outcome · original source: one of three initial reviewer runs passed. Runs 2 and 3 exceeded the canonical 90-second untimed warmup limit on KMNIST before the second timed call. Those incomplete runs cannot qualify, and that source has no qualifying median. This history belongs to the original 8b7790… source, separately from the revised c5ed1d… source.

D4 repair scope: four C++ binding files replace the torch/extension header with tensor types and pybind casters; their four host builds use -O0 and -g0. All archived CUDA kernels, the exhaustive GEMM tuner and teacher/student configuration remain unchanged. All three reviewer runs completed warmup. Runtime attribution between compilation and tuning was not independently measured, and these records do not establish the cause of the accuracy failures.

Author-supplied D4 repair records: all three passing energy-enabled records were separately audited against the canonical scorer, including full energy reconstruction. Their median is 985.074 ms and 286,669.683 mJ, with aggregate error 2.225152%. These saved author outcomes remain separate from the fresh reviewer set and do not establish reviewer qualification.

Separate pruned-compilation diagnostic: an alternative source completed warmup but scored 107,468 / 110,000 correct predictions against the 107,470 required, a shortfall of two. Its failed no-energy record is preserved; it is not the submitted repaired source. The initial lean-compilation and bounded-tuning no-energy diagnostics are also retained separately.

Other pull requests reviewed

#108 · merged

Two MNIST leaderboard rows were checked by independently rescoring 330,000 saved predictions. The archived result met both reported accuracy bands.

#109 · corrected and merged

The rotating-eight matmul calculation now charges one selected weight matrix per call and records the eight-matrix bank separately. All 24 consistency checks passed; 23 timing and theory rows were independently verified. Raw measurements and the harness stayed unchanged. This was a saved-data review, without a GPU rerun.

#110 · non-record experiment

The requested A100 SXM4 40 GB diagnostic completed at 19.85 J and 193.52 ms per task. It reproduced the saved prediction hash for one existing draw. Prior selection exposure and the missing fresh accuracy protocol leave it ineligible for a record merge.

#109’s corrected times are compatible with its one-pass HBM estimate, which does not establish an HBM bottleneck. #110 measured the full learner invocation, including selection, refit, host projection and transfers; dataset loading was excluded. Its single draw does not qualify an 11-draw accuracy result.

How the records were checked

The October 12 source reviews found no static compliance blocker: current inputs are used for fresh fitting, caller tensors are preserved, and query truth is isolated. #115’s 30-file archive exactly matches its readable source manifest. #116 and #117 use kernel solvers and current-query pseudo-support. #118 and its credited #119 derivative freshly fit tangents and reset training state each call; #119’s shared query pools and 400-update refresh cadence are explicit. Merged #117 and #118 documentation adds reviewer evidence without changing either qualified learner. Fresh reviewer blockers were #115’s accuracy miss, #116’s timing dispersion and #119’s later warmup timeouts.

Earlier native submissions: The native submissions were inspected for label isolation and fresh training state. Source identities, sandbox status, accuracy gates, ranked-time reconstruction and saved energy gates were checked against the canonical records. Each complete scoring run contained 11 MNIST calls and four foreign holdout calls. D1 and D2 retain their original learner bytes in the merged submissions. The D3 repair uses the author’s revised learner unchanged; its fresh reviewer records are separate from both the earlier K5-only source and the author-supplied repair records. The author’s three passing energy-enabled D3 repair records and initial no-energy diagnostic are retained; the reported native D3 result comes from its new reviewer qualification set. D4’s cold-compilation repair was reviewed at frozen PR head 74dc7ad1eb3f0d5d4fd4c75ec1394b6fb8e1e9c1 with its submitted source unchanged. Its author records, initial diagnostic alternatives and fresh reviewer records remain separate. Two complete reviewer accuracy failures establish that the revised D4 source has no qualifying three-run median.

Frozen source identities · SHA-256

Every source version used the same canonical scorer and runner. These hashes distinguish the current spherical D3 and tangent-jitter D5, the three new unmerged candidates, and the earlier native D3 and D4 versions. Each reviewer set froze one learner; no source was changed or tuned within a set.

Current D3 learner · spherical kernel #117 · 3 / 3 reviewer passes
43c487a621cd5e563f5636181db4af74816daecfa37c90bf3f850cf699c1df51
Current D5 learner · tangent-jitter ensemble #118 · 3 / 3 reviewer passes
a3b18bac64dddc3f7aa8c614e2279932aa74040720cb861e8dfa40eaa67923c3
D5 learner · depth ensemble #115 · 2 / 3 reviewer passes
86a1a749f5b12c34be86c2e5025487827bda6b8b20f55fac72bb2911dcaa9066
Upstream D3 learner · RBF kernel #116 · 2 / 3 reviewer passes
13dfbe8b83a2cd92c422e576ff4b0b8b756c440f8132b61beb4ffa821b144dc4
D5 learner · tangent refresh400 #119 · 1 / 3 complete reviewer passes
6f0e1a109ece2679f8ee1e0527cd98abc6a58e664d11fc6033740765ba4a990f
Canonical scorer
1bc6d8d96a776565bc70f3ccdb91be7de57187d4495b2b0437fac98f5e120064
Canonical Modal runner
1180b843cf6c32414774da90c835a8b7298d2db8719989237a793ca1c62a10ab
D1 learner
d165b1cb8c395e3b2f3aa2f228c101818eaf389893b271b5d6d15358aaba8987
D2 learner
3f949ff304de18b537e660fcdbea24bf60697d2799294d8af348bff4f4971023
Historical D3 learner · earlier K5-only source · 2 / 3 reviewer passes
6c00a6f593d3e4015c9e6d0ce126fac7fd0519654906421682d631c394750083
Historical native D3 repair learner · three-step query-graph propagation
c6082659a49a6bd942077e359db42f28016626bb49303d68032467710d765a7e
Historical D4 learner · original source · 1 / 3 reviewer passes
8b7790e61ce658ba5ee0409c6b59014e4312331ae2a2f892ec57cbabeda81175
D4 repair learner · cold-compilation headers and host flags · reviewer accuracy failure
c5ed1d8ffb115cc58a6a6b4ac468b07c780d773b3f8d7387ec3f295b6f1b3eb1

Source reports and pull requests