Skip to main content
MICHIEL HORSTMAN
← Back home

RESEARCH / MODEL EDITING

Changing what a model can do by editing its weights.

Qwen3.5-0.8B improved by editing its existing weights. Knowledge from other networks and a published reasoning repair are written into four of the model's own tensors: 0 added parameters, same architecture and size (873.4M), no donor model or custom code at inference. About 3.28M of 873.4M weights changed (0.38%).

MORE WORK — BLASPHEMOUS, CODE CARTOGRAPHER, OPEN VISION

Before and after

Showing the NKC model. MATH-500, thinking mode: 53%; GSM8K, thinking mode: 76%; IFEval strict prompt-level, thinking mode: 62%; Push-T: will the pusher move the block?: 88.4%; Depth (NYU): which point is farther from the camera?: 82.6%.

  • MATH-500, thinking mode

  • GSM8K, thinking mode

  • IFEval strict prompt-level, thinking mode

  • Push-T: will the pusher move the block?

  • Depth (NYU): which point is farther from the camera?

On scenes where some pushes move the block and others do not, NKC ranks a moving push above a static one 90-94% of the time (original: 48-62%; a model that only judged the scene would score 50%).

Two edits, four tensors

  • Layers 15 and 19, mlp.down_proj weights (130,188 entries): removal of a reasoning-loop direction from 64 output columns per layer. This is the published edit RikiyaT/Qwen3.5-0.8B-weight-space-programming (run p50c), applied bit-exactly.
  • model.visual.merger.linear_fc2, weight and bias (3,146,752 entries): a rank-4 update (depth channel rank 1 plus physics channel rank 3). Knowledge sources: Depth Anything V2 Small and JEPA-WM (Push-T world model).

Every number

Text benchmark results, original model versus NKC
BenchmarkOriginalNKC
MATH-500 thinking (100 problems)6%53%
GSM8K thinking (50)16%76%
IFEval thinking (50)20%62%
Answers cut at 32k tokens (MATH-500 / GSM8K / IFEval)96% / 82% / 78%32% / 2% / 2%
MATH-500 default mode (500)46.6%44.4%
GSM8K default mode (1,319)57.7%55.4%
IFEval default mode (541)53.0%48.8%
Vision benchmark results in percent, except caption NLL; original, NKC and change
BenchmarkOriginalNKCChange
NYU depth, "closer"77.182.2+5.2 [+2.4, +7.9]
NYU depth, "farther"75.382.6+7.2 [+3.8, +10.9]
NYU depth, reworded prompt84.086.2+2.2 [-0.3, +4.8]
Push-T, "will it move"50.088.4+38.4 [+34.9, +41.7]
Push-T, "will it stay"50.089.2+39.2 [+36.1, +42.4]
Push-T, reworded (never used in calibration)50.071.8+21.8 [+16.5, +26.9]
BLINK relative depth64.566.9+2.4
BLINK spatial relation62.262.9+0.7
BLINK counting50.054.2+4.2
BLINK object localisation66.453.3-13.1
COCO localisation (321)81.377.3-4.0
COCO caption NLL (lower is better)2.832.91+0.08

Three steps

  1. Reasoning-loop repair (published, reproduced). With thinking on, the base model loops until the 32,768-token cap on most problems. The published p50c edit subtracts the loop direction from the MLP output columns that write it most (layers 15 and 19), applied with the authors' own script after verifying the base revision by tensor hashes.
  2. Depth (amplifying an existing ability, training-free). Depth Anything V2 depth maps on unlabeled COCO images are read out of Qwen's own 3,072 merger neurons with closed-form ridge regression (held-out R^2 = 0.36) and written along the language model's own "near minus far" token-embedding direction. No gradient steps.
  3. Push-T physics (introducing a new ability, small calibration). JEPA-WM's predicted latent change supervises the edit through an interference-regularised generalised ridge readout, calibrated in 300 Adam steps on 2,400 scenes labelled only by the world model.

What this does not show.

  • The math gain is a repair of thinking-mode looping; it does not add mathematical knowledge. In default mode the edit costs 2 to 4 points. Use thinking mode for math.
  • Object localisation and caption likelihood regress; the visual edits sit in the patch merger, which never sees the question, so they reach every image task.
  • The physics ability is a narrow, learned move/stay decision for Push-T-style scenes (reworded prompt: 71.8%), not general physical reasoning.
  • Thinking-mode subsets are small (50 to 100 problems; about plus or minus 5 to 7 points of sampling noise).

Base model: the Qwen team. Reasoning edit: RikiyaT et al., Sparks of Weight-Space Programming. Depth donor: Depth Anything V2 Small. Physics donor: JEPA-WM (Push-T checkpoint).

More research and projects

Surface. Thanks for diving.

Shark model by alextkd2003, CC BY, converted to ASCII.