Planning Against the Model
DEV Community

Planning Against the Model

The last post gave Claude the room with the door and lost the first round. Claude fitted the physics from six readings, and my thirteen-weight model from 104, handed to Claude as a tool, talked it into a heating plan that left the room at 10.6 degrees.

The second round went the other way once the model borrowed the shape Claude had described: a leak proportional to the gap between inside and outside, a constant from the heater, and a second leak that comes and goes. That model got one code snippet and one table at the end of a long post, and a promise that this one would build it properly and put a planner on top.

So this post has two halves. The first walks through the physics-shaped model the way 104 walked through the thirteen weights. The second searches for heater schedules against it: first every schedule there is, then random guesses, then a method that learns where to guess, and finally a planner that replans every step and runs an evening against a thermostat. All of it is in tools/heater_plan.ts, and the playground now has the model and the planner in it.

The model Claude described

Here is the class. The private field, reset and observe are 104's, line for line. Only act is new.

class PhysicsModel {
  #field = 0;
  w: PhysicsWeights;
  constructor(w: PhysicsWeights) { this.w = w; }
  reset() { this.#field = 0; }
  get field() { return this.#field; }
  act(state: State, action: Action): State {
    const w = this.w, heaterOn = heaterAfter(state, action);
    const gap = state.outsideTemp - state.roomTemp;
    const change = (w.leak + w.doorLeak * this.#field) * gap + (heaterOn ? w.heat : 0);
    return { roomTemp: state.roomTemp + change, outsideTemp: state.outsideTemp, heaterOn };
  }
  observe(state: State, action: Action, nextState: State) {
    const f = inputs(state, action), w = this.w;
    const change = (nextState.roomTemp - state.roomTemp) / 5;
    this.#field = Math.tanh(w.uField * this.#field + w.uRoom * f.room + w.uOutside * f.outside + w.uHeater * f.heater + w.uChange * change + w.uBias);
  }
}

The 104 act had seven weights and let the gradient decide how room, outside, heater and field combined. This one has three and decides for it. The room and outside temperatures only ever appear as their difference, the heater only ever adds a constant, and the field only ever scales the leak.

Read as physics, that says heat flows out of the room in proportion to how much warmer it is than outside, the heater puts in a fixed amount, and something the log never mentions makes the room leakier. That is realRoom from 104, with the door replaced by a number the model has to infer.

Nothing in the logs or the training loop changed to accommodate this. The 4,800 rows are the same rows, and training is the same walk forwards and backwards through each episode, with three lines different.

The prediction error uses the new act:

const gap = (t.state.outsideTemp - t.state.roomTemp) / 10;
const err = (w.leak + w.doorLeak * field) * gap * 10 + w.heat * f.heater - target;

The gradients for the three prediction weights follow from it:

g.leak += s.err * s.gap;
g.doorLeak += s.err * s.field * s.gap;
g.heat += s.err * s.f.heater;

And the line that passes blame back to the previous step's field now asks how much the field mattered through the door leak:

dField = s.err * w.doorLeak * s.gap * 10 + dPre * w.uField;

The gap is divided by ten inside the loop and the two leak weights get a tenth of the learning rate. A gap of minus fifteen degrees would otherwise give the leak gradients fifteen times the push of the heater's, and the loop would spend its epochs bouncing across the valley the 103 post described.

With that, three thousand epochs over 4,800 rows take 454 milliseconds in Node 23. Nine weights, read back:

  • leak 0.102, uField 0.18
  • doorLeak 0.154, uRoom -0.24
  • heat 1.16, uOutside 0.24
  • uHeater 0.76
  • uChange -3.15
  • uBias -0.56

The left-hand column needs the field before it means anything, because the leak the model uses is leak + doorLeak * field. On the test room from 104 (22 degrees, minus four outside, heater on throughout, door open from step 10 to step 20) the field sits around -0.17 for the first ten steps, which puts the leak at 0.076 on average over those steps and 0.072 at step 5, where the planning question below starts. The true closed-door leak is 0.07.

The step after the door opens, the field jumps to 0.97 and the leak to 0.252, against a true 0.25. The heater weight is 1.16 against a true 1.2.

The model did not put "door shut" at a field of zero. It put it at about -0.17, and gave zero a leak of 0.102 that the room never has. The field's scale is the model's own business, as it was in 104, and only the leak that comes out of it can be checked against the room, which it matches.

The right-hand column is the same recipe for the field as last time. uChange dominates again: the field is mostly a reading of how far the room fell last step, with the heater and temperatures taken off. uField, its memory of itself, is 0.18 this time against 0.007 in 104, so this model holds on to the door slightly better.

Step by step through the door, next to the 104 model:

  • step 10, door closed, real room 17.19, physics-shaped, one step ahead 16.83, field after -0.23, 104, one step ahead 16.97, field after -0.15
  • step 11, door open, real room 13.14, physics-shaped, one step ahead 16.94, field after 0.97, 104, one step ahead 16.90, field after 0.91
  • step 12, door open, real room 10.10, physics-shaped, one step ahead 9.98, field after 0.96, 104, one step ahead 11.16, field after 0.86
  • step 13, door open, real room 7.76, physics-shaped, one step ahead 7.73, field after 0.93, 104, one step ahead 8.52, field after 0.81
  • step 20, door open, real room 1.94, physics-shaped, one step ahead 2.31, field after 0.64, 104, one step ahead 2.27, field after 0.59
  • step 21, door closed, real room 2.83, physics-shaped, one step ahead 1.91, field after -0.04, 104, one step ahead 1.77, field after 0.12
  • step 22, door closed, real room 3.55, physics-shaped, one step ahead 3.34, field after -0.11, 104, one step ahead 3.56, field after 0.15

Step 11 is a miss for both, as it has to be. At step 12 the physics-shaped model is a tenth of a degree out where the 104 model was over a degree out. Leaving out the two steps where the door moved, the one-step error over the thirty steps is 0.26 degrees against 0.32.

On short forecasts inside the range it was trained on, the two models are close, and not always in the physics-shaped model's favour. Asked at step 5 to run five steps blind, it says 16.92 against a real 17.19, where the 104 model said 17.12. At step 23, two steps after the door shut, its field has not quite settled and it forecasts 5.84 against a real 6.80, where the 104 model said 6.11.

Its fit to the log is slightly worse too, a loss of 0.179 against 0.172. A model with four fewer weights than its rival fits the data it was trained on a little less closely, which is what you would expect, and the difference between the two shows up outside that data.

Why the bias had to go

The test that broke the 104 model in 106 was the heater left off for twenty steps from the step 5 reading, at minus four outside, door shut. Every outside temperature in the log was between 2 and 12, so this is a long way outside it. Heater off for twenty steps from 19.15 degrees at minus four outside. Solid lines are the real room, averaged over 200 runs, the physics-shaped model and the 104 model. Dashed lines are the two fixes that failed: the physics-shaped model with a bias term, and the 104 model refitted on logs down to minus six. Both end colder than the air outside. Download the data (CSV)

The physics-shaped model ends at 1.15 against a real 1.42. The 104 model ends at 5.25. The two dashed lines are the two things I tried before settling on the model above, and both are worth a look, because both fit the log well.

The first is the physics-shaped model with a fourth prediction weight, a bias, added to act the way every model in this series has had one. Trained from the same starting point, it reaches a loss of 0.1793, against 0.1792 without it. On the log the two are indistinguishable. Its leak, with the field where the test room leaves it at step 5, is 0.068, which is about right. Its heater is 1.22 and its bias is -0.504, a constant half-degree pull downwards every step, and on the test room it ends at -3.96.

Set the change to zero with the heater off and the bias model says an unheated room settles where the leak balances the bias, 7.4 degrees colder than outside. On the rows in the log, the other weights pay that half degree back, and the predictions come out the same as the model without a bias. With the heater off for twenty steps in a cold room, nothing pays it back.

The log cannot tell the two models apart, and minus four outside can. The second is more data. If the problem is that the log never saw minus four, record a log that has, so I refitted the thirteen weights on 4,800 rows with outside temperatures from minus six to twelve. The loss came out at 0.227, and the heater-off room ended at -4.85, colder than the air outside it, which no room without a fridge in it can be. More rows of the wrong shape gave the wrong shape more confidence.

The bias came out because the physics has none, and because Claude's description in 106 had none. I made that call after seeing the bias version fail, so the result is not a blind test of the shape. The fair way to put it is that the shape is the part of the model that carries knowledge from outside the log, and the log only has to supply three numbers. If the shape is wrong, as with the bias, no amount of fitting will notice, because the error only shows up where there is no data.

Searching for the plan

Back to the question from 106. At step 5 the room reads 19.15, it is minus four outside, and the door is shut. Keep every reading from step 6 to step 25 at or above 12 degrees, with the heater on for as few steps as possible.

A planner needs a way to score a schedule, so here is the score. One point per heater step, a hundred per degree that any predicted reading falls short, and a hundredth of a point off for having room to spare, so that between two plans with the same number of heater steps the safer one wins.

function cost(temps: number[], heater: boolean[], target: number[]) {
  let c = 0, slack = Infinity;
  temps.forEach((t, i) => {
    if (heater[i]) c += 1; // one per heater step
    if (t < target[i]) c += 100 * (target[i] - t); // a hundred per degree short
    slack = Math.min(slack, t - target[i]);
  });
  return c - 0.01 * Math.min(1, Math.max(0, slack)); // and, between equals, the one with more room to spare
}

temps comes from rolling the model forward under the schedule, from the step 5 reading, with the field where observe left it. That is the "what if?" from 101, asked once per schedule.

Twenty steps of on or off is 1,048,576 schedules, and with a model this small, trying all of them takes about a second. The cheapest on the physics-shaped model is sixteen heater steps: off for four, then on for sixteen. Run in the real room 200 times with fresh draughts, it never drops below 12.

The same search on the 104 model finds thirteen, and the real room under that plan drops to 10.83 on average and fails in all 200 runs. With the true constants the answer is fifteen, which leaves a tenth of a degree to spare and dips below 12 in 75 of the 200 runs. Those three numbers are the 106 result again, reached without Claude doing the searching.

Trying everything stops working fast. Every extra step doubles the count, so forty steps would take about twelve days at the same speed, and a planner that decides every minute needs its answer well inside a minute. A real planner has to be satisfied with trying some of the schedules.

Random guesses, and better ones

The simplest thing that tries some of them is random shooting: draw schedules at random, score each one, keep the best.

function shoot(predict: (h: boolean[]) => number[], target: number[], n: number): Plan {
  let best: Plan = { heater: [], cost: Infinity };
  for (let k = 0; k < n; k++) {
    const heater = target.map(() => rand() < 0.5);
    const c = cost(predict(heater), heater, target);
    if (c < best.cost) best = { heater, cost: c };
  }
  return best;
}

The trouble is that the good schedules are rare. A random schedule has the heater on for about half its steps, and the answer needs sixteen of twenty, mostly at the end. I ran each method a hundred times from different seeds.

  • method every schedule, schedules tried 1,048,576, plans that keep the room at 12: 100 of 100, plans with 16 heater steps: 100 of 100, time per plan about 1 s
  • method random shooting 100, schedules tried 100, plans that keep the room at 12: 4, plans with 16 heater steps: 3, time per plan 0.1 ms
  • method random shooting 1,000, schedules tried 1,000, plans that keep the room at 12: 39, plans with 16 heater steps: 12, time per plan 0.7 ms
  • method random shooting 10,000, schedules tried 10,000, plans that keep the room at 12: 100, plans with 16 heater steps: 69, time per plan 6.3 ms
  • method random shooting 100,000, schedules tried 100,000, plans that keep the room at 12: 100, plans with 16 heater steps: 100, time per plan 63 ms
  • method cross-entropy method, 100 × 6 rounds, schedules tried 600, plans that keep the room at 12: 100, plans with 16 heater steps: 98, time per plan 0.5 ms
  • method cross-entropy method, 200 × 6 rounds, schedules tried 1,200, plans that keep the room at 12: 100, plans with 16 heater steps: 100, time per plan 0.8 ms

Random shooting gets there with a hundred thousand guesses. The last two rows get there with about a thousand, because they stop guessing blindly after the first round.

The cross-entropy method keeps a probability for each step that the heater is on, starting at a half. Each round it draws a batch of schedules from those probabilities, keeps the best tenth, and resets each step's probability to how often the best tenth had the heater on at that step. After one round the early steps drift towards off and the late steps towards on, and the next batch is drawn mostly from the neighbourhood of plans that worked.

function cem(predict: (h: boolean[]) => number[], target: number[], samples = 200, rounds = 6): Plan {
  let odds = target.map(() => 0.5);
  let best: Plan = { heater: [], cost: Infinity };
  for (let r = 0; r < rounds; r++) {
    const tried: Plan[] = [];
    for (let k = 0; k < samples; k++) {
      const heater = odds.map(p => rand() < p);
      tried.push({ heater, cost: cost(predict(heater), heater, target) });
    }
    tried.sort((a, b) => a.cost - b.cost);
    if (tried[0].cost < best.cost) best = tried[0];
    const elite = tried.slice(0, samples / 10);
    odds = odds.map((_, i) => elite.filter(e => e.heater[i]).length / elite.length);
  }
  return best;
}

All hundred runs at 200 a round found a s

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.