OPT-11 P4 review: Cheap hours ignores the plan; midnight-stale forecast; stricter ids

- Cheap hours only fills to the ceiling with or without a forecast, as the
  design (section 1) says. Skipping the sun's share cost 0.31 EUR on real
  2026 prices (29.72 against P3's 29.41) and saved nothing on the P3 set.
- A forecast reading from before local midnight is stale: until the
  integration refreshes, "left today" is still yesterday's 0 and
  "tomorrow" is really today.
- Forecast entity ids must be sensor.[a-z0-9_]+ (they go into the
  /api/states/<id> path).
- House profile: a stored day that is not a date, or is after today, is
  dropped instead of kept forever.
- Text: the forecast line says "for the rest of today"; the learning line
  no longer promises "tomorrow" (a plan needs one full day).
- Tests for four surviving review mutants (hour sample floor, the dear run
  after the cheap run, share by room, daytime use) and the fixes above.
- Replay tool: `--fc zero` (a forecast of no sun).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
glenn schrooyen
2026-09-29 22:08:47 +02:00
co-authored by Claude Opus 5.5
parent aabc2a27fd
commit fbd884797d
10 changed files with 127 additions and 56 deletions
+1 -1
View File
@@ -545,7 +545,7 @@ With battery modes on, these choices replace the *Charging and selling on price*
- *Solar first* never charges from the grid.
- The house use is the add-on's own measurement (grid + battery power) per hour, over the last 7 days. It needs one day of it before the first plan.
- The plan is checked every 15 minutes. A planned level moves only by 1 kWh or 10 %, so a wobbling forecast does not flip it.
- No forecast, one older than 3 hours, or no house use yet: *Leftover solar* takes solar only and *Cheap hours only* fills to the top level. The Batteries screen says so.
- No forecast, one older than 3 hours or from before midnight, or no house use yet: *Leftover solar* takes solar only and *Cheap hours only* fills to the top level. The Batteries screen says so.
- A battery needs its capacity (`inverters.capacity_kwh`) for a plan.
**What always comes first.**
+18 -12
View File
@@ -12,8 +12,10 @@ The sums (OPT-11-design section 1, "Excess solar", with the owner's answers):
expensive run the cheap run is for, x the charge efficiency:
`trust x sum_h max(0, forecast_h - house_h) x EFF`.
* Solar first batteries take it first, up to the ceiling. What is left is
shared by the Leftover solar and Cheap hours batteries by their room.
* A Cheap hours battery charges from the grid up to `ceiling - its share`.
shared by the Leftover solar batteries by their room.
* Cheap hours only ignores the plan and fills to the ceiling, as in P3 and
the design (section 1). Skipping the sun's share cost money in the replay
(PR #156 review: 29.72 against 29.41 EUR on real 2026 prices).
* A Leftover solar battery charges only up to what the house will draw from
it in the expensive runs before the next cheap run, less its share:
`min(ceiling, floor + house use / EFF) - share`. So it buys only the
@@ -32,7 +34,7 @@ import statistics
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from .packmode import CHEAP_HOURS, LEFTOVER_SOLAR, SOLAR_FIRST
from .packmode import LEFTOVER_SOLAR, SOLAR_FIRST
from .tariff import BILLING_TZ
EFF = 0.95 # charge (and discharge) efficiency
@@ -67,6 +69,10 @@ class HouseProfile:
def load(self, data):
for day, hours in data.items():
try:
datetime.strptime(str(day), "%Y-%m-%d")
except ValueError:
continue
if not isinstance(hours, dict):
continue
row = {}
@@ -105,8 +111,11 @@ class HouseProfile:
return closed
def _prune(self, lt):
"""Keep the last `PROFILE_DAYS` days; a day after today (the clock
was wrong once) would never age out, so it goes too."""
keep = (lt.date() - timedelta(days=PROFILE_DAYS)).isoformat()
for d in [d for d in self.days if d < keep]:
today = lt.date().isoformat()
for d in [d for d in self.days if d < keep or d > today]:
del self.days[d]
def ready(self) -> bool:
@@ -142,7 +151,7 @@ class Plan:
sun_kwh: float # solar the batteries can count on
fc_kwh: float # the forecast over the same span
until: datetime # the expensive run the plan is for
targets: dict # name -> level % (LS and CH only)
targets: dict # name -> level % (Leftover solar only)
need_kwh: dict # name -> kWh from the grid (0: none)
share_kwh: dict # name -> kWh of solar counted for it
@@ -195,7 +204,7 @@ def plan(forecast, house, runs, now, packs, *, trust_pct, floor_pct,
usable = [p for p in packs if p.use and p.live and p.idle
and p.soc is not None and p.capacity_kwh]
sf = [p for p in usable if p.mode == SOLAR_FIRST]
price = [p for p in usable if p.mode in (LEFTOVER_SOLAR, CHEAP_HOURS)]
price = [p for p in usable if p.mode == LEFTOVER_SOLAR]
sf_room = sum(_room(p, ceiling_pct) for p in sf)
excess = max(0.0, sun - sf_room)
rooms = {p.name: _room(p, ceiling_pct) for p in price}
@@ -218,12 +227,9 @@ def plan(forecast, house, runs, now, packs, *, trust_pct, floor_pct,
for p in price:
sh = excess * rooms[p.name] / total_room if total_room > 0 else 0.0
sh_pct = sh / caps[p.name] * 100.0
if p.mode == CHEAP_HOURS:
top = float(ceiling_pct)
else:
want = use_kwh * caps[p.name] / cap_sum / EFF
top = min(float(ceiling_pct),
float(floor_pct) + want / caps[p.name] * 100.0)
want = use_kwh * caps[p.name] / cap_sum / EFF
top = min(float(ceiling_pct),
float(floor_pct) + want / caps[p.name] * 100.0)
level = max(0.0, top - sh_pct)
targets[p.name] = round(level, 2)
need[p.name] = round(_room(p, level), 3)
+11 -10
View File
@@ -24,10 +24,10 @@ Roles (the page's Why line names them):
Rules that keep this safe (tests: test_modes_p3.py):
* every range holds 0 W (a mode may forbid a direction, never force one);
* a Solar first battery never charges from the grid;
* P4: with a day plan (`targets`), a Leftover solar or Cheap hours battery
charges from the grid only up to its planned level, never above the
ceiling; with none, Cheap hours fills to the ceiling and Leftover solar
takes solar only (P3 option A, `charge_to()`);
* P4: with a day plan (`targets`), a Leftover solar battery charges from
the grid only up to its planned level, never above the ceiling; with
none it takes solar only (P3 option A). Cheap hours only fills to the
ceiling either way (`charge_to()`);
* grid charging needs a measured peak guard that is clear and a measured
headroom (SAFETY-45); the trade is sized to that headroom, whose aim is
`peak_cap_w - peak_margin_w` (SAFETY-79). It starts only at START_W or
@@ -98,12 +98,13 @@ def charge_to(p: Pack, ceiling_pct, targets=None):
`targets` is the day plan (`dayplan.plan()`: name -> level), or None
when there is none (no forecast, stale, or not configured). With no plan
this is P3 option A exactly: Cheap hours to the ceiling, the others never.
With a plan, a Leftover solar or Cheap hours battery charges only to its
planned level (the sun fills the rest), and a battery the plan leaves out
falls back to the no-plan answer. Solar first never charges."""
if targets is None or p.name not in targets:
return float(ceiling_pct) if p.mode in GRID_CHARGE_MODES else None
if p.mode not in PLAN_CHARGE_MODES:
With a plan, a Leftover solar battery charges only to its planned level
(the sun fills the rest); a battery the plan leaves out, Cheap hours only
included, gets the no-plan answer. Solar first never charges."""
if p.mode in GRID_CHARGE_MODES:
return float(ceiling_pct)
if targets is None or p.name not in targets \
or p.mode not in PLAN_CHARGE_MODES:
return None
t = targets[p.name]
return None if t is None else min(float(ceiling_pct), float(t))
+4 -4
View File
@@ -49,10 +49,10 @@ SELLING_ENABLED = False
# The modes that may charge from the grid in a cheap run with no day plan.
# Leftover solar takes solar only then (owner decision 2026-09-29, option A).
GRID_CHARGE_MODES = (CHEAP_HOURS,)
# OPT-11 P4: with a fresh solar forecast, the day plan (dayplan.py) may also
# let Leftover solar charge from the grid, by the shortfall only. Solar first
# never charges from the grid.
PLAN_CHARGE_MODES = (LEFTOVER_SOLAR, CHEAP_HOURS)
# OPT-11 P4: with a fresh solar forecast, the day plan (dayplan.py) lets
# Leftover solar charge from the grid, by the shortfall only. Cheap hours only
# ignores the plan (fills to the ceiling); Solar first never grid-charges.
PLAN_CHARGE_MODES = (LEFTOVER_SOLAR,)
def enabled(opts) -> bool:
+7 -4
View File
@@ -24,8 +24,9 @@ state only. The unit is read, never guessed (the rule `pv.watts()` has).
Without an hourly split, the day's energy is spread over a fixed bell from
06:00 to 20:00 local time (`BELL_FROM_H`, `BELL_TO_H`).
A reading older than `FRESH_S` is stale, and a stale or missing forecast is
no forecast: the day plan is off and every mode acts as in P3 option A.
A reading older than `FRESH_S`, or from before local midnight, is stale,
and a stale or missing forecast is no forecast: the day plan is off and
every mode acts as in P3 option A.
"""
import math
@@ -238,14 +239,16 @@ def read(today_state, tomorrow_state, now: datetime, *,
if today is None:
return None, why
at = _reported(today_state)
if at is None or (now - at).total_seconds() > FRESH_S \
# A reading from before local midnight is yesterday's "left today" (0
# after sunset) until the integration refreshes: stale, not a dark day.
if at is None or at < day0 or (now - at).total_seconds() > FRESH_S \
or (at - now).total_seconds() > 300:
return None, STALE
tomorrow = None
if tomorrow_state is not None:
t, _w = _day(tomorrow_state, day1, now, today=False)
t_at = _reported(tomorrow_state)
if t is not None and t_at is not None \
if t is not None and t_at is not None and t_at >= day0 \
and (now - t_at).total_seconds() <= FRESH_S:
tomorrow = t
return Forecast(today, tomorrow, at), None
+1 -1
View File
@@ -339,7 +339,7 @@
"tops_up": "Too little sun is forecast, so it charges some from the grid.",
"no_forecast": "No solar forecast, so it charges from solar only."
},
"solar": {"line": "Solar forecast today: {kwh}, we count on {count}. Updated {at}.", "no_forecast": "No solar forecast now, so Leftover solar charges from solar only.", "learning": "The add-on is learning your house's use. The plan starts tomorrow."},
"solar": {"line": "Solar forecast for the rest of today: {kwh}, we count on {count}. Updated {at}.", "no_forecast": "No solar forecast now, so Leftover solar charges from solar only.", "learning": "The add-on is learning your house's use. The plan starts after one full day."},
"today": {
"not_planned": "Not planned yet.",
"no_runs": "No cheap or expensive hours left today.",
+1 -1
View File
@@ -281,7 +281,7 @@
"tops_up": "Te weinig zon voorspeld, dus ze laadt wat van het net.",
"no_forecast": "Geen zonneprognose, dus ze laadt alleen met zonne-energie."
},
"solar": {"line": "Zonneprognose vandaag: {kwh}, we rekenen op {count}. Bijgewerkt om {at}.", "no_forecast": "Nu geen zonneprognose, dus Overschot zonne-energie laadt alleen met zon.", "learning": "De add-on leert eerst het verbruik van uw huis. Het plan start morgen."},
"solar": {"line": "Zonneprognose voor de rest van vandaag: {kwh}, we rekenen op {count}. Bijgewerkt om {at}.", "no_forecast": "Nu geen zonneprognose, dus Overschot zonne-energie laadt alleen met zon.", "learning": "De add-on leert eerst het verbruik van uw huis. Het plan start na één volle dag."},
"today": {
"not_planned": "Nog niet gepland.",
"no_runs": "Vandaag geen goedkope of dure uren meer.",
+4 -1
View File
@@ -328,6 +328,7 @@ WHAT IS DELIBERATELY *NOT* REFUSED, because it is untidy rather than unsafe:
"""
import logging
import re
from typing import NamedTuple
from . import dayplan, messages, pv, solarfc
@@ -2066,6 +2067,8 @@ def _margin_problems(opts: dict) -> list[str]:
TRUST_RANGE_PCT = (30, 100)
# A Home Assistant sensor id; it goes into the /api/states/<id> path.
_SENSOR_ID = re.compile(r"sensor\.[a-z0-9_]+")
FORECAST_KEYS = ("pv_forecast_entity", "pv_forecast_tomorrow_entity")
@@ -2083,7 +2086,7 @@ def _forecast_problems(opts: dict) -> list[str]:
e = opts.get(key)
if e is None or e == "":
continue
if not isinstance(e, str) or not e.strip().startswith("sensor."):
if not isinstance(e, str) or not _SENSOR_ID.fullmatch(e):
out.append(f"{S(key)} must be a sensor, like "
f"sensor.energy_production_today_remaining. It is "
f"{messages.shown(e)}.")
+75 -21
View File
@@ -116,15 +116,15 @@ check("the harness bites: Leftover solar grid-charging with no plan moves "
_read = solarfc.read
def _as_sunny(today, tomorrow, now, **kw):
return _read({"state": "200", "attributes": {"unit_of_measurement":
"kWh"},
def _as_dark(today, tomorrow, now, **kw):
return _read({"state": "0", "attributes": {"unit_of_measurement":
"kWh"},
"last_updated": now.isoformat()}, None, now)
check("...and an unavailable forecast read as a sunny one moves "
"n2_sf_ch_fc_down (Cheap hours would skip its charge)",
_moved("n2_sf_ch_fc_down", lambda: setattr(solarfc, "read", _as_sunny),
check("...and an unavailable forecast read as a dark one moves "
"n2_sf_ls_fc_down (Leftover solar would buy its shortfall)",
_moved("n2_sf_ls_fc_down", lambda: setattr(solarfc, "read", _as_dark),
lambda: setattr(solarfc, "read", _read)))
# ============================================================================ #
@@ -204,6 +204,18 @@ check("with it, tomorrow's sun is counted",
fcs, _ = solarfc.read(st("12.5"), st("20.0", at=old), NOW)
check("a stale tomorrow is dropped, today is kept",
fcs is not None and fcs.tomorrow is None)
# Review (PR #156): just after local midnight the integration may not have
# refreshed yet, so "left today" is still yesterday's 0 and "tomorrow" is
# really today. That is stale, not a dark day.
MID = datetime(2026, 9, 22, 0, 10, tzinfo=L).astimezone(timezone.utc)
EVE = datetime(2026, 9, 21, 21, 30, tzinfo=L).astimezone(timezone.utc)
check("a today reading from before local midnight is stale (not 0 kWh)",
solarfc.read(st("0.0", at=EVE), st("15", at=EVE), MID)
== (None, solarfc.STALE))
fcm, _ = solarfc.read(st("15", at=MID - timedelta(minutes=2)),
st("18", at=EVE), MID)
check("...and a tomorrow reading from before midnight is dropped",
fcm is not None and fcm.tomorrow is None)
# ============================================================================ #
print("3. the house profile")
@@ -241,6 +253,14 @@ check("only the last 7 days are kept", len(hp9.days) <= 8)
hp_rt = dayplan.HouseProfile(hp.dump())
check("dump and load give the same figures", hp_rt.hour_w(18) == round(
hp.hour_w(18), 1) and hp_rt.ready())
bogus = dayplan.HouseProfile({"2099-01-01": {str(h): 9e5 for h in range(24)},
"zz": {str(h): 9e5 for h in range(24)}})
check("a stored day that is not a date is dropped on load",
"zz" not in bogus.days)
bogus.add(UTC0, 500.0)
bogus.add(UTC0 + timedelta(hours=1), 500.0)
check("a stored day after today (a wrong clock) is pruned, not kept forever",
"2099-01-01" not in bogus.days)
check("a bad stored profile is ignored, not fatal",
not dayplan.HouseProfile({"x": "y", "2026-09-20": {"99": 1,
"a": "b"}}).ready())
@@ -316,11 +336,38 @@ ch = dplan(12.0, packs=[pk(A, packmode.SOLAR_FIRST, 60.0),
pk(B, packmode.CHEAP_HOURS, 30.0)])
ch_dark = dplan(1.0, packs=[pk(A, packmode.SOLAR_FIRST, 60.0),
pk(B, packmode.CHEAP_HOURS, 30.0)])
check("Cheap hours: dark fills to the ceiling (as P3)",
abs(ch_dark.targets[B] - 95.0) < 1e-6)
check("...and skips the part the sun covers on a partly sunny day",
ch.targets[B] < 95.0 and ch.share_kwh[B] > 0
and abs(ch.targets[B] - (95.0 - ch.share_kwh[B] * 10.0)) < 0.01)
check("...and on a partly sunny day too: Cheap hours ignores the plan "
"(design section 1; review of PR #156)", B not in ch.targets
and B not in ch_dark.targets
and modetrade.charge_to(pk(B, packmode.CHEAP_HOURS, 30.0), 95.0,
{B: 40.0}) == 95.0)
# Review of PR #156: four mutants that survived.
_hi = {h: (3000.0 if 17 <= h < 22 else 2000.0 if 8 <= h < 17 else 600.0)
for h in range(24)}
check("a house that uses more by day leaves less sun: a higher level",
dplan(20.0, est=(lambda h: _hi[h], 600.0)).targets[B]
> dplan(20.0).targets[B] + 1.0)
_two = dplan(40.0, packs=[pk(A, packmode.SOLAR_FIRST, 60.0),
pk(B, packmode.LEFTOVER_SOLAR, 30.0),
pk("inverter-c", packmode.LEFTOVER_SOLAR, 80.0)])
check("two Leftover solar batteries share the sun by their room (65:15)",
abs(_two.share_kwh[B] / _two.share_kwh["inverter-c"] - 6.5 / 1.5)
< 0.01)
_h = lambda hh: datetime(2026, 9, 21, hh, tzinfo=L).astimezone(timezone.utc)
_rp = priceruns.RunPlan(priceruns.Run(_h(13), _h(14), 0.02),
(priceruns.Run(_h(8), _h(9), 0.4),
priceruns.Run(_h(19), _h(20), 0.4)),
None, 0.3, None, True)
_pl = dayplan.plan(fc_of(12.0), EST, _rp, T0, PACKS, trust_pct=70,
floor_pct=20.0, ceiling_pct=95.0)
check("the plan is for the dear run after the cheap run, not an earlier one",
_pl is not None and _pl.until == _h(19))
_hp = dayplan.HouseProfile()
for i in range(30):
_hp.add(_h(10) + timedelta(seconds=5 * i), 500.0)
_hp.add(_h(11), 500.0)
check("an hour with under 5 minutes of readings is not kept",
not any(10 in r for r in _hp.days.values()))
check("trust 100 % counts more sun than 70 %: a lower level",
dplan(12.0, trust=100).targets[B] < partly.targets[B])
check("...and the sun counted scales with the trust exactly",
@@ -437,7 +484,8 @@ for i in range(5000):
bad.append(("range without 0", p, lo, hi))
if t.roles[p.name] == modetrade.CHARGE and (
p.mode == packmode.SOLAR_FIRST
or p.soc >= min(95.0, targets[p.name])):
or p.soc >= (95.0 if p.mode == packmode.CHEAP_HOURS
else min(95.0, targets[p.name]))):
bad.append(("charges past its level", p, targets))
check(f"decide() with a plan on 5000 random cases: ranges hold 0, nothing "
f"charges at or past its level, Solar first never ({len(bad)} bad)",
@@ -567,17 +615,17 @@ check("dark: ...only up to its level (the shortfall), not 95 %",
check("dark: A rests while B charges", all(r["roles"][A] == modetrade.REST
for r in charge))
# Cheap hours on a partly sunny day: skips the part the sun covers.
# Cheap hours on a partly sunny day: ignores the plan, fills as in P3.
rows, st_, c = run_day(SF_CH, fc=14.0, soc0={"a": 60.0, "b": 30.0})
charge = invariants("cheap hours, partly", rows, c)
tgt = nth([r["targets"][B] for r in rows if r["targets"]], 0, "a level")
top = max(r["soc"]["b"] for r in rows)
print(f" cheap hours partly: B level {tgt:.1f} %, reached {top:.1f} %")
check("Cheap hours with sun forecast: charges, but stops below 95 %",
charge and present(tgt) and tgt < 90.0 and top <= tgt + 2.0)
print(f" cheap hours partly: B reached {top:.1f} %")
check("Cheap hours gets no planned level",
not any(r["targets"] and B in r["targets"] for r in rows))
rows_nf, _s, c_nf = run_day(SF_CH, fc=None, soc0={"a": 60.0, "b": 30.0})
check("...and with the forecast down it fills towards 95 % as in P3",
max(r["soc"]["b"] for r in rows_nf) > top + 10.0)
check("Cheap hours with a sun forecast charges as with none (P3)",
charge and abs(max(r["soc"]["b"] for r in rows_nf) - top) < 1.0
and top > 80.0)
# The forecast goes away mid-run: Leftover solar stops at once (P3 option A).
DROP = UTC0 + timedelta(hours=1, minutes=30)
@@ -660,8 +708,8 @@ rows, st_, c = run_day(SF_LS, fc=1.0, soc0={"a": 60.0, "b": 30.0},
rp.m.datetime = datetime
sb = homeview.solar_block(c, c.solar_fc.at)
check("the forecast line names the forecast, what is counted and when",
sb["line"].startswith("Solar forecast today:") and "we count on" in
sb["line"] and sb["state"] == "plan")
sb["line"].startswith("Solar forecast for the rest of today:")
and "we count on" in sb["line"] and sb["state"] == "plan")
why = homeview.day_key(c, B)
check("B's Why key (plan, dark) is tops_up", why == "tops_up")
check("...and says so in plain words", homeview.why_words(why)
@@ -723,6 +771,12 @@ check("a forecast entity that is not a sensor is refused, in plain words",
validate._forecast_problems({"pv_forecast_entity": "number.x"})
and "must be a sensor" in validate._forecast_problems(
{"pv_forecast_entity": "number.x"})[0])
check("an entity id with a path or odd characters is refused",
all(validate._forecast_problems({"pv_forecast_entity": e}) for e in
("sensor.x/../config", "sensor.x?y=1", "sensor.X", "sensor.",
" sensor.x", "sensor.a b"))
and not validate._forecast_problems(
{"pv_forecast_entity": "sensor.energy_production_today_remaining"}))
check("empty entities are fine", not validate._forecast_problems(
{"pv_forecast_entity": "", "pv_forecast_tomorrow_entity": ""}))
notes = []
+5 -1
View File
@@ -20,6 +20,8 @@ days; the 2026-09-24 PV recording is only in the live Home Assistant):
* `national`: the Belgian day-ahead forecast itself, scaled to the site
(one factor: the site's total over the national total, all days). It
carries local weather differences too, so it is the harder test.
* `zero`: a forecast of no sun at all (is the gain the forecast's, or only
Leftover solar buying the evening's shortfall?).
* `high`: `national` x 1.5. `missing`: the sensor is configured but reads
`unavailable`. The sensor has Open-Meteo's shape (state = kWh left today,
`wh_period` = Wh per hour); `--shape state` drops `wh_period`
@@ -98,6 +100,8 @@ def forecast_table(kind, house, days):
day_of = {h: datetime.fromtimestamp(h, LOCAL).date() for h in pv}
if kind in ("perfect", "missing"):
out = dict(pv)
elif kind == "zero":
out = {h: 0.0 for h in pv}
elif kind == "recorded":
ratio = {}
for d in set(day_of.values()):
@@ -176,7 +180,7 @@ def main():
ap.add_argument("--variant", default="p4_ls")
ap.add_argument("--fc", default="recorded",
choices=("none", "recorded", "national", "perfect",
"high", "missing"))
"high", "missing", "zero"))
ap.add_argument("--prices", default="epex2025",
choices=("epex2025", "real2026"))
ap.add_argument("--shape", default="periods",