TEL-01 review fixes: stop the age sensor tripping installs that have no meter

T-1, and it was a fleet-wide trip to zero. publish() emitted p1_age
unconditionally, and published_age_s counts from P1Ingest.__init__ when no
sample has ever arrived. With meter_source defaulting to off, every existing
install would have published sensor.p1_sample_age_s climbing without bound; the
ESP32 does `has_state() && state >= max_age_s` and forces the layer-1 failsafe,
so each of them would have pinned its inverter at 0 W within 30 s. Exactly the
opposite of the zero-regression the off default was for. The key is now omitted
from the payload AND from MQTT discovery when P1 is off, so the entity does not
exist at all - which is the status quo, and what has_state() is testing for.
The predicate is one function, is_enabled(), because the grid reading, the task
start and the discovery announcement have to agree or this comes back.

T-2, connect no longer manufactures a sample. get_states returns whatever HA
currently holds, which after a Core restart is a RestoreEntity value of unknown
age; stamping it with ingest_ts=now reset the age and reported a fresh meter
that could have been dead for an hour. run()'s own docstring already said a
reconnect must emit nothing - the code disagreed with it, and a test asserted
the violation. The cache is still primed, so the first real state_changed
builds a complete sample; the age just stays honest until one arrives.

T-3, gaps are no longer filled with the last held value. The averager held a
sample forward across any interval, so a meter dying at 5 kW and returning ten
minutes later credited 5 kW x 600 s to the capacity-tariff accumulator - a
fabricated peak on a permanent record. The hold is capped at max_age_s: past
that the stretch is walked so block boundaries still land correctly, but
nothing accumulates and elapsed does not grow, which is what finally makes the
comment about a gap dragging the billed average down true. Same threshold for
control and billing: a reading too old to steer by is too old to bill by.

T-4, the out-of-order/duplicate guard is covered. It was untested, and the
reason is worth recording: the obvious assertion passes without the guard,
because the negative interval is separately refused by the covered > 0 test.
What the guard prevents is the timestamp REWIND, which only shows up one sample
later as a re-integrated window. The test now goes one sample later.

T-6, DOCS was wrong about latency. meter_max_age_s and stale_input_s stack, so
meter death to 0 W is 45 s and not 30. Documented as a table with both clocks.

Also documented the T-5 asymmetry rather than papering over it: the age
measures arrival, not change, so a stuck MQTT bridge republishing its last
telegram still looks fresh. Correct on ha_dsmr, not detectable on mqtt_p1
without a change-detector. Written up as a known limit.

Writing the T-1 test caught a second defect in the test itself: it recorded
only MQTT topics, and object_id lives in the payload, so "the age sensor is not
announced" had been passing for the wrong reason.

test_p1.py: 99 -> 122 checks. 14 mutations run, all 14 red, files restored
byte-identical - including one per fix above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa
This commit is contained in:
glenn schrooyen
2026-08-24 22:16:34 +02:00
co-authored by Claude Opus 5
parent 2097b7aaf6
commit 0a617c5482
5 changed files with 245 additions and 22 deletions
+145 -5
View File
@@ -229,6 +229,47 @@ before = a.partial_ws
a.add(sample(1000.0, at=BASE + timedelta(seconds=5)))
check("an out-of-order telegram is dropped, not integrated backwards",
a.partial_ws == before and a.elapsed_s == 10.0)
# ⚠️ The assertion above is NOT sufficient on its own, and that is the whole
# lesson: deleting the guard still passes it, because the negative interval is
# separately refused by the `covered > 0` test. What the guard actually prevents
# is the REWIND - without it the held timestamp moves back to +5 s and the next
# telegram re-integrates the 5..10 s window that was already counted. The damage
# only becomes visible one sample later, so the test has to go one sample later.
a.add(sample(1000.0, at=BASE + timedelta(seconds=20)))
check("...and the held timestamp is not rewound, so the next telegram "
"cannot double-count", a.elapsed_s == 20.0 and a.partial_ws == 20000.0)
# A duplicate telegram (identical timestamp) is the same rule.
a = QuarterAverager(1)
a.add(sample(1000.0, at=BASE))
a.add(sample(1000.0, at=BASE + timedelta(seconds=10)))
a.add(sample(4000.0, at=BASE + timedelta(seconds=10)))
a.add(sample(1000.0, at=BASE + timedelta(seconds=20)))
check("a duplicate timestamp neither re-integrates nor replaces the held value",
a.elapsed_s == 20.0 and a.partial_ws == 20000.0)
# A gap must not be filled with the last held value. The meter dies at 5 kW and
# returns ten minutes later; hold-forward would credit 5 kW x 600 s to the
# capacity-tariff accumulator - a fabricated peak, on a permanent record, from
# data nobody measured.
a = QuarterAverager(1, max_hold_s=30.0)
a.add(sample(5000.0, at=BASE))
a.add(sample(5000.0, at=BASE + timedelta(seconds=600)))
check("a 600 s gap is held for at most max_hold_s, not for the whole gap",
a.partial_ws == 5000.0 * 30.0)
check("the unobserved stretch does not count as elapsed time", a.elapsed_s == 30.0)
closed = a.add(sample(5000.0, at=BASE + timedelta(seconds=900)))
check("the outage drags the billed quarter down instead of inventing a peak",
len(closed) == 1 and abs(closed[0].offtake_avg_w - 300000.0 / 900.0) < 1e-9)
check("...nowhere near the 5000 W a hold-forward would have billed",
closed[0].offtake_avg_w < 400.0)
# The cap must not disturb a normally-spaced stream.
a = QuarterAverager(1, max_hold_s=30.0)
for i in range(0, 121, 5): # a healthy 5 s telegram cadence
a.add(sample(2000.0, at=BASE + timedelta(seconds=i)))
check("a healthy 5 s cadence is untouched by the hold cap",
a.elapsed_s == 120.0 and abs(a.offtake_avg_w - 2000.0) < 1e-9)
# --------------------------------------------------------------------------- #
print("ingest timestamp, age and staleness")
@@ -482,9 +523,15 @@ async def _e2e():
live, wire = asyncio.run(_e2e())
check("the websocket handshake and subscription complete", live.samples >= 1)
# Six state_changed events arrived (two per telegram). The debounce is what
# makes that three consistent samples instead of six half-updated ones.
check("three telegrams produce three samples, not six", live.samples == 3)
# ⚠️ TWO, not three. get_states primes the cache but must NOT build a sample:
# HA returns whatever it currently holds, which after a Core restart is a
# RestoreEntity value of unknown age, and stamping that with ingest_ts=now
# resets the age and reports a fresh meter that may have been dead for an hour.
# Only the two real state_changed telegrams become samples. Four state_changed
# events arrived (two per telegram); the debounce is what makes those two
# consistent samples rather than four half-updated ones.
check("connecting does not manufacture a sample from cached HA state",
live.samples == 2)
check("the final export-dominant telegram nets negative",
live.last.net_w == -800.0)
check("the sample was built over the wire, tagged with its transport",
@@ -492,9 +539,102 @@ check("the sample was built over the wire, tagged with its transport",
check("an entity we did not subscribe to is never cached",
"sensor.something_else" not in wire.cache and len(wire.cache) == 1)
check("a mid-stream unavailable is a parse error, not a sample",
live.parse_errors == 1 and live.samples == 3)
live.parse_errors == 1 and live.samples == 2)
check("the last good reading survives the unavailable", live.net_w == -800.0)
check("the averager integrated the live stream", live.averager.elapsed_s > 0.5)
check("the averager integrated the live stream", live.averager.elapsed_s > 0.2)
# The reason get_states still matters: it is what lets the FIRST real telegram
# build a complete sample instead of waiting for every entity to change once.
check("the primed cache let the first telegram build immediately",
live.samples == 2 and live.last.import_w == 0.0)
# --------------------------------------------------------------------------- #
print("the age sensor must not exist when P1 is off")
# ⚠️ This is a fleet-wide regression guard, not a nicety. The ESP32 watchdog
# does `id(p1_age_s).has_state() && id(p1_age_s).state >= max_age_s` and forces
# the layer-1 failsafe. published_age_s counts from P1Ingest.__init__, so if the
# age were published with meter_source off it would climb past 30 s on every
# existing install within half a minute and pin the inverter at 0 W forever.
from app.p1 import is_enabled # noqa: E402
from app.mqtt import SENSORS, MqttPublisher # noqa: E402
check("meter_source off is disabled", is_enabled({"meter_source": "off"}) is False)
check("a missing meter_source is disabled", is_enabled({}) is False)
check("an empty meter_source is disabled", is_enabled({"meter_source": ""}) is False)
check("ha_dsmr is enabled", is_enabled({"meter_source": SOURCE_HA}) is True)
check("mqtt_p1 is enabled", is_enabled({"meter_source": SOURCE_MQTT}) is True)
# The entity id SAFETY-01's firmware subscribes to, pinned by object_id.
row = [s for s in SENSORS if s[0] == "p1_age"]
check("the age sensor is declared exactly once", len(row) == 1)
check("its object_id pins entity_id to sensor.p1_sample_age_s",
row[0][1] == "p1_sample_age_s")
check("it is published in seconds", row[0][3] == "s")
class _RecordingClient:
def __init__(self):
self.sent = []
def publish(self, topic, payload=None, retain=False):
# Topic AND payload: object_id, the thing that actually pins the entity
# id, only appears in the discovery payload. Recording topics alone made
# the "is not announced" check pass for the wrong reason.
self.sent.append(f"{topic} {payload}")
def _announced(omit):
pub = MqttPublisher(None, 1883, omit=omit) # host None -> never connects
pub.client = _RecordingClient()
pub._announce()
return " ".join(pub.client.sent)
check("with P1 off the age sensor is never announced",
"p1_sample_age_s" not in _announced(("p1_age",)))
check("the other status entities are still announced with P1 off",
"goodwe_grid_power" in _announced(("p1_age",)))
check("with P1 on the age sensor IS announced",
"p1_sample_age_s" in _announced(()))
# And the publish dict itself, through the real Controller.
from app.main import Controller # noqa: E402
class _Store:
data = {}
def set(self, *a):
pass
def get_time(self, *a):
return None
class _Pub:
def __init__(self):
self.last = {}
def publish(self, values):
self.last = values
def close(self):
pass
pub_off = _Pub()
Controller({"meter_source": "off"}, None, _Store(), pub_off).publish()
check("with P1 off, p1_age is absent from the published payload",
"p1_age" not in pub_off.last)
check("...while the normal status keys are still published",
"setpoint" in pub_off.last and "grid" in pub_off.last)
pub_on = _Pub()
Controller({"meter_source": SOURCE_HA}, None, _Store(), pub_on).publish()
check("with P1 on, p1_age is published", "p1_age" in pub_on.last)
check("...as a number, so has_state() becomes true only once we feed it",
isinstance(pub_on.last["p1_age"], float))
print()
if fails: