TEL-01: P1 ingestion - two transports, the EMS-owned sign derivation, and sensor.p1_sample_age_s #2

2026-08-24T21:54:13+02:00

admin commented

2026-08-24 21:54:13 +02:00

Implements TEL-01. Stacked on SAFETY-04 - base is that branch, not release/1.0, so this diff shows only TEL-01 work. Nothing here touches control.py.

What this is

A Belgian P1 meter publishes two unsigned registers, not one signed figure. Until now the add-on asked the installer to bridge that with a template sensor, which put the sign convention of the entire control loop in a text box. This moves it into the EMS: net = import - export, derived once, in one place, with a test that fails if anyone inverts it.

Two transports behind one contract, selected by meter_source:

ha_dsmr - HA WebSocket, subscribing to the DSMR integration's entities
mqtt_p1 - a configurable topic, strict JSON schema documented in DOCS.md
off (default) - the existing meter_entity path, unchanged

Everything downstream reads P1Ingest, never a transport, so switching is a config edit. off being the default means no installed system changes behaviour until it opts in.

The sample-age entity

sensor.p1_sample_age_s - seconds since the newest accepted telegram, published over MQTT discovery, refreshed every second rather than only when a telegram lands.

The mechanism matters and is the reason SAFETY-01 asked for it: Home Assistant pushes a state only when the state changes. A meter sitting at a genuinely constant reading emits nothing at all, which is indistinguishable - to anything watching the value - from a meter that has died. Recomputing the age against the clock separates the two. On the ENV-01 rig that difference is a reproducible false trip roughly every six minutes.

The entity id deliberately breaks the goodwe_ prefix the other sensors use, because the firmware subscribes to that literal string. There is a warning comment on it in mqtt.py saying so.

Notable decisions

No inverter-side fallback, anywhere. Section 5.1: the inverter's own AC power correlates 0.998 with battery power and 0.09 with the real meter. A gap stays a gap - a reconnect emits no synthetic sample, and a rejected telegram never resolves to 0 W nor refreshes the timestamp. When P1 is stale, grid power reads as missing and the existing "inputs missing -> 0 W" path handles it.
Time-weighted quarter-hour averages, clock-aligned, offtake only. A mean of samples is biased by cadence: in the committed fixture a cadence change makes a per-sample mean read ~574 W where the true figure is 190 W.
Block alignment is floor(epoch/900). That is genuinely clock-aligned and DST-proof for Belgium because every offset there is a whole number of hours. Marked ponytail: with the ceiling named - a sub-hour offset (India +05:30) would need real tz maths.
Per-phase import kept separately from the total. On the section 17 unbalanced three-phase case the phase sum is 2100 W while the connection nets 600 W. Only one of those is billed; a naive sum picks the wrong one.
Ages measured with time.monotonic(), not the wall clock - an NTP step on a freshly booted Pi would otherwise fake or mask staleness.
Staleness is derived from the stored sample, not a separate flag. One immutable frozen dataclass and a single attribute rebind, so a reader can never see a fresh value with a stale flag or the reverse. TEL-02's flag-before-visible ordering is free rather than enforced.
Validation is strict at this boundary: unsigned registers may not be negative, JSON strings where numbers belong are refused, and the plausibility ceiling is 50 kW specifically so section 20 open question 5's 64954 (an unsigned 16-bit register decoded without its sign) is rejected rather than averaged in as 65 kW.

Tests

test_p1.py - 99 checks, plain asserts, bare interpreter, no meter, no framework. Follows test_control.py. Includes an end-to-end run of the HA transport against a fake Home Assistant websocket (real handshake, subscribe_events, get_states, per-entity events), which is the only thing that actually proves the protocol path works.

Full suite: test_control.py, test_arbiter.py, test_maintenance.py, test_p1.py all pass.

Non-vacuous: eight rules were deliberately broken one at a time and every one went red, then the file was restored byte-identical:

mutation	failures
invert the sign convention	24
drop the plausibility ceiling	2
accumulate signed power instead of offtake	1
attribute a boundary-straddling sample to one block	2
let a stale sample read as fresh	5
let a rejected telegram clear the last good sample	crash
ignore the phase count mismatch	3
trust receive time for a retained MQTT message	1

Not verifiable without a live meter

Real DSMR entity ids and units (the fixtures use plausible ones); whether a real P1 bridge's MQTT payload matches the documented schema; the 0.35 s debounce against real telegram timing; and MQTT broker reconnect against a real broker.

Implements TEL-01. **Stacked on `SAFETY-04`** - base is that branch, not `release/1.0`, so this diff shows only TEL-01 work. Nothing here touches `control.py`. ## What this is A Belgian P1 meter publishes two **unsigned** registers, not one signed figure. Until now the add-on asked the installer to bridge that with a template sensor, which put the sign convention of the entire control loop in a text box. This moves it into the EMS: `net = import - export`, derived once, in one place, with a test that fails if anyone inverts it. Two transports behind one contract, selected by `meter_source`: - `ha_dsmr` - HA WebSocket, subscribing to the DSMR integration's entities - `mqtt_p1` - a configurable topic, strict JSON schema documented in DOCS.md - `off` (default) - the existing `meter_entity` path, unchanged Everything downstream reads `P1Ingest`, never a transport, so switching is a config edit. `off` being the default means no installed system changes behaviour until it opts in. ## The sample-age entity `sensor.p1_sample_age_s` - seconds since the newest **accepted** telegram, published over MQTT discovery, refreshed **every second** rather than only when a telegram lands. The mechanism matters and is the reason SAFETY-01 asked for it: Home Assistant pushes a state only when the state *changes*. A meter sitting at a genuinely constant reading emits nothing at all, which is indistinguishable - to anything watching the value - from a meter that has died. Recomputing the age against the clock separates the two. On the ENV-01 rig that difference is a reproducible false trip roughly every six minutes. The entity id deliberately breaks the `goodwe_` prefix the other sensors use, because the firmware subscribes to that literal string. There is a warning comment on it in `mqtt.py` saying so. ## Notable decisions - **No inverter-side fallback, anywhere.** Section 5.1: the inverter's own AC power correlates 0.998 with battery power and 0.09 with the real meter. A gap stays a gap - a reconnect emits no synthetic sample, and a rejected telegram never resolves to 0 W nor refreshes the timestamp. When P1 is stale, grid power reads as *missing* and the existing "inputs missing -> 0 W" path handles it. - **Time-weighted quarter-hour averages**, clock-aligned, offtake only. A mean of samples is biased by cadence: in the committed fixture a cadence change makes a per-sample mean read ~574 W where the true figure is 190 W. - **Block alignment is `floor(epoch/900)`.** That is genuinely clock-aligned and DST-proof *for Belgium* because every offset there is a whole number of hours. Marked `ponytail:` with the ceiling named - a sub-hour offset (India +05:30) would need real tz maths. - **Per-phase import kept separately from the total.** On the section 17 unbalanced three-phase case the phase sum is 2100 W while the connection nets 600 W. Only one of those is billed; a naive sum picks the wrong one. - **Ages measured with `time.monotonic()`**, not the wall clock - an NTP step on a freshly booted Pi would otherwise fake or mask staleness. - **Staleness is derived from the stored sample, not a separate flag.** One immutable frozen dataclass and a single attribute rebind, so a reader can never see a fresh value with a stale flag or the reverse. TEL-02's flag-before-visible ordering is free rather than enforced. - Validation is strict at this boundary: unsigned registers may not be negative, JSON strings where numbers belong are refused, and the plausibility ceiling is 50 kW specifically so section 20 open question 5's `64954` (an unsigned 16-bit register decoded without its sign) is rejected rather than averaged in as 65 kW. ## Tests `test_p1.py` - **99 checks**, plain asserts, bare interpreter, no meter, no framework. Follows `test_control.py`. Includes an end-to-end run of the HA transport against a fake Home Assistant websocket (real handshake, `subscribe_events`, `get_states`, per-entity events), which is the only thing that actually proves the protocol path works. Full suite: `test_control.py`, `test_arbiter.py`, `test_maintenance.py`, `test_p1.py` all pass. **Non-vacuous:** eight rules were deliberately broken one at a time and every one went red, then the file was restored byte-identical: | mutation | failures | |---|---| | invert the sign convention | 24 | | drop the plausibility ceiling | 2 | | accumulate signed power instead of offtake | 1 | | attribute a boundary-straddling sample to one block | 2 | | let a stale sample read as fresh | 5 | | let a rejected telegram clear the last good sample | crash | | ignore the phase count mismatch | 3 | | trust receive time for a retained MQTT message | 1 | ## Not verifiable without a live meter Real DSMR entity ids and units (the fixtures use plausible ones); whether a real P1 bridge's MQTT payload matches the documented schema; the 0.35 s debounce against real telegram timing; and MQTT broker reconnect against a real broker.

admin force-pushed TEL-01 from 147456c2a2 to 0a617c5482

2026-08-24 22:17:20 +02:00

Compare

admin added 2 commits 2026-08-24 23:17:04 +02:00

TEL-01: P1 ingestion, with the derivation and the age the EMS owns f47f1f0129

A Belgian P1 meter publishes two UNSIGNED registers, not one signed figure.
Until now the add-on asked the installer to bridge that gap with a template
sensor, which put the sign convention of the whole control loop in a text box.
This moves it into the EMS: net = import - export, derived once, in one place,
with a test that fails if anyone inverts it.

Two transports behind one contract, chosen by `meter_source`: the HA WebSocket
subscribing to the DSMR integration's entities, and MQTT on a configurable
topic. Everything downstream reads P1Ingest, so switching is a config edit.
`meter_source: off` is the default and keeps the existing meter_entity path,
so no installed system changes until it opts in.

The other half is the timestamp. Every accepted sample is stamped at ingest
with a monotonic clock, `meter_max_age_s` is applied to it, and the age is
published as sensor.p1_sample_age_s for the ESP32's stale-input watchdog. That
entity is recomputed against the clock every second rather than only when a
telegram lands, because HA pushes state only on change: a meter frozen at a
constant reading emits nothing and looks, to anything watching the value,
exactly like a meter that has died. The age tells them apart.

Deliberately absent: any fallback to an inverter-side power figure. The
inverter's own AC power correlates 0.998 with battery power and 0.09 with the
real meter, so failing over to it means regulating against your own output.
A gap stays a gap - a reconnect emits no synthetic sample, and a rejected
telegram never resolves to 0 W or refreshes the timestamp.

Quarter-hour averages are time-weighted over clock-aligned blocks rather than
a mean of samples, so a cadence change cannot bias the capacity-tariff figure,
and only offtake is accumulated so a quarter of pure export averages to 0 kW.
Per-phase import is kept separately: on an unbalanced three-phase load the
phase sum and the connection net are different numbers, and only one of them
is billed.

test_p1.py: 99 checks, runnable with a bare interpreter and no meter. Includes
an end-to-end run of the HA transport against a fake Home Assistant websocket.

Stacked on SAFETY-04; nothing here touches control.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa

TEL-01 review fixes: stop the age sensor tripping installs that have no meter 4bd659c499

T-1, and it was a fleet-wide trip to zero. publish() emitted p1_age
unconditionally, and published_age_s counts from P1Ingest.__init__ when no
sample has ever arrived. With meter_source defaulting to off, every existing
install would have published sensor.p1_sample_age_s climbing without bound; the
ESP32 does `has_state() && state >= max_age_s` and forces the layer-1 failsafe,
so each of them would have pinned its inverter at 0 W within 30 s. Exactly the
opposite of the zero-regression the off default was for. The key is now omitted
from the payload AND from MQTT discovery when P1 is off, so the entity does not
exist at all - which is the status quo, and what has_state() is testing for.
The predicate is one function, is_enabled(), because the grid reading, the task
start and the discovery announcement have to agree or this comes back.

T-2, connect no longer manufactures a sample. get_states returns whatever HA
currently holds, which after a Core restart is a RestoreEntity value of unknown
age; stamping it with ingest_ts=now reset the age and reported a fresh meter
that could have been dead for an hour. run()'s own docstring already said a
reconnect must emit nothing - the code disagreed with it, and a test asserted
the violation. The cache is still primed, so the first real state_changed
builds a complete sample; the age just stays honest until one arrives.

T-3, gaps are no longer filled with the last held value. The averager held a
sample forward across any interval, so a meter dying at 5 kW and returning ten
minutes later credited 5 kW x 600 s to the capacity-tariff accumulator - a
fabricated peak on a permanent record. The hold is capped at max_age_s: past
that the stretch is walked so block boundaries still land correctly, but
nothing accumulates and elapsed does not grow, which is what finally makes the
comment about a gap dragging the billed average down true. Same threshold for
control and billing: a reading too old to steer by is too old to bill by.

T-4, the out-of-order/duplicate guard is covered. It was untested, and the
reason is worth recording: the obvious assertion passes without the guard,
because the negative interval is separately refused by the covered > 0 test.
What the guard prevents is the timestamp REWIND, which only shows up one sample
later as a re-integrated window. The test now goes one sample later.

T-6, DOCS was wrong about latency. meter_max_age_s and stale_input_s stack, so
meter death to 0 W is 45 s and not 30. Documented as a table with both clocks.

Also documented the T-5 asymmetry rather than papering over it: the age
measures arrival, not change, so a stuck MQTT bridge republishing its last
telegram still looks fresh. Correct on ha_dsmr, not detectable on mqtt_p1
without a change-detector. Written up as a known limit.

Writing the T-1 test caught a second defect in the test itself: it recorded
only MQTT topics, and object_id lives in the payload, so "the age sensor is not
announced" had been passing for the wrong reason.

test_p1.py: 99 -> 122 checks. 14 mutations run, all 14 red, files restored
byte-identical - including one per fix above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa

admin force-pushed TEL-01 from 0a617c5482 to 4bd659c499

2026-08-24 23:17:04 +02:00

Compare

admin commented

2026-08-28 01:19:01 +02:00

Closing: the content of this branch is already on release/1.0 — git rev-list --count origin/release/1.0.. returns 0.

The work shipped, but it reached the release branch by a different route (a direct push, or carried in with another merge) rather than through this PR, so Gitea never auto-closed it. That is a process failure on my part: it left the PR list looking like nothing had shipped, and it orphans any review discussion held here.

Nothing is lost and nothing needs re-doing. Going forward, work lands by merging its PR.

Closing: **the content of this branch is already on `release/1.0`** — `git rev-list --count origin/release/1.0..` returns **0**. The work shipped, but it reached the release branch by a different route (a direct push, or carried in with another merge) rather than through this PR, so Gitea never auto-closed it. That is a process failure on my part: it left the PR list looking like nothing had shipped, and it orphans any review discussion held here. Nothing is lost and nothing needs re-doing. Going forward, work lands by merging its PR.

admin closed this pull request

2026-08-28 01:19:01 +02:00

admin referenced this pull request

2026-09-19 14:19:34 +02:00

CI-9: a stacked PR is skipped, and a skipped check clears the merge gate #95

admin referenced this issue from a commit

2026-09-19 14:25:24 +02:00

CI-9: guard the trigger mechanically, and re-derive two figures I made up

admin referenced this pull request

2026-09-19 14:35:26 +02:00

CI-9: a stacked PR is skipped, and a skipped check clears the merge gate #95

admin referenced this pull request