doc-1-operator-docs
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  adminandClaude Opus 5
|
5d7e275df4 |
SAFETY-39: slew_w is a ratio to the rail, and gain is what is still wrong
checks / unit (pull_request) Successful in 5m16s
`slew_w: 1000` was derived on 2026-08-22 against a clamp of 1500 W, where "capping how far the loop can jump ahead of the inverter" was true because the rail itself bounded the excursion. The live install went to max_w 5000 and the constant did not move, so the loop crosses its whole output range in 10 s against a plant NOTES.md §5.10 measured at ~18 s. Measured on the owner's system 2026-09-05 11:43-11:46: a 3.4 kW export drove the setpoint 380 -> 5000 W in five 2 s ticks, the battery to -5154 W past a 5000 VA rating, and the grid from 3.4 kW of EXPORT to 2.8 kW of IMPORT. Nothing was broken; the rail moved. REPRODUCED FIRST, and one half of it is not reproducible at all. On plantsim.Plant and on an independent in-repo model: the setpoint stepping a full 1000 W per tick to the rail (48 of 51 export steps), and the meter crossing from export into import. The -5154 W figure is NOT reproducible in either model, because both clamp the plant at its rating - so the card's AC "battery no longer exceeds max_w in simulation" is unfalsifiable and was rewritten rather than met. THE DELIVERABLE IS THE RELATIONSHIP. SLEW_RAIL_FRAC = LOOP_TICK_S / PLANT_SETTLE_S = 2 s / 18 s = 1/9 slew_w = 0 now means FOLLOW THE RAIL - SLEW_RAIL_FRAC * max_w - exactly as integrator_max_w = 0 already means follow max_w. It is the shipped default: 222 W at max_w 2000, 556 W at 5000, 1067 W at a ~9600 W two-inverter site. validate.py rule 10 refuses an explicit value faster than that, graded against sum(unit_caps(opts)) - the roster sum, i.e. Controller.site_max_w - so a second inverter moves the bound with no operator action. VERIFIED BY BREAKING IT: restoring slew_w 1000 at max_w 5000 is refused, and the refusal names the rail, the crossing time, the measurement and what to do. Equality, one watt of slack, the live interim 400 and slew_w 0 at every rating all still start. GAIN IS THE CONSTANT THAT IS STILL WRONG, and it is NOT changed here. Over 51 export steps the over-absorption is FLAT in slew (958 W at 1000, 1081 W at 250) and monotone in gain (987 W at 0.6, 876 at 0.3, 435 at 0.2): slew_w bounds the OUTPUT, while the integrator moves gain*error per cycle and reaches a 5000 W rail in under three ticks whatever the slew is. Two control-law constants moving in one release on a running house makes the next measurement uninterpretable. ⚠️ N=2: the site rail becomes sum(max_w) ~ 9600 W (inverter B FROM DATASHEET AND SERIAL, not measured, and sim/plantsim.py's INVERTER_B is A copied), and the enforced ratio produces 1067 W there automatically. But a growing rail LOOSENS this bound, and the >= 900 W symptom goes stale at N=2 the same way the constant did - both stated in the record rather than left to be discovered. Nothing reached the inverter. The live install still runs the interim slew_w 400 applied by hand on 2026-09-05, which rule 10 accepts unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU |
||
|
adminandClaude Opus 5
|
f000547518 |
SAFETY-30: grade each inverter against its own power, and the site's total against the meter
checks / unit (pull_request) Successful in 4m38s
THE DEFECT, in one line. `inv.sign_wd.evaluate(stood, self.grid, ...)` compared
the command THIS controller put on ONE actuator against what the WHOLE SITE is
exchanging with the network. At a roster of one those are two views of the same
thing - specs.md §6.2's identity - and that is what makes the check sound and
what earned it the best record in the safety table (NOTES.md: 93 % agreement
while healthy against 4 % during runaway, four observed episodes, zero false
positives). At two inverters they are not, and it fails in BOTH directions:
FALSE TRIP inverter A commanded -1500 W while the meter reads +1500 W is
evidence of nothing when inverter B is discharging 3000 W exactly
as told. Nothing is faulty and the highest-value check fires.
MISSED the converse, and it is worse. A unit commanded +1500 W and
MEASURED at -1500 W, beside a sibling at +1500 W, leaves P1
exactly where a healthy site leaves it. The summation destroys the
evidence and no threshold recovers it.
DEV-02 had already shipped the per-INSTANCE half - one SignWatchdog per
inverter, namespaced latch, a trip on B no longer latching A. What was left was
the COMPARISON, and it is two questions rather than one:
PER UNIT this unit's standing command against ITS OWN measured power
(`Inverter.batt`, holding register 0x0518, per-instance since
DEV-02). Conclusion: "the actuator I am driving is not doing what I
told it" - a fact about one device, so the remedy stays the per-unit
`failsafe` SET that arbiter.CLAIM_SCOPE has classified SCOPE_UNIT
from the day it was written. That entry read "the evidence is
site-wide and the remedy is not"; the first half is now simply true.
SITE the SUM of the standing commands against the meter, as a new
`sign_site` SET at P_SAFETY on the site actuator, latched under flat
`site_sign_tripped` / `site_sign_reason`. This is the 2026-08-20
fault day and NO per-unit check can see it, because every inverter
may be faithfully executing its share of a number that is itself
wrong. It stops every inverter and the banner names the site.
WHAT IT IS NOT, said here because two guards that overlap without saying so is
how a false-alarm generator gets built. It is not SAFETY-31: circulation.py
compares the units AGAINST EACH OTHER and catches them pushing power THROUGH
each other, which nets to zero at the meter and about which no single command is
evidence. This compares each unit against ITS OWN COMMAND. They overlap in
exactly one scenario - a unit reversing while its sibling pulls the other way -
and test_watchdog.py asserts on that fixture that both fire and what each
concludes. It is also not the saturation freeze, which is a MAGNITUDE test
(|command - measured| > 500 W) that shapes the loop and fires on a merely
saturated unit; this is a DIRECTION test that latches and parks one.
THE FOUR NUMBERS, RE-DERIVED AT TWO INVERTERS - one carried with a justification
and one refused. 1890 closed-loop runs on admin/goodwe's TEST-19 two-master rig
in which NOTHING is faulty, swept over the whole range config.yaml accepts for
`gain` and `slew_w`, across load reversals with write staggers and flapping
loads, and RUN AT BOTH ROSTERS so the N=1 arm is the control:
unit_sign_disagree_w 250 `saturation_w / 2`. The check gates on both sides,
opposed, so the smallest gap it can see is 500 W -
exactly where compute()'s own anti-windup test
already says the inverter is not following. It
cannot fire on a divergence the control law calls
ordinary tracking. Worst legitimate opposition at
250 W: 10.0 s; the curve is nearly flat, because a
unit's own power chases its own command.
unit_sign_disagree_s 85 8.5x that 10.0 s, 2.8x the firmware's own 30 s
link veto (inverter.LINK_STALE_S), under half the
180 s site hold.
site_sign_disagree_w 300 CARRIED. Its argument is the meter's noise floor
and the 15 W deadband; the roster touches neither,
and the campaign agrees - the gate opens in 994 of
1050 runs at two inverters and 955 of 1050 at one.
site_sign_disagree_s 325 NOT CARRIED, and this is the number the card
exists to not assume. The worst legitimate
crossing at 300 W is 110 s with one inverter and
215 s with TWO, because the command must travel a
rail that is the roster sum at a `slew_w` that is
not - reachable at gain 0.05, slew_w 50 and a
2.2 kW load reversal, all schema-legal. 180 s
would have shipped a false trip. 325 s is 1.51x
the measured worst, inside the 900 s billing
quarter and inside every episode of the fault day.
AT ONE INVERTER NOTHING CHANGES, and "nothing" is asserted rather than assumed.
The two questions are the same sentence there, so only the detector that shipped
is armed: same comparison against the meter, same 300 W / 180 s, same reason
string byte for byte, same namespaced latch and flat rollback mirror, same
`failsafe` claim, same banner, same single status row - and the site detector is
NEVER CALLED, counted on the object over twelve cycles.
VERIFIED BY BREAKING IT. Thirteen mutations on fresh per-mutation trees with no
__pycache__, mounted :ro, with the battery refusing to report at all unless the
baseline is green. Twelve red on the first pass. M4 - the site half armed at N=1
as well - SURVIVED, because the check read `disagree_since is None` and the unit
detector trips first, parks the inverter and leaves the site standing at 0 W:
the assertion ran, the sentence was true, and the data could not reach the
failure it named. Replaced with a call counter on the object; M4 is red now.
Recorded as a survivor in DECISIONS/SAFETY-30-per-unit-sign.md §5, with the
first battery's own RED BASELINE - a shared mutation directory read mid-copy -
discarded rather than reported.
RECORDED, NOT FIXED: the site comparison is defeated by a square-wave load whose
half period is about one control cycle, which holds it opposed for ~1000 s. The
N=1 control arm is what makes that interpretable - it is identical at the roster
this add-on ships, so it is a property of comparing a lagged command against a
meter and has nothing to do with the second inverter. It wants an alternation
test rather than a number, and a card of its own.
SAFETY-24's lo-only asymmetry proof re-run unchanged: 1296 combinations, 0
negative floors, `hi` +inf on every claim, every claim on the SITE actuator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
|
||
|
glenn schrooyenandClaude Opus 5
|
d5b1ef19b3 |
CFG-03: give maintenance option defaults one source of truth
checks / unit (pull_request) Successful in 3m44s
Option defaults for the maintenance cycle were declared in three places that
could disagree, and MAINT-04 proved they already did. config.yaml shipped
`maintenance_charge_w: 2000`; main.py carried TWO `opts.get(..., 2500)`
fallbacks; MaintConfig's dataclass carried a third `2500.0`. So an options dict
with the key ABSENT was ACCEPTED by validate.py's rule 6 - which read 2000 -
and then built by main.py at 2500, against a 2000 W ceiling:
options dict with maintenance_charge_w ABSENT, max_w=2000
validate_startup_config -> [] (accepted, no problems)
MaintConfig().charge_w -> 2500.0
arbiter.resolve() -> 0.0 W, contradiction=True
That is the bug rule 6 exists to refuse, passing rule 6. It is unreachable on a
Supervisor install, which always supplies every key from config.yaml, and fully
reachable from sim/inject.py and from the test suite - which is where anyone
debugging maintenance actually works.
MAINT-04 shipped a TRIPWIRE and said so: test_validate.py pinned all three
literals and asserted the consequence through the real arbiter, so the
divergence could not silently widen or be half-closed. Its PASSING CONDITION
WAS THE BUG. Both it and validate.py's comment said to delete each other when
the divergence was closed. It is closed, so both are deleted.
THE DIVERGENCE WAS WIDER THAN ONE KEY. `maintenance_discharge_w` had the
identical three-way split (config.yaml 2000, MaintConfig 2500) and the card
named only the charge side. Closing one and leaving its twin would have
satisfied the numbers and left three sources of truth one line down, which is
the state that produced the bug. Both are closed, and so is the whole family:
all eight `maintenance_*` keys go through the same mechanism.
WHAT THE SINGLE SOURCE IS. config.yaml's `options:` block - it is what the
Supervisor reads and what the operator edits, so it wins by definition. There
is now exactly one in-code copy, `maintenance.MAINT_OPTION_DEFAULTS`, which is
TARIFF-02's `PRICE_OPTION_DEFAULTS` and SAFETY-24's `PEAK_OPTION_DEFAULTS`
pattern applied to the family that named the card - validate.py's rule 8 comment
already called that "CFG-03 read as a rule rather than as a complaint", so this
makes the rule hold where it was first broken rather than inventing a fourth
mechanism for one option family.
A MIRROR RATHER THAN A RUNTIME READ, and the reason is mechanical: the
Dockerfile copies `app/` and `run.sh` into the image and NOT config.yaml. The
running add-on cannot open the file it is configured by, so there is nothing to
read at 03:00 on a client's Pi, and a YAML-subset parser that could fail at
import is a new way for the add-on not to start. What makes the copy safe is
that test_maintenance.py pins every entry by READING the shipped file - drift is
a CI failure, not a comment.
EVERY OTHER SITE NOW DERIVES FROM IT OR HAS CEASED TO EXIST:
* MaintConfig's eight dataclass defaults are indexed out of the mirror. No
numeric literal survives in the class that held the third opinion.
* `MaintConfig.from_options()` is the ONLY place in the add-on that reads a
`maintenance_*` option. main.py's eight inline `opts.get(key, <literal>)`
calls are gone: moving the read next to the mirror means the fallback cannot
be anything but the mirror, because there is no literal in sight of it to
disagree with. It uses `key in opts` rather than truthiness, so a shipped
`false` or `0` is honoured instead of being quietly replaced by the default
- this card's own bug in a different costume.
* main.py's SECOND fallback site - the cheap-window charge floor at the
P_MAINT_SHAPE claim - reads `self.maint.cfg.charge_w` instead of re-opening
the options dict with its own literal. The Controller had already parsed
this once in __init__; asking the object that owns it cannot drift from it,
and the rate the shaping asks for can no longer be a different number from
the rate the state machine drains at on the same install.
* validate.py's rule 6 and its `maintenance_discharge_w` warning take their
fallbacks from the mirror.
WHERE THE RANGE LIVES, STATED AND CHECKED. config.yaml's `schema:` block carries
`maintenance_charge_w: int(500,5000)` - a fourth declaration site of a different
kind. It is a BOUND, not a default, so it is not this bug; but leaving the
question open is how a fifth site arrives. The decision recorded in config.yaml
and in maintenance.py is: bounds live in the schema block ONLY, because the
Supervisor is what enforces them and the Supervisor reads only that file, so a
Python copy of `int(500,5000)` would be a place to disagree and nothing else.
test_maintenance.py checks both halves of that: every mirrored key must have a
schema entry - without one the Supervisor drops the option and the add-on
silently runs on the mirror instead of the operator's value - and every shipped
default must lie INSIDE its own range, since a default the schema rejects is a
config that cannot be re-entered once it is edited. What the schema grammar
cannot express is a relation BETWEEN keys (`maintenance_charge_w` against
`max_w`), which is exactly why rule 6 is written in Python and why it does not
belong in the mirror either.
THE REPLACEMENT CHECKS, AND HOW THEY WERE VERIFIED. test_validate.py's new
section exercises the same four consumers the defect ran through, the same way:
the validator ACCEPTS a dict with every maintenance key missing; a REAL
Controller built from that same dict holds 2000 W (through main.py, not through
a regex over its source); `MaintConfig()` answers 2000 W with no dict at all;
and the real arbiter emits `-2000 W, limited by supervised(max_w)` instead of
`0 W, contradiction=True`. test_maintenance.py pins the mirror to config.yaml in
both directions - equal values, and full coverage of every `maintenance_*` key
the file ships, so a ninth key cannot arrive unmirrored - and asserts the
consumers move as a RELATION rather than against numbers, reading MaintConfig's
defaults back out of `dataclasses.fields()`.
Each new check was verified by BREAKING the thing it watches and observing red,
this project's rule:
1. config.yaml -> 1500, mirror left at 2000: test_maintenance 1 FAILED,
test_validate 4 FAILED.
2. config.yaml AND the mirror -> 1500: green again, and every consumer moved
together - MaintConfig() 1500 W, the Controller 1500 W, and the arbiter
`-1500 W, limited by maintenance(cheap-window charge)` where it had said
`-2000 W, limited by supervised(max_w)`. That is AC4: change the single
source and watch the consumers follow.
3. `charge_w: float = 2500.0` retyped into the dataclass: 2 FAILED in each of
the two files.
4. `opts.get("maintenance_charge_w", 2500)` restored in main.py: 1 FAILED,
naming the file and the literal it found.
5. `maintenance_hold_min` deleted from config.yaml's options block: 3 FAILED.
Full suite 17/17 in goodwe-addon-ci, twice, once under parallel load.
Stale prose that asserted the now-closed divergence as current is corrected in
app/watchdog.py, test_watchdog.py and test_peakguard.py - watchdog.py's example
still uses 1500 W, but because the point of the example is a charge strictly
inside the ceiling, not because 2500 is what ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
glenn schrooyenandClaude Opus 5
|
e40eb0768d |
SAFETY-02: the P_PEAK assertion was pinning the test's copy of the gate
Review finding 2, and it is the same duplication `disagrees()` just removed, so
it gets the same fix rather than a follow-up card.
`following_from()` in test_watchdog.py was a TRANSCRIPTION of the gate in
main.py, carrying "keep the two in step" as an instruction - which is exactly the
arrangement that does not keep two things in step. Measured: editing the shipped
gate alone to `winner in ("loop", "peak")` left every suite green while the
watchdog went blind for the duration of every peak event. The row whose whole job
is to force whoever lands peak shaving to come and change an expectation did not
do that.
The expression is now `watchdog.loop_is_following()`, called by main.py and by
the test. One definition, two callers, same reasoning as `disagrees()`: the copy
that drifts is always the one in the harness doing the grading.
⚠️ AND EXTRACTING IT WAS NOT SUFFICIENT - the drift proof still came out green
after the extraction, which is why it was worth running rather than assuming. The
realistic P_PEAK row has peak shaving asking for MORE than the loop, so the
magnitude term rejects it whatever `winner` says and masks the term under test. A
second row with peak asking for LESS makes `winner` the only deciding term. With
that row, widening the shipped gate fails two checks; without it, none.
Also enforcing the write ordering that was documented and not asserted: swapping
the two `store.set` calls passed everything. A store that fails the second write
now proves the property the ordering exists for - a crash mid-trip leaves a
reason with no latch, never a latch with no explanation. Swapping them now turns
that check red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa
|
||
|
glenn schrooyenandClaude Opus 5
|
cc4e88490b |
SAFETY-02 review: a latch that a restart clears is not a latch
BLOCKER. `auto` was persisted and `sign_wd.tripped` was not, so reconstructing the Controller against the same store - which is what a container restart does - brought it back ARMED, into the same fault, unattended: tripped=False, banner back to NOT READY, and it resumed commanding -1000 then -2000 W before re-tripping 180 s later. "Restart the add-on" is the first thing an operator looking at a FAILSAFE banner and a 0 W inverter will try, and under a supervisor that restarts a crashed add-on it is a loop of three-minute wrong-direction bursts. DOCS.md's "it does not clear itself" was false across a restart. `sign_tripped` and `sign_reason` are Store keys now, beside the maintenance state and for the reason store.py's own docstring gives. Written the moment the trip happens - reason first, flag second, so a crash between the two can never persist a latch with no explanation - restored in Controller.__init__, cleared only by the operator's start. The regression is covered: reverting the restore fails seven checks in test_watchdog.py. Review follow-ups, no behaviour change: - The arming gate is tested against the REAL arbiter. It is the decision the ticket turns on and it was asserted in three comment blocks and verified by nothing - every case passed `following` in as a literal. Twelve claim configurations now derive it from resolve(), including the redirect case `winner == "loop"` alone would have got wrong and the `lo` limit that raises magnitude without flipping sign. - The MQTT `status: "failsafe"` value is tested. The check that claimed to cover it restated the line above it and never touched the payload. - The worked example described a config that does not ship: at the defaults maintenance_charge_w 2500 against max_w 2000 is an arbiter contradiction and 0 W, not a -2500 W charge. Rewritten at 1500 W. The contradiction is a real pre-existing bug - a default cheap-window maintenance charge silently never happens - filed separately and deliberately not fixed here. - Forward hazard recorded at the gate: `winner == "loop"` disarms the watchdog under any higher-priority SET, and a peak-shaving strategy at P_PEAK IS grid-following in sign, so it would blind this for every peak event. - disagrees() is public so the replay harness stops carrying its own copy of the predicate it grades. - held_s takes the clock instead of reading one. - The two limits are now each stated in both places rather than one each. - The no-false-positive argument is tied to control.py's freeze permitting the command to cross zero; under the shrink-only encoding control.py records as rejected, this would false-trip on a load step. - Measured figures corrected: 93.4 % / 4212 of 17280, per-cluster 0.3/0.9/10.5/0.0, and the near-circularity of those figures is now stated rather than leaned on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa |
||
|
glenn schrooyenandClaude Opus 5
|
9aed43990b |
SAFETY-02: a loop can be fresh, tracking and inside its rating and still push the wrong way
The stale-input watchdog asks whether the meter is still talking. The saturation freeze asks whether the inverter can follow. The output clamp asks whether the number is too big. A loop can pass all three while charging the battery out of the grid, because the only thing wrong is the sign. On 2026-08-20 a commercial controller did exactly that four times in one day, for up to 77 minutes at a stretch, reaching a commanded -14 768 W against a ±5000 W inverter. app/watchdog.py compares the emitted setpoint against the authoritative P1 figure and, when they oppose each other with both over 300 W for 180 s continuously, files a failsafe SET claim at P_SAFETY, commands 0 W and latches there until an operator starts control again. Validated against that day rather than against an argument: sim/replay_sign.py in the infra repo drives this module over the committed captures of both 2026-08-20 and the healthy 2026-08-23 and reaches 18 trips on the fault day - every one inside a window NOTES.md §4.7 accounts for - and 0 on the healthy day. The four episodes do share one signature: 0.3 / 0.9 / 10.5 / 0.0 % sign agreement inside them against 92.9 % healthy. Three decisions the ticket could not make: - It is DISARMED whenever the grid-following loop is not what is driving. Sign agreement is a property of a loop that follows the meter, and a maintenance grid charge opposes the meter by design - 2500 W of charge into a house drawing 500 W is +3000 W on the meter against a -2500 W command, for hours. The gate tests the emitted number, not the winning claim's name, because a limit claim can let the loop "win" and still replace its value. - The trip LATCHES. Commanding 0 W removes the command side of the comparison, so the disagreement always clears the moment the trip works. - sign_disagree_w at or above max_w is refused at startup. The arbiter never lets a command exceed max_w, so such a threshold leaves the watchdog incapable of firing while the status page still reports it armed. What it does not catch, said out loud: an inverter that ACTS opposite to its command moves the meter further the same way, leaving command and meter in agreement. That is the saturation freeze's job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa |