6 Commits

Include renames

Author	SHA1	Message	Date
glenn schrooyenandClaude Opus 5	375a1e29d2	FCST-05: one byte stops stopping the add-on, and the write path gets the read path's standard ... HIGH - a non-UTF-8 byte in /data/price_history.json stopped the add-on from starting. `_load()` caught `(json.JSONDecodeError, OSError)`. `UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError`: neither of those, and `issubclass(UnicodeDecodeError, OSError)` is False. So it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which `Controller.__init__` does not guard - breaking `_load()`'s own stated contract, "a corrupt cache must not stop the controller". The handler now names the ValueError FAMILY (the file's content) beside OSError (its bytes), because a third subclass nobody has thought of is how this arrived the first time. The suite could not see it: its "unreadable file" fixture writes "{ this is not json", which is valid UTF-8 and fails one step later, in `json`. A fixture holding a genuinely undecodable byte is now beside it, with the difference between the two asserted as a precondition. MEDIUM - a malformed `start` under the day the feed is writing to killed the tariff plan permanently. `_remember()` merged with `{q["start"]: q ...}` and sorted on `q["start"]`, trusting a value read off /data to be a hashable, sortable string. An int, null, a dict or a list raised TypeError out of `carry()` -> `resolve_tariff()` -> `run_tariff_refresh`, which catches Exception - so the add-on stayed UP while every refresh raised forever, the plan retiring after TARIFF_MAX_AGE_S with arbitrage and the cheap window off for good, and no self-heal because `_remember()` raised before `_save()`. The read path was properly defensive throughout, which is why `yesterday()` survived and the existing fixture passed - it files its malformed rows under a day the feed is NOT writing to. Both paths now share one validator, `_point_of()`. A row that is not a point is dropped, reported, and written back out clean. ...and the status row said the store was healthy while the plan was dead: `words` read '2026-01-15 (1 h)'. `_load()` now counts the rows inside a kept record as well as the record itself, so the store stops reporting plain health from the moment it is opened; the note names what was FOUND at open and what was DROPPED by the refresh, and clears once the file is rewritten. ...and the merge key is the validated INSTANT rather than its spelling, which is what rule 4 already said. The lexicographic sort beside the string key also put the second 02:00 of the 25-hour October day (+01:00) before the first (+02:00); stored order only, since `_parse_points()` re-sorts. LOW - the status row counted POINTS and printed them under the HOUR's name: '2026-01-13 (96 h)' for a 24-hour day on the quarter-hourly feed this add-on already supports. It is now '96 points over 24 h' - two facts, each under its own unit. LOW - two stated reasons were false. `_audit`'s "WHY IT IS STRICTER" claimed the rework refused the same published point emitted twice and a carried point sharing a published hour, "which the old count never looked at". A differential over the whole domain the audit can express finds ZERO divergence in both directions: the old `seen` count already refused the first (one point twice is one hour twice) and the old IDENTITY rule already refused the second. The reworked audit is EQUAL to FCST-03's, not stronger; the only behavioural change is the intended relaxation, and it stays. The differential is committed as a check rather than left as prose. ...and the audit's real limit is named: it does not check COMPLETENESS. Fed the decimation defect FCST-04 was raised for it ACCEPTS, and it accepts an empty output. So `_audit` is not what stops that recurring - the count and identity checks are, and the text now points at them. Not made complete: `published` is the whole read while the fill emits only [since, until), so the rule as stated is wrong, and building it takes the 1/12/48 h horizon checks red on a healthy feed. Measured, not assumed. 12 mutations, purged tree, every one RED, no survivors. All 29 test files green. Nothing deployed; no option added; config.yaml untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU	2026-09-10 01:04:30 +02:00
glenn schrooyenandClaude Opus 5	b59f5ce31b	FCST-04: the carry-forward stops decimating a quarter-hourly feed, and a broken cache stops stopping the add-on ... Rework of the FCST-04 branch against review. Every finding below was proven by a runnable script before it was touched and re-run after; every check added or repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md, TEST-15). 29/29 test files green. ⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD WENT WITH IT. `fill_from_yesterday()` keyed the published set as `{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the other three dropped. `eneco_eddy` is the Belgian imbalance product and settles per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so this is a shape the product supports today. Measured on a 96-point quarter-hourly feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh, and the cheapest "hour" became a `:45` point whose `at()` window runs into the next one. Not the windowing effect this card is about; a feed losing 84 % of its own prints, with no flag anywhere reporting it. The fill now decides only WHICH WHOLE HOURS NEED CARRYING: every published point inside the window passes through BY IDENTITY, and a carried point is added only for a whole hour with no published coverage at all. Same feed, same fixture: 60 of 60 firm points kept, spread 0.134 over the hours still ahead. ⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it at all. The old post-condition refused any repeated `_utc_hour`, a count that cannot express four points in one hour. Two rules replace it and SUBSUME it: a published point must BE one of that hour's published points by identity and may be emitted only once; a carried point may stand only in an hour absent from the published set, must be flagged True, and may not repeat. So the duplicated carried hour it existed to catch still raises, the same published point emitted twice now raises, and a carried point sharing an hour with a real print now raises too - which the count never looked at. Three mutations, three reds. ⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING. `_load()` validated `version` and that `days` is a dict, never that each VALUE is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True, last_error=None, and then `rec.get(...)` raised AttributeError out of `words`, `yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are both unguarded in `Controller.__init__`, so the controller did not come up - and it never self-healed, because `_remember()` raised on the same record before it could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A BROKEN CONTROLLER" in the one direction that matters. The asymmetry that produced it is visible in the file: `_parse_points()` is carefully defensive about every field of a POINT and `_load()` was not defensive about the record holding them. Unusable records are now dropped and REPORTED (`words` says how many and why), and `words` is total by construction as a second line. M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD. `brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned 2026-03-28's prices as "yesterday", or `no_data` naming a date that is not yesterday. The arithmetic is now on the local DATE, which has no offset to lose, and both transitions have a check - the module's headline claim had none. M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited on `not ok` while `_remember()` populated `self._days` anyway: `words` said "unavailable" and `yesterday()` answered A SERIES that is gone on the next restart. The module docstring rejects exactly that ("a yesterday that silently forgets") as its reason for not calling makedirs, so the code now matches the docstring rather than contradicting it. M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return` fired and the body it is named after never ran: pruning by arithmetic against the clock - which empties the store on exactly the morning the fill is for - passed the whole 29-file suite. The fixture is now written to a file and reopened (`_load()` does not prune, which is how a store legitimately holds more than the cap) with KEEP_DAYS + 2 days, and the mutant is red. M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades `ast.walk` BFS DEPTH: `self._remember(...)` is a statement and `self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so `_remember` came out first whichever statement was written first - shipped and reversed both scored True. It is now line numbers AND, the half that matters, the behaviour: a feed carrying yesterday alongside today into an EMPTY store must fill the whole horizon; reversed, the lookup finds nothing and the plan is only the published hours. Reversing the two statements is red on both. L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no slack: a feed publishing to D+2, or ONE stray point in the wrong year - which sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now PINS the two days a lookup can actually ask for on top of the cap. Pinning is additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted by clock arithmetic) is preserved by construction; `stamp` stops being a dead parameter; the store is bounded at KEEP_DAYS + 2. L8 - the horizon end was not floored while `since` was, so a 48 h request made at any instant off the hour returned 49 points. §1's count was green only because its NOW was exactly :00:00, a shape production never has; the fixture now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes a day a persistent write failure leaked 48 inodes a day onto the volume that was already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 - the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME, so a comment does not fail it and a call in `tools/` cannot pass unseen. CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL: * L7. DOCS said "*Tariff* on the status page reads `provisional` through the morning". FALSE - `tariff_words` is assigned only from the HA price-entity branch and from `static_status()`, both on the 30 s tick, and the plan's own sentence reaches `_LOG.info`. The doc now says what an operator can see (the Price history row, the Arbitrage row's own refusal sentence, the refresh log line) and a check pins the PAIRING - wiring the plan into the row is fine, correcting the doc is fine, doing one without the other is red. * L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves the local hours it never held out of the series (§11.4 for those hours, and `at()` raises rather than answering). Correct behaviour, overclaimed wording, and untested - all three fixed. * L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION CALLER on the provider plan (`self.cheap` comes from `static_status()`, which builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal alone, which does reach control; the evidence is now stated at the reach it has. * L13. §1's "passed through by identity" check compared `eur_per_kwh` between two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The value comparison is kept and now says what it grades; identity is graded on a read the file constructs and still holds. * L17. `PriceHistory.words` and the new status row had no check at all - and `words` is one of the two things H2 crashed. Both now have one. * L18. main.py's inline note carried the "widens what the optimiser can SEE, never what it may act on" guarantee without the windowing caveat that the CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one place a maintainer reads it unqualified. EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps 60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script still scores True/True because it re-implements the OLD algorithm - the reversal is now driven as a mutation instead, and is red. p3's script always showed the shipped code correct; what was wrong was the fixture, and its mutant is now red. Twenty mutations driven, no survivors, tree fingerprinted and `__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored `until` (2), unvalidated day records (7 checks), instant-subtraction yesterday, in-memory fill on a volume-less store, clock-arithmetic prune, cap without the pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2), a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a copy instead of identity (2), the DOCS claim in both directions, and the caveat removed from main.py. Baseline and the unmutated tree both green. No option added, config.yaml untouched, and the CHANGELOG entry still picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has been pulled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU	2026-09-09 23:54:23 +02:00
glenn schrooyenandClaude Opus 5	7b4877d56d	TEST-15: purge the bytecode cache before a mutation run, not after ... FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran, and reported the mutation as a survivor. It was not one. CPython validates a cached `.pyc` on (source size, source mtime truncated to whole seconds), and that edit moved neither number - same length, same second - so the interpreter loaded the bytecode it already had. Disassembly settled it at the time: the source read `here`, the running code loaded `self`. A mutation run had graded code that was not there, with no error and exit status 0. This does not make the results already on the record wrong. It makes them UNVERIFIED, and that distinction is the whole card - so nothing here re-runs a campaign and nothing here quietly assumes the old numbers hold. REPRODUCED, THEN CLOSED. test_bytecode.py builds a real package whose two return values are the same length, imports it in a real subprocess, rewrites it inside the same second (retrying until both writes land in one whole second - milliseconds, no sleep) and reads back which value ran. Part 1 shows the interpreter running SELF off a file that says HERE. AND THE TWO OBVIOUS MITIGATIONS DO NOT WORK. Measured, not read off documentation: `python3 -B` and PYTHONDONTWRITEBYTECODE=1 are write-side only. Neither stops an EXISTING stale .pyc being read, and a mutation run always has one, because the previous run wrote it. Part 2 grades both still running the old code. The advice in test_control.py's coverage audit led with `-B`; it is corrected here - the purge is the half that works, and `-B` earns its place by keeping THIS run from leaving a cache for the NEXT edit to inherit. NO SLEEP ANYWHERE. One-second mtime resolution is the mechanism, not the enemy. Waiting it out would slow every run down and would still only cover the edit somebody remembered to slow down. THE HARNESS. tools/mutation_run.sh purges, prints a sha256/byte fingerprint of every app/*.py it is about to grade - so a mutation report can say WHICH source produced it, which is the reviewer's addition to this card - runs each suite under `python3 -B`, refuses a tree with no test_*.py (CI-01's rule, for CI-01's reason), and fails if a cache appears during the run. It takes the tree as an argument so the infra repo's replay harnesses can call it without being moved. WHERE IT IS SAID (AC 2). MUTATION.md carries the mechanism, the measurement, the reporting rules and the survey, and every one of the 16 suites now points at it from its docstring. That list is DERIVED FROM THE TREE by test_bytecode.py part 5, so a new suite that joins this directory without the pointer is red - the same failure shape CI-01's file glob exists to prevent. CI WAS ALREADY IMMUNE, AND IS DESCRIBED THAT WAY (AC 4). Verified rather than reasoned: with the checkout bind mounted `-v "$WS:/w:ro"` the interpreter writes no __pycache__ into the tree at all, and it fails SILENTLY - no error, exit status 0 - while the same probe on a writable mount produces one. Each job also rm -rf's its workspace and clones afresh, so nothing crosses between jobs on the shared runner. PYTHONDONTWRITEBYTECODE=1 is added as belt and braces only: the `:ro` is one character wide and the next person debugging a write will be tempted by it. Part 6 grades both ON THE DOCKER COMMAND LINE rather than on the file - the first draft searched the whole workflow and stayed green after the flag was deleted, because the comment explaining it still named it. THE SHIPPED ADD-ON IS NOT AFFECTED and the question is closed: goodwe_controller/Dockerfile has set PYTHONDONTWRITEBYTECODE=1 since it was written. There is no deploy-side finding. THE SURVEY (AC 3), in MUTATION.md, scoped BY THE PROPERTY. The defect is one-directional - a mutation that never took leaves the suite green, so stale bytecode manufactures SURVIVORS and cannot manufacture a red - which puts every "all red, no survivors" claim in the window out of scope, along with mutations executed in-process (SAFETY-07's B1/B2/B3, test_peakguard parts 7 and 10, where no source file is edited) and everything run after the purge discipline landed on 2026-08-31 (SAFETY-24, SAFETY-26, OPT-04, FCST-03's re-run). What is left is the recorded SURVIVORS from source-edit runs before that date. Two were re-run here under the harness and both still stand: TARIFF-01's record_block() ignoring its policy argument, and SAFETY-03's abs() on the clamp limit - 15 suites green each, on the tree as it stood. The rest are labelled still-unverified rather than assumed or re-run; each is one harness run to close, and the moment to spend it is when somebody next touches that module. OUT OF SCOPE AND RECORDED: the infra repo's sim/ replay harnesses import the add-on and write bytecode into both trees (goodwe/sim/__pycache__ exists today), so SAFETY-02's headline replay carries the same exposure. mutation_run.sh takes a tree argument for exactly that, but the change belongs in that repo and some of those harnesses want the rig. RED AND GREEN. Every new check was broken and watched fail: the purge commented out of the harness turns part 4 red (and its non-vacuity probe already proves the fixture would mis-grade without it); one docstring pointer removed turns part 5 red naming the file; `:ro` and the `-e` flag removed from the workflow turn part 6 red. Full suite green, 16 files, in the CI image on a read-only mount - the shape the runner uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 02:07:09 +02:00
glenn schrooyenandClaude Opus 5	bdd7f18a54	OPT-04: charge cheap, discharge expensive - and only when it pays ... Four cards built the price stack and nothing traded on it. app/arbitrage.py is §10.1's priority 3, and it consumes all four rather than re-deciding anything: whether a trade is permitted at all, on which two hours, against which widened threshold is TariffPlan.arbitrage()'s answer, and that answer cannot be obtained without first saying whether the peak guard is holding. §9.1's "peak shaving outranks arbitrage" is therefore honoured structurally - there is no path to a spread that skips the question - and again in the arbiter, where P_PEAK (50) clamps a claim filed at P_ARBITRAGE (30). A guard that never ran is not a guard that is clear. On meter_source off the peak guard is never evaluated; that third state is carried as None and refuses, because charging at the rail creates the billed quantity while nothing measures the quarter. validate.py refuses that configuration too, along with arbitrage armed with no price source and a SoC floor at or above the ceiling. §17: never export into a negative price. The guard is on the hour being sold into, not on the spread - a day priced -0.30 .. -0.05 clears any threshold and its dearest hour is still negative. Buying in a negative hour stays allowed. resolve_tariff() may open an HTTPS connection and the tick feeds the ESP32 heartbeat, so the fetch runs in its own task at a 30 min cadence and the tick reads an attribute and an age bound. test_arbitrage.py section 8 asserts that structurally on the shipped AST: a fetch put back on cycle() is red. AC4 is NOT met and is not restated - this repository holds no price history. Section 7 replays a synthetic year and asserts the property that does not need real prices: every executed round trip cleared its own widened requirement, and zero discharges landed in the 522 negative-priced hours. At the shipped 0.10 EUR/kWh threshold a realistic Belgian day/night site does not trade at all. Reported, not tuned: that number and FCST-03's widening factors are one product decision and it is the owner's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC	2026-08-31 01:36:09 +02:00
glenn schrooyenandClaude Opus 5	71cd7849e4	Merge release/1.0 (SAFETY-07) into FCST-03, and close two degenerate stimuli ... CHANGELOG.md was the entire collision - SAFETY-07 lands in peakguard.py, main.py and the arbiter, none of which this branch touches. Both entries kept. X1. `_firm_or_die` guarding only `cheapest` survived, because the check passed the SAME provisional point as both cheapest and dearest: a guard on either role alone raises, so the stimulus could not tell "the buy hour is guarded" from "both are". Now one call per role with a firm point in the other seat, plus the positive arm so the two refusals are discriminating. X6. `firm_points()` returning the carried hours too survived, because its two consumers - the §11.4 refusal and cheap_now - guard provenance independently and both keep answering correctly. The accessor is now graded directly: exactly the published points, by identity and in order. Both are the degeneracy rule this file states at the top, in a new dress - a stimulus that cannot distinguish the two things it grades. ⚠️ AND THE REASON THE M9 MUTATION "SURVIVED" IN MY EARLIER REPORT: a stale __pycache__. Reverting `self.provisional` to `here.provisional` changes neither the file size nor, at one-second resolution, the mtime, so CPython kept the MUTATED bytecode and a green run afterwards proved nothing about that line. Every mutation here now clears __pycache__ first; re-run that way, M9 is caught. The committed source was always correct - the checks were not being run against it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC	2026-08-31 00:15:32 +02:00
glenn schrooyenandClaude Opus 5	dd40c56ab4	FCST-03: yesterday's prices, flagged as yesterday's, and unable to move anything ... §8.3 fills the hours a day-ahead feed has not published yet with yesterday's profile and flags them provisional, and forbids any irreversible action on such a price. The whole risk of this ticket is one line - a carried point and a published point are three fields and a bool, so `provisional=y.provisional` copies yesterday's False onto today's guess and produces stale data presented as current. That is SAFETY-18's frozen-but-fresh meter one layer up. So the flag is never passed through. Carried points come from a constructor with no provisional parameter; published points pass through BY IDENTITY, so no copy exists that could lose one; an audit re-derives the expected flag for every emitted point and raises rather than repairing; and the check walks the module's own AST so the pass-through idiom cannot come back looking ordinary. A published point may itself be provisional (eneco_eddy's imbalance print is), so the rule for a published hour is "whatever the feed said", never False. The fill keys on the Brussels wall clock rather than t-24h, so the 23- and 25-hour days are right; a late publication rebuilds the series, moving the number and the flag together. AC2 and AC3 landed on TARIFF-03's existing verdict surface rather than beside it. Refusal 3 became "fewer than two FIRM hours" - refusing on "any point is provisional" would have switched arbitrage off every morning on every dynamic site - and refusal 5 is new: the spread must clear a threshold widened by the widest active uncertainty source (max, never a product). The permitting variant now refuses to be constructed around a provisional point at all, and the cheap-window flag became per-hour for the same reason. The irreversible action is a value carrying the hours it commits, derived from maintenance.py's own DRAIN_TIMEOUT_MIN; the gate blocks a maintenance drain whose window reaches into a carried hour, and the phase machine stays idle. It is the price-justified entry point only - a gate that blocked on no_data would stop the monthly cycle on every default install. Zero new config options; TEST-14's signature scan ported to cover both Gate's and Arbitrage's constructors, closing _Refused.reason's open hole. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC	2026-08-30 23:54:07 +02:00