diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 973b73f..6c84caa 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -17,6 +17,91 @@ if it has, it needs 0.10.0. Whoever deploys decides. | _this branch_ | FCST-06 - "cannot persist" stops meaning "has no /data" and starts meaning what it says | | _this branch_ | TARIFF-08 - the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser | | _this branch_ | TARIFF-09 - the day-ahead feed is asked for the horizon it was given, so tomorrow arrives firm instead of carried | +| _this branch_ | TARIFF-14 - the price a trade is decided against is the price of the interval being traded | + +### TARIFF-14 - the trade was decided against the wrong price + +**On the only price feed that works, `_Series.at()` returned the wrong point, +and the error grew with depth into the series.** It asked +`s <= u < s + 1 h` of every point **in list order** and returned the **first** +match. On the hourly shape every fixture in this repository carried before +TARIFF-08 that is right by accident; Belgium moved to a **15-minute MTU on +2025-10-01** and the live day-ahead feed returns **96 points at 900 s**, where +it is wrong twice over. Measured against the shipped code at `5059e28`: + +``` +series: 02:00=0.10 02:15=0.11 02:30=0.12 02:45=0.13 03:00=0.14 ... + +ask + 0min -> 02:00 0.10 +ask +15min -> 02:00 0.10 <-- WRONG (wanted 02:15) +ask +45min -> 02:00 0.10 <-- WRONG +ask +60min -> 02:15 0.11 <-- WRONG (wanted 03:00) +ask +75min -> 02:30 0.12 <-- WRONG (wanted 03:15) +``` + +Inside the first hour the `:00` point wins because its hour has not run out. +**From the second hour on the answer is not even the top of the hour** - the +first point still inside its own hour wins - so the error compounds. + +**What it cost is not "a slightly wrong price".** `arbitrage._act()` tests +`here.start == v.cheapest.start` / `== v.dearest.start` to decide whether this +interval is the buy one or the sell one. That identity almost never held, so +**the strategy idled through exactly the interval it exists to trade**, +reporting *"this hour is neither the buy hour nor the sell hour"* - and read +§8.3's `here.provisional` and §17's `price < 0` off the same wrong point. +`static_tariff.py` emits hourly points, so **static sites were never affected**; +this was reachable only down the dynamic path TARIFF-08/09 made work. + +**The rule now, in one sentence: a point covers from its own start until the +next point's start, and never more than one hour.** + +| | before | after | +|---|---|---| +| 900 s | `at(02:15)` -> the 02:00 point | -> the 02:15 point | +| 3600 s | `at(02:30)` -> the 02:00 point | **unchanged** | +| **mixed** `[900, 3600]` | each point one hour wide | **each point takes its own gap** | + +The mixed case is not an edge case: FCST-04's carry-forward emits **hourly +carried points beside quarter-hourly firm ones**, and the soak measured spacing +`[900, 3600]` in **70 % of samples**. A window derived from one global spacing +would be wrong in the normal steady state. + +**The one-hour cap is load-bearing.** FCST-04 leaves an hour *out* of the +series when nothing published it and yesterday has no value either, and +`at()` raising `Absence` for that hour is what sends the caller to the static +profile (§11.4). Uncapped, "until the next point's start" would stretch the +point before a three-hour hole across the whole hole and answer every instant +in it with a stale price - filling holes, introduced by the fix for a different +defect. + +**The last point's window is a decision and it is written down.** +`DECISIONS/TARIFF-14-the-window-a-point-covers.md` §2 argues it: the last point +is given **the same span as the point before it** (one hour when it is the only +point). A flat one-hour tail would answer **00:30 tomorrow with today's 23:45 +price**, which is reaching for a neighbour across a day boundary. The cost of +the choice - a mixed series ending at the firm/carried seam loses its last 45 +minutes to `Absence` - is stated there rather than buried, and the direction is +the safe one. + +**`at()` now reads a point's neighbours, so it would depend on the series being +sorted - and it establishes that itself** rather than trusting the three +producers that happen to guarantee it today. The lookup index is sorted by UTC +start once, at construction; a series handed its 96 points in reverse order +answers identically, which is a check. `covers()` shared the defect exactly +(it has no rule of its own - it asks `at()`) and is fixed by the same change; +`window()` did not (it grades a point's **start** against a range and asks +nothing about duration), and its docstring now says so. + +**Nothing about §9.3 or a point's `start` changed.** `PricePoint` still has +three fields; the span is derived from the neighbours, not stored. + +Graded by `test_price.py` section 13 - 900 s, 3600 s, **mixed**, holes, the +tail, caller order, `covers()`/`window()` - and end to end by +`test_arbitrage.py` section 10, which drives `decide()` over a 96-point +quarter-hourly day whose cheapest and dearest quarters are deliberately **not** +on a whole hour. Eight mutations, all red, including a **zero-byte** one on the +tail rule; the table with byte deltas and the checks that were NOT reached is +in the decision note §6. ### TARIFF-09 - the feed is asked for the range the caller wanted diff --git a/goodwe_controller/DECISIONS/TARIFF-14-the-window-a-point-covers.md b/goodwe_controller/DECISIONS/TARIFF-14-the-window-a-point-covers.md new file mode 100644 index 0000000..a70492e --- /dev/null +++ b/goodwe_controller/DECISIONS/TARIFF-14-the-window-a-point-covers.md @@ -0,0 +1,234 @@ +# TARIFF-14 — how long one price point covers, and what the last one does + +One defect and one genuinely open decision. The defect had no defensible +alternative once it was measured; the decision — the **last** point's window — +did, and the alternative is argued below rather than dismissed. + +--- + +## 0. The measurement this card exists for + +`_Series.at()` asked `s <= u < s + 1 h` of every point **in list order** and +returned the **first** match. Run against the shipped code at `5059e28` on a +quarter-hourly series: + +``` +series: 02:00=0.10 02:15=0.11 02:30=0.12 02:45=0.13 03:00=0.14 ... + +ask + 0min -> 02:00 0.10 +ask +15min -> 02:00 0.10 <-- WRONG (wanted 02:15) +ask +30min -> 02:00 0.10 <-- WRONG +ask +45min -> 02:00 0.10 <-- WRONG +ask +60min -> 02:15 0.11 <-- WRONG (wanted 03:00) +ask +75min -> 02:30 0.12 <-- WRONG (wanted 03:15) +``` + +**Two failure modes, not one.** Inside the first hour the `:00` point wins +because its hour has not run out. From the second hour on the answer is not +even the top of the hour — the first point still inside its own hour wins — so +**the error compounds with depth into the series**. + +Same code, same run, after the fix: + +``` +ask + 0min -> 02:00 0.10 ask +60min -> 03:00 0.14 +ask +15min -> 02:15 0.11 ask +75min -> 03:15 0.15 +ask +30min -> 02:30 0.12 ask +90min -> 03:30 0.16 +ask +45min -> 02:45 0.13 ask+120min -> 04:00 0.18 +``` + +An hourly series is **unchanged at every instant**, before and after — which is +the whole reason nothing caught this. Every price fixture in this repository +before TARIFF-08 was the invented hourly row shape, `static_tariff.py` emits +hourly points, and the DST checks count points on the 23- and 25-hour days +without ever asking `at()` for a quarter-hour. **The suite had been exercising +the one resolution on which the broken rule and the right one agree.** + +Belgium moved to a 15-minute MTU on 2025-10-01 and the live day-ahead feed +returns **96 points at 900 s**, so this is reachable only down the dynamic path +TARIFF-08/09 just made work. Static sites are immune. + +## 0.1 What it cost, which is not "a slightly wrong price" + +`arbitrage._act()` calls `_point_at(plan, now)` → `series.at(now)` and then +tests `here.start == v.cheapest.start` / `== v.dearest.start` to decide whether +this interval is the buy one or the sell one. That is an **identity test +between the matched point and the point the verdict was built from**, and on a +quarter-hourly series it almost never matched — so the strategy idled through +exactly the interval it exists to trade, reporting *"this hour is neither the +buy hour nor the sell hour"*. It also read `here.provisional` (§8.3) and §17's +`price < 0` off that same wrong point. + +--- + +## 1. The rule + +> **A point covers from its own start until the next point's start, and never +> more than one hour.** + +* **900 s** → each point covers its own quarter. +* **3600 s** → each point covers its own hour. Identical to what shipped. +* **MIXED** → **each point takes its own gap.** FCST-04's carry-forward emits + hourly carried points beside quarter-hourly firm ones, and the soak measured + spacing `[900, 3600]` in **70 % of samples** — so a window derived from one + global spacing would be wrong in the **normal steady state**, not in an edge + case. + +### 1.1 Why the one-hour cap is not decoration + +FCST-04 **leaves holes on purpose**: an hour with no published price and no +yesterday value is left *out* of the series, and `price_forecast.py`'s module +note names `at()` raising `Absence` for it as the behaviour that sends the +caller to the static profile (§11.4 for that hour). + +Uncapped, "until the next point's start" stretches the point **before** a +three-hour hole across the whole hole and answers every instant in it with a +stale price. Filling holes is the defect this module exists to refuse, and it +would have been introduced **by the fix for a different one**. Graded in +`test_price.py` 13d; the mutation that removes the cap is red there. + +--- + +## 2. The decision: what the **last** point covers + +There is no next start, and §9.3 says nothing about how wide the final interval +is — a `PricePoint` has **three fields** and must not learn a fourth, because +that is the wire format every store, replay, fixture and captured body in this +repository is written against. So the tail is a **choice**, and both candidates +are defensible. + +**Chosen: the last point is given the same span as the point before it** (one +hour when it is the only point). + +| | flat one hour | the span in front of it (**chosen**) | +|---|---|---| +| 96-point 900 s day, `at(23:50)` | answers 23:45 | answers 23:45 | +| the same day, `at(00:30)` **tomorrow** | **answers with today's 23:45 price** | `Absence` | +| 24-point hourly day, `at(23:50)` | answers 23:00 | answers 23:00 | +| mixed series ending at the firm/carried seam | answers for an hour | `Absence` after 15 min | + +**The second row is why.** A flat one-hour tail reaches across a day boundary +and prices tomorrow morning with yesterday's last print — which is +"reaching for a neighbouring one", the single thing the `Absence` message has +always refused to do. The chosen rule never extends a point past a span the +series itself demonstrates. + +### 2.1 The cost of the choice, stated rather than buried + +On a **mixed** series whose last point is a carried **hourly** one sitting +behind quarter-hourly firm ones, the tail is **900 s**, and the last 45 minutes +of the horizon raise `Absence` for an hour something did carry a value for. + +That is accepted, not tolerated, and the direction is the safe one: + +* the caller falls back to the static profile (§11.4) instead of trading on a + span the series never demonstrated; +* the point is **carried**, so §8.3 forbids any irreversible action resting on + it regardless; +* it is at the far end of a 48 h horizon, where nothing is decided at 30 s. + +Under-covering costs a missed decision. Over-covering prices a real interval +with a number that does not describe it. Those are not symmetric. + +### 2.2 The degenerate case + +A series of **one** point has no gap in front of it to copy and gets **one +hour** — the shipped behaviour, kept deliberately. `test_price.py` 13e states +it as its own check so it cannot decay into an accident. + +--- + +## 3. Does `at()` now depend on sortedness? Yes — and it establishes it itself + +The old rule read one point at a time and did not care about order. The new one +reads a point's **neighbours**, so it *would* depend on the series being +ascending. + +It does not, because **the lookup index is sorted by UTC start at construction** +(`_cover_table()`, called once from `_Series.__init__`). No caller's list order +can reach `at()`. + +That matters because the three things that "guarantee" order today each +guarantee it somewhere else: + +| | what it guarantees | where | +|---|---|---| +| `_validate_raw()` | strictly increasing starts | the `load_prices()` path only | +| `price_history._parse_points()` | re-sorts by instant | the store read path only | +| FCST-04's fill | emits ascending | that one producer only | +| `PriceRead.series()` | **nothing** | every other caller, including tests | + +Three producers agreeing today is not an invariant; it is a coincidence with +good references. `test_price.py` 13f drives a series handed its 96 points in +**reverse** order and asserts `at()` answers identically at every one of them — +red the moment the sort is removed. + +**Duplicated starts** (refused by `_validate_raw()`, constructible by hand) give +the earlier list entry a zero-width window and the later one the real span. One +of the pair becomes unreachable through `at()`; **nothing raises**, at +construction or on the 30 s control path. Stated as a check rather than left to +be discovered. + +--- + +## 4. `covers()` shared it. `window()` did not + +* **`covers()`** has no window rule of its own — it calls `at()` — so it shared + the defect exactly and the same change fixes it. It is kept as a delegation + for that reason: a second copy of the rule is a second place for it to be + wrong. +* **`window(since, until)`** grades a point's **start** against a half-open + range and never asks how long a point lasts, so it carries no resolution + assumption at all. It is **not** a coverage query — a point that starts before + `since` and runs into it is absent from its result by design. That is now + said in its docstring and pinned by a check, because the next reader will ask. + It has **no callers** in `app/`. + +--- + +## 5. What this card does NOT fix, with its shape + +`price_forecast._walk()` — §8.3's irreversible-action gate — iterates +`committed_hours()`, which steps in **whole hours**, and asks `series.at(u)` at +each. On a quarter-hourly series that now reliably returns the hour's `:00` +point (it previously returned something earlier and arbitrary), so the gate +reads **one point per hour** and never looks at the `:15`/`:30`/`:45` flags. + +A partially-provisional hour would therefore be gated on its first quarter +alone. It is not reachable from anything shipped — FCST-04 carries **whole** +hours and `generic_dayahead` publishes whole days, so the four quarters of an +hour are firm or carried together — but it is an assumption about the +*producers*, not a property of the *gate*, and it is written down here rather +than left for the next sub-hourly feed to find. Fixing it means teaching +`committed_hours()` about resolution, which is a data-model question this card +deliberately does not open. + +--- + +## 6. Mutations + +Eight, each purged before the run (TEST-15), each reverted with the tree +fingerprint checked afterwards (MUTATION.md habit 18), backup at a `mktemp` +path per invocation. Clean `app/price.py` = `30d1a56f0117e571`, 83742 B. + +| | mutation | Δ | red in | +|---|---|---|---| +| M1 | `at()` back to the shipped fixed one-hour scan over `self.points` | +68 B | **24 named checks**: `test_price.py` 13b ×4, 13c ×3, 13d, 13e ×3, 13f, 13g; `test_arbitrage.py` 10b ×6, 10c ×3, 10d ×2 | +| M2 | drop the one-hour cap | −68 B | `test_price.py` 13d | +| M3 | flat one-hour tail instead of the span in front | **+0 B** | `test_price.py` 13e ×3, 13g — 4 checks | +| M4 | trust the caller's list order — drop the sort | −61 B | `test_price.py` 13f | +| M5 | `covers()` grows its own hour-wide rule | +95 B | `test_price.py` 13e, 13g | +| M6 | `window()` becomes a coverage query | −37 B | `test_price.py` 13g | +| M7 | the window's end becomes inclusive | +1 B | `test_price.py` 13b ×4, 13c ×4, 13d, 13e ×4, 13f ×2, 13g — 16 checks | +| M8 | the window's start becomes exclusive | −1 B | `test_price.py` 13b ×4, 13c ×4, 13d, 13f ×2, 13g, 13h, plus the pre-existing section 6 check — 14 checks | + +**M3 is the zero-byte one and it is the tail decision itself** — the case +MUTATION.md names as the risky size, caught by the check written for it. + +**What the arbitrage file does NOT see, with sizes.** M2, M4, M5, M6 and M8 +leave `test_arbitrage.py` green, and that is honest rather than a gap to round +off: they are `at()`-level properties (holes, caller order, `covers()`, +`window()`, the start boundary) that no arbitrage fixture reaches, because the +probes are seven minutes into a slot and the plans have no holes. The one that +matters end to end — M1, the shipped rule — is red in **both** files, on the +named check `THE PROPERTY THE DEFECT BROKE`. diff --git a/goodwe_controller/app/price.py b/goodwe_controller/app/price.py index aeb63fd..6d0ef4a 100644 --- a/goodwe_controller/app/price.py +++ b/goodwe_controller/app/price.py @@ -541,6 +541,121 @@ class PriceRead(_Outcome): return _NoData(reason) +# --------------------------------------------------------------------------- # +# How long ONE point covers - TARIFF-14 +# --------------------------------------------------------------------------- # + +# ⚠️ A POINT DOES NOT CARRY ITS OWN DURATION AND MUST NOT LEARN TO. §9.3 lists +# THREE fields and `PricePoint` has three; a fourth would change the wire +# format every store, replay, fixture and captured body in this repository is +# written against. The span is DERIVED from the neighbours instead - here, +# once, at construction. +MAX_POINT_SPAN = timedelta(hours=1) + + +def _cover_table(points) -> tuple: + """`(start_utc, end_utc, point)` per point, ascending. The `at()` index. + + THE DEFECT THIS PREVENTS (TARIFF-14, HIGH). `at()` asked + `s <= u < s + 1 h` of every point IN LIST ORDER and returned the FIRST + match. On an hourly series - the only resolution any fixture carried + before TARIFF-08 - that is right by accident. Belgium moved to a + 15-minute MTU on 2025-10-01 and the live day-ahead feed returns 96 points + at 900 s, where it is wrong twice over. Measured on shipped `5059e28`: + + series: 02:00=0.10 02:15=0.11 02:30=0.12 02:45=0.13 03:00=0.14 + + at(02:15) -> 02:00 the :00 point's HOUR has not run out yet + at(03:15) -> 02:30 and from the second hour on it is not even the + top of the hour: the first point still inside + its own hour wins, so the error COMPOUNDS with + depth into the series + + `arbitrage._act()` compares `here.start` against `v.cheapest.start` and + `v.dearest.start` to decide whether this interval is the buy or the sell + one. That identity test almost never matched the extremum, so the + strategy idled through exactly the interval it exists to trade - and read + `here.provisional` and §17's `price < 0` guard off the wrong point while + doing it. `static_tariff.py` emits hourly points, so static sites never + saw it; this is reachable only down the dynamic path TARIFF-08/09 made + work. + + THE RULE, in one sentence: + + A POINT COVERS FROM ITS OWN START UNTIL THE NEXT POINT'S START, AND + NEVER MORE THAN ONE HOUR. + + * 900 s -> each point covers its own quarter, so `at(02:15)` is the + 02:15 point; + * 3600 s -> each point covers its own hour, unchanged from shipped; + * MIXED -> EACH POINT TAKES ITS OWN GAP. FCST-04's carry-forward emits + HOURLY carried points beside QUARTER-HOURLY firm ones, and the soak + measured spacing `[900, 3600]` in 70 % of samples - so mixed is the + normal steady state, and a window derived from one global spacing + would be wrong THERE rather than in an edge case. + + ⚠️ WHY THE ONE-HOUR CAP, WHICH IS NOT DECORATION. FCST-04 LEAVES HOLES ON + PURPOSE: an hour with no published price AND no yesterday value is left + OUT of the series, and `price_forecast.py`'s module note names `at()` + raising `Absence` for it as the behaviour that sends the caller to the + static profile (§11.4 for that hour). Uncapped, "until the next point's + start" would stretch the point BEFORE a three-hour hole across the whole + hole and answer every instant in it with a stale price. Filling holes is + the defect this module exists to refuse, and it would have been + introduced by the fix for a different one. + + ⚠️ THE LAST POINT'S WINDOW IS DECIDED, NOT INHERITED. There is no next + start and nothing in §9.3 says how wide the final interval is, so THE + LAST POINT IS GIVEN THE SAME SPAN AS THE POINT BEFORE IT - one hour when + it is the only point. Three consequences, all intended: + + * on a 96-point 900 s day the 23:45 point covers 23:45 .. 00:00, so + `at(23:50)` ANSWERS and `at(00:00)` raises `Absence`. A flat one-hour + tail would have answered 00:30 TOMORROW with today's last price, + which is reaching for a neighbour across a day boundary; + * on an hourly series the tail is an hour, exactly as it shipped, so no + static or hourly site changes behaviour; + * on a MIXED series whose last point is a carried HOURLY one behind + quarter-hourly firm ones, the tail is 900 s and the last 45 minutes + of the horizon raise `Absence`. That is the SAFE direction and it is + accepted rather than tolerated: the caller falls back to the static + profile (§11.4) instead of trading on a span the series never + demonstrated, and a carried point is provisional, so §8.3 forbids any + irreversible action resting on it regardless. + + ⚠️ ORDER IS ESTABLISHED HERE, NOT ASSUMED - and that is the difference + between this and the rule it replaces. `at()` now reads neighbours, so it + WOULD depend on sortedness; the table is therefore sorted by UTC start at + construction, and no caller's list order can reach it. `_validate_raw()` + rejects out-of-order and duplicate starts on the `load_prices()` path, + `price_history._parse_points()` re-sorts what it reads off disk, and + FCST-04's fill emits ascending - but `PriceRead.series()` itself has + never enforced any of that, and an assumption that holds today and is + unstated is how this class of defect recurs. + + ⚠️ NOTHING HERE RAISES ON A MALFORMED SERIES. A duplicated start - not + reachable through `load_prices()`, which refuses it - gives the earlier + list entry a ZERO-width window and the later one the real span, so one of + the pair becomes unreachable through `at()` and neither the 30 s control + path nor construction raises. + """ + rows = sorted(((p.start.astimezone(timezone.utc), i, p) + for i, p in enumerate(points)), + key=lambda r: (r[0], r[1])) + out = [] + for n, (start, _i, point) in enumerate(rows): + if n + 1 < len(rows): + span = rows[n + 1][0] - start # -> the next point's start + elif n: + span = start - rows[n - 1][0] # -> the tail, see above + else: + span = MAX_POINT_SPAN # -> a series of one point + if span > MAX_POINT_SPAN: + span = MAX_POINT_SPAN + out.append((start, start + span, point)) + return tuple(out) + + class _Series(PriceRead): kind = "series" @@ -564,6 +679,10 @@ class _Series(PriceRead): self.provider = provider self.fetched_at = fetched_at self.regime = volatility_regime(product) + # ⚠️ TARIFF-14. Built ONCE, here, so that `at()` scans an index whose + # order it established itself rather than one it hopes it was handed. + # See `_cover_table()` for the window rule and for what the tail does. + self._cover = _cover_table(self.points) self.words = (f"{len(self.points)} points from {provider} " f"({product.value}, {self.regime} thresholds)") @@ -572,25 +691,47 @@ class _Series(PriceRead): return {"prices": [p.canonical() for p in self.points]} def at(self, instant: datetime) -> PricePoint: - """The point covering `instant`. Raises `Absence` if the series does not. + """The point whose OWN interval covers `instant`, at ANY resolution. ⚠️ RETURNS A POINT OR RAISES; it never returns None. `series.at(t) or fallback` is the same defaulting bug one level down, so the falsy answer does not exist. Ask `covers()` first if you mean to branch. + + ⚠️ TARIFF-14: THE WINDOW IS NOT ONE HOUR WIDE ANY MORE, AND WAS NEVER + ENTITLED TO BE. It runs to the next point's start, capped at an hour - + `_cover_table()` above holds the rule, the tail decision and the + reproduction on a quarter-hourly series. A fixed `s + 1 h` here + returned the FIRST point still inside its own hour, which on the live + 900 s feed is neither the covering point nor, past the first hour, + even the top of the hour. + + A linear scan over the index, which is 192 entries on the shipped 48 h + horizon. This runs on the 30 s control path through + `arbitrage._point_at()`; a bisect would be correct and is not worth + the second way to be wrong at this size. """ u = instant.astimezone(timezone.utc) - for p in self.points: - s = p.start.astimezone(timezone.utc) - if s <= u < s + timedelta(hours=1): - return p + for start, end, point in self._cover: + if start <= u < end: + return point raise Absence( f"no price point covers {u.isoformat()}; the series runs " f"{self.points[0].start.isoformat()} .. " - f"{self.points[-1].start.isoformat()}. An hour outside the series " - f"is an hour with no price, which is §11.4 for that hour - not a " - f"reason to reach for a neighbouring one.") + f"{self.points[-1].start.isoformat()}. An instant outside the " + f"series - or inside a hole FCST-04 left because nothing " + f"published that hour - is an instant with no price, which is " + f"§11.4 for it, not a reason to reach for a neighbouring one.") def covers(self, instant: datetime) -> bool: + """Whether `at(instant)` would answer. + + ⚠️ SHARED TARIFF-14 EXACTLY, AND IS FIXED BY THE SAME CHANGE. It has + no window rule of its own - it asks `at()` - so on the 900 s feed it + answered True for instants no point covered and False for none of + them, because the hour-wide window is never SHORTER than the truth. + It is kept as a delegation for that reason: a second copy of the rule + is a second place for it to be wrong. + """ try: self.at(instant) return True @@ -598,7 +739,17 @@ class _Series(PriceRead): return False def window(self, since: datetime, until: datetime) -> tuple: - """Points whose start falls in [since, until). May legitimately be empty.""" + """Points whose START falls in [since, until). May legitimately be empty. + + ⚠️ DID NOT SHARE TARIFF-14, AND THE REASON IS WORTH STATING RATHER + THAN REDISCOVERING. This grades a point's `start` against a half-open + range and never asks how long a point lasts, so it carries no + resolution assumption to be wrong about - the same call answers the + same set at 900 s, at 3600 s and mixed. It is therefore NOT a + coverage query: a point that starts before `since` and runs into it + is absent from this tuple by design. `at()`/`covers()` are the + coverage questions. + """ a = since.astimezone(timezone.utc) b = until.astimezone(timezone.utc) return tuple(p for p in self.points diff --git a/goodwe_controller/test_arbitrage.py b/goodwe_controller/test_arbitrage.py index 5509acf..7552a44 100644 --- a/goodwe_controller/test_arbitrage.py +++ b/goodwe_controller/test_arbitrage.py @@ -1376,6 +1376,312 @@ check("soc_bounds() returns the shipped pair, floor first", float(PRICE_OPTION_DEFAULTS["arbitrage_soc_ceiling"]))) +# --------------------------------------------------------------------------- # +print() +print("10. TARIFF-14 - the QUARTER-HOURLY day, end to end through decide()") +# ⚠️ THIS IS THE PROPERTY THE DEFECT ACTUALLY BROKE, WHICH IS WHY IT IS GRADED +# HERE AND NOT ONLY ON `at()`. `_act()` decides whether this interval is the +# buy one or the sell one with `here.start == v.cheapest.start` - an IDENTITY +# test between the point `_point_at()` matched and the point the verdict was +# built from. `_Series.at()` used to return the first point whose start was +# within one hour BEFORE the instant, in list order, so on a 900 s series that +# identity almost never held: the strategy idled through exactly the interval +# it exists to trade, and read `here.provisional` and §17's `price < 0` off a +# point that was not the one being traded. +# +# Belgium moved to a 15-minute MTU on 2025-10-01 and the live day-ahead feed +# returns 96 points at 900 s. `static_tariff.py` emits HOURLY points, so every +# fixture above this line - and every static site - is immune; this is +# reachable only down the dynamic path TARIFF-08/09 made work. + +QD0 = datetime(2026, 1, 15, 0, 0, tzinfo=timezone(timedelta(hours=1))) +QUARTER = timedelta(minutes=15) + + +def qplan_of(prices, *, provisional=False, start=QD0, gaps=None): + """A TariffPlan over `prices` spaced by `gaps` seconds (900 s by default). + + `gaps` is what makes the MIXED fixture below expressible: FCST-04's + carry-forward emits HOURLY carried points beside QUARTER-HOURLY firm ones, + and the soak measured spacing [900, 3600] in 70 % of samples. + """ + g = [900] * (len(prices) - 1) if gaps is None else list(gaps) + flags = ([bool(provisional)] * len(prices) + if isinstance(provisional, bool) else list(provisional)) + pts, t = [], start + for i, (p, f) in enumerate(zip(prices, flags)): + pts.append(PricePoint(start=brussels(t), eur_per_kwh=round(float(p), 6), + provisional=bool(f))) + if i < len(g): + t += timedelta(seconds=g[i]) + return TariffPlan(PriceRead.series(pts, PriceProduct.DAY_AHEAD, + "generic_dayahead", start), + None, "provider") + + +def qprobe(i, *, start=QD0, gaps=None): + """An instant 7 minutes into slot `i` - inside every window, never on a + boundary, and NOT the midpoint of an hour either, so an implementation that + answered per hour and one that answers per quarter differ here.""" + g = [900] * i if gaps is None else list(gaps)[:i] + return start + timedelta(seconds=sum(g)) + timedelta(minutes=7) + + +def shipped_at(series, u): + """THE RULE TARIFF-14 REPLACED, kept here as the COUNTERFACTUAL ONLY. + + The first point whose start is within one hour before `u`, in list order. + Nothing below is graded against it; it exists so that the fixtures can be + shown to SEPARATE the two rules. A check whose data cannot reach the + failure it names is decoration (MUTATION.md), and on an hourly series the + two rules agree at every instant - which is why 29 files stayed green. + """ + uu = u.astimezone(UTC) + for p in series.points: + s = p.start.astimezone(UTC) + if s <= uu < s + timedelta(hours=1): + return p + return None + + +# The hourly day interpolated across its quarters, with the two extremes +# planted OFF the hour boundary on purpose. +QBASE = [] +for _i in range(24): + _a, _b = DAY[_i], DAY[(_i + 1) % 24] + for _q in range(4): + QBASE.append(round(_a + (_b - _a) * _q / 4 + 0.000001 * (_i * 4 + _q), + 6)) +Q_CHEAP_I = 4 * 5 + 3 # 05:45 - a :45 quarter, not a whole hour +Q_DEAR_I = 4 * 19 + 1 # 19:15 - a :15 quarter, not a whole hour +QDAY = list(QBASE) +QDAY[Q_CHEAP_I] = round(min(QBASE) - 0.030, 6) +QDAY[Q_DEAR_I] = round(max(QBASE) + 0.030, 6) +QPLAN = qplan_of(QDAY) +QPTS = QPLAN.read.points + +print("10a. the fixture is not degenerate, and it SEPARATES the two rules") +_qgaps = sorted({int((b.start - a.start).total_seconds()) + for a, b in zip(QPTS, QPTS[1:])}) +check(f"the day is 96 points at {_qgaps} s - the shape the live feed has " + f"returned since Belgium's 2025-10-01 move to a 15-minute MTU", + len(QPTS) == 96 and _qgaps == [900]) +check(f"the cheapest quarter is slot {Q_CHEAP_I} and the dearest slot " + f"{Q_DEAR_I} - neither first, neither last, {abs(Q_CHEAP_I - Q_DEAR_I)} " + f"slots apart, and NEITHER ON A WHOLE HOUR, without which this section " + f"would grade nothing an hourly fixture does not already grade", + QDAY.index(min(QDAY)) == Q_CHEAP_I and QDAY.index(max(QDAY)) == Q_DEAR_I + and Q_CHEAP_I % 4 != 0 and Q_DEAR_I % 4 != 0 + and Q_CHEAP_I not in (0, 95) and Q_DEAR_I not in (0, 95) + and abs(Q_CHEAP_I - Q_DEAR_I) > 1) +check("...and all 96 prices are distinct, so 'the cheapest quarter' names one " + "quarter and no tie-break is what any check below is really grading", + len(set(QDAY)) == 96) +check(f"...and its spread is {round(max(QDAY) - min(QDAY), 6):.4f} EUR/kWh, " + f"clear of the widened threshold, so what follows grades the strategy " + f"and not the spread rung", + round(max(QDAY) - min(QDAY), 6) > required_spread(_shipped, NO_WEATHER)) +# ⚠️ HABIT 4, AND THE ONLY REASON THIS SECTION IS WORTH ITS LINES. State, as a +# check, that the shipped rule and the fixed one give DIFFERENT points at the +# two probes that matter. Without it, every check below could be passing on a +# fixture the defect could not have changed. +_cf_buy = derived(lambda: shipped_at(QPLAN.read, qprobe(Q_CHEAP_I))) +_cf_sell = derived(lambda: shipped_at(QPLAN.read, qprobe(Q_DEAR_I))) +check(f"the shipped rule and this one give DIFFERENT points at both probes - " + f"buy: {derived(lambda: _cf_buy.eur_per_kwh)} instead of " + f"{QDAY[Q_CHEAP_I]}, sell: {derived(lambda: _cf_sell.eur_per_kwh)} " + f"instead of {QDAY[Q_DEAR_I]} - so the fixture can reach the failure it " + f"names", + _cf_buy is not QPTS[Q_CHEAP_I] and _cf_sell is not QPTS[Q_DEAR_I] + and _cf_buy is not None and _cf_sell is not None) + +print() +print("10b. the buy/sell identity test matches the actual extremum interval") +_qv = QPLAN.arbitrage(peak_guard_active=False, min_spread_eur_kwh=_shipped, + uncertainty=NO_WEATHER) +check("the verdict's cheapest and dearest are the two QUARTERS, not the hours " + "containing them - `static_tariff` reduces over every firm point and " + "carries no resolution assumption of its own", + derived(lambda: _qv.cheapest) is QPTS[Q_CHEAP_I] + and derived(lambda: _qv.dearest) is QPTS[Q_DEAR_I]) +_pa_buy = derived(lambda: _arb._point_at(QPLAN, qprobe(Q_CHEAP_I))) +_pa_sell = derived(lambda: _arb._point_at(QPLAN, qprobe(Q_DEAR_I))) +check("THE PROPERTY THE DEFECT BROKE: at the buy probe `_point_at()` returns " + "the verdict's OWN cheapest point, by identity - which is what makes " + "`_act()`'s `here.start == v.cheapest.start` true, and what made it " + "false on every quarter-hourly series the add-on has ever fetched", + _pa_buy is derived(lambda: _qv.cheapest)) +check("...and the same at the sell probe, against the verdict's dearest", + _pa_sell is derived(lambda: _qv.dearest)) +_qbuy = decide(QPLAN, qprobe(Q_CHEAP_I), peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0) +_qsell = decide(QPLAN, qprobe(Q_DEAR_I), peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0) +check("...so decide() CHARGES in the cheapest quarter and DISCHARGES in the " + "dearest one, which is the trade the strategy exists to make and the " + "one it slept through", + _qbuy.action == CHARGE and _qsell.action == DISCHARGE) +check(f"...at THAT QUARTER's price and not at the price of the hour " + f"containing it (buy {_qbuy.price_now_eur_kwh}, sell " + f"{_qsell.price_now_eur_kwh})", + near(_qbuy.price_now_eur_kwh, QDAY[Q_CHEAP_I]) + and near(_qsell.price_now_eur_kwh, QDAY[Q_DEAR_I])) +check("...and both reasons quote that quarter's price, so the operator's log " + "names the interval that was actually traded", + f"{QDAY[Q_CHEAP_I]:.4f}" in _qbuy.reason + and f"{QDAY[Q_DEAR_I]:.4f}" in _qsell.reason) +# The negative arm. Without it "it acted" could be "it acts everywhere". +_QOFF_I = Q_CHEAP_I + 2 +check(f"the off-peak probe's own price differs from its hour's :00 price " + f"({QDAY[_QOFF_I]} vs {QDAY[4 * (_QOFF_I // 4)]}), so the check below " + f"can tell which point the decision read", + QDAY[_QOFF_I] != QDAY[4 * (_QOFF_I // 4)] + and _QOFF_I not in (Q_CHEAP_I, Q_DEAR_I)) +_qoff = decide(QPLAN, qprobe(_QOFF_I), peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0) +check("a quarter that is neither extremum still idles, for the reason that " + "names the right rule, and reports ITS OWN price - not the price of the " + "quarter the shipped scan would have matched", + _qoff.action is None and _qoff.reason == _arb.IDLE_OFF_PEAK_HOUR + and near(_qoff.price_now_eur_kwh, QDAY[_QOFF_I])) + +print() +print("10c. §17 and §8.3 are evaluated on the QUARTER being traded") +# ⚠️ §17 AT 900 s. The whole day is below zero and the dearest QUARTER is the +# least negative one - section 2's shape, at the resolution the live feed +# actually publishes. Under the shipped scan this probe never reached the +# guard at all: the identity test failed first and the decision idled as +# "neither the buy hour nor the sell hour", so §17 was not so much wrong as +# never consulted. +_shift = round(max(QDAY) + 0.020, 6) +QNEG = [round(p - _shift, 6) for p in QDAY] +QNEG_PLAN = qplan_of(QNEG) +check(f"the negative day's dearest quarter is slot {Q_DEAR_I} and it is BELOW " + f"ZERO ({max(QNEG):+.4f}), and its spread still clears the widened " + f"threshold - so the refusal below is §17's and not the spread rung's", + QNEG.index(max(QNEG)) == Q_DEAR_I and max(QNEG) < 0 + and round(max(QNEG) - min(QNEG), 6) > required_spread(_shipped, + NO_WEATHER)) +_qneg_sell = decide(QNEG_PLAN, qprobe(Q_DEAR_I), peak_guard_active=False, + soc=50.0, min_spread_eur_kwh=_shipped, + uncertainty=NO_WEATHER, charge_w=MAX_W, discharge_w=MAX_W, + soc_floor_pct=20.0, soc_ceiling_pct=95.0) +check("selling into a negative-price QUARTER is refused, and the reason names " + "§17 and THAT QUARTER's price", + _qneg_sell.action is None and "§17" in _qneg_sell.reason + and f"{max(QNEG):+.4f}" in _qneg_sell.reason) +# ...and the other side, so the refusal is about the sign and nothing else. +QPOS = list(QNEG) +QPOS[Q_DEAR_I] = round(-max(QNEG), 6) +check("...while the SAME quarter-hourly series with that ONE price's SIGN " + "FLIPPED discharges instead", + decide(qplan_of(QPOS), qprobe(Q_DEAR_I), peak_guard_active=False, + soc=50.0, min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0).action == DISCHARGE + and sum(1 for a, b in zip(QNEG, QPOS) if a != b) == 1) +# §8.3's flag, read off the covering point. `_act`'s own provisional guard +# stays unreachable here for section 4's reason - the extrema are firm by +# `_firm_or_die` and starts are unique - and THAT ARGUMENT IS EXACTLY WHAT THE +# DEFECT BROKE at 900 s, because the point `at()` returned was not the +# verdict's point at all. It is restated here at the resolution that matters. +check("the point matched for the buy quarter IS the verdict's own cheapest " + "point and is FIRM, so `_act`'s §8.3 guard is unreachable at 900 s for " + "the same construction section 4 states at 3600 s", + _pa_buy is derived(lambda: _qv.cheapest) + and derived(lambda: _pa_buy.provisional) is False + and len({p.start for p in QPTS}) == len(QPTS)) + +print() +print("10d. the MIXED series FCST-04 actually emits, traded end to end") +# ⚠️ 70 % OF SOAK SAMPLES, NOT AN EDGE CASE. The carry-forward emits HOURLY +# carried points beside QUARTER-HOURLY firm ones, so a window derived from one +# global spacing would be wrong in the NORMAL steady state. Here: 12 carried +# hourly points for the morning, then firm quarters for the rest of the day. +MIX_N_H, MIX_N_Q = 12, 48 +MIX_GAPS = [3600] * MIX_N_H + [900] * (MIX_N_Q - 1) +MIX_PRICES = [QDAY[4 * i] for i in range(MIX_N_H)] + QDAY[4 * MIX_N_H:] +MIX_FLAGS = [True] * MIX_N_H + [False] * MIX_N_Q +MIX_PLAN = qplan_of(MIX_PRICES, provisional=MIX_FLAGS, gaps=MIX_GAPS) +MIX_PTS = MIX_PLAN.read.points +_mix_gaps = sorted({int((b.start - a.start).total_seconds()) + for a, b in zip(MIX_PTS, MIX_PTS[1:])}) +check(f"the mixed plan really is mixed - spacing {_mix_gaps} s, " + f"{MIX_N_H} carried and {MIX_N_Q} firm points, so neither half is what " + f"the other half's window rule would give", + _mix_gaps == [900, 3600] + and sum(1 for p in MIX_PTS if p.provisional) == MIX_N_H + and sum(1 for p in MIX_PTS if not p.provisional) == MIX_N_Q) +_mix_cheap_i = MIX_N_H + (MIX_PRICES[MIX_N_H:].index(min(MIX_PRICES[MIX_N_H:]))) +_mix_dear_i = MIX_N_H + (MIX_PRICES[MIX_N_H:].index(max(MIX_PRICES[MIX_N_H:]))) +check(f"...and its firm half's extremes are quarters {_mix_cheap_i} and " + f"{_mix_dear_i}, neither on a whole hour, so the identity test below is " + f"about the quarter and not about the hour", + (_mix_cheap_i - MIX_N_H) % 4 != 0 and (_mix_dear_i - MIX_N_H) % 4 != 0) +_mv = MIX_PLAN.arbitrage(peak_guard_active=False, min_spread_eur_kwh=_shipped, + uncertainty=NO_WEATHER) +_mix_buy_t = qprobe(_mix_cheap_i, gaps=MIX_GAPS) +_mix_sell_t = qprobe(_mix_dear_i, gaps=MIX_GAPS) +check("the verdict over a MIXED series still names firm quarters, and " + "`_point_at()` matches them by identity at their own probes", + derived(lambda: _mv.cheapest) is MIX_PTS[_mix_cheap_i] + and derived(lambda: _mv.dearest) is MIX_PTS[_mix_dear_i] + and derived(lambda: _arb._point_at(MIX_PLAN, _mix_buy_t)) + is MIX_PTS[_mix_cheap_i] + and derived(lambda: _arb._point_at(MIX_PLAN, _mix_sell_t)) + is MIX_PTS[_mix_dear_i]) +check("...so the mixed day CHARGES and DISCHARGES in its firm quarters, at " + "their own prices", + decide(MIX_PLAN, _mix_buy_t, peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0).action == CHARGE + and decide(MIX_PLAN, _mix_sell_t, peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0).action == DISCHARGE) +# An instant inside a CARRIED hour of the same series. The covering point is +# the carried one; the shipped scan answered a point from the FIRM block +# earlier in the list, which is §8.3's flag read off the wrong point. +_mix_carried_t = QD0 + timedelta(hours=MIX_N_H - 1, minutes=37) +check("an instant inside a CARRIED hour of the mixed series matches the " + "CARRIED point, whose provisional flag is what §8.3 reads - and the " + "strategy does nothing there, because a carried point is not in the " + "firm trade set", + derived(lambda: _arb._point_at(MIX_PLAN, _mix_carried_t)) + is MIX_PTS[MIX_N_H - 1] + and MIX_PTS[MIX_N_H - 1].provisional is True + and decide(MIX_PLAN, _mix_carried_t, peak_guard_active=False, soc=50.0, + min_spread_eur_kwh=_shipped, uncertainty=NO_WEATHER, + charge_w=MAX_W, discharge_w=MAX_W, soc_floor_pct=20.0, + soc_ceiling_pct=95.0).action is None) + +print() +print("10e. the HOURLY day is unchanged, which is the regression half") +# ⚠️ A REGRESSION GUARD, NOT A WITNESS. On an hourly series the shipped rule +# and this one agree at every instant, which is precisely why 29 files stayed +# green through the defect. It is here so that a fix which got 900 s right by +# breaking 3600 s cannot pass. +_h_buy = act(PLAN, CHEAP_H, thresh=_shipped, unc=NO_WEATHER) +_h_sell = act(PLAN, DEAR_H, thresh=_shipped, unc=NO_WEATHER) +check("the hourly fixture still charges in its cheapest hour and discharges " + "in its dearest, at those hours' prices", + _h_buy.action == CHARGE and _h_sell.action == DISCHARGE + and near(_h_buy.price_now_eur_kwh, DAY[CHEAP_H]) + and near(_h_sell.price_now_eur_kwh, DAY[DEAR_H])) +check("...and a probe 59 minutes into an hourly slot still matches that slot, " + "so the hourly window did not shrink under the fix", + derived(lambda: _arb._point_at( + PLAN, D0 + timedelta(hours=CHEAP_H, minutes=59))) + is derived(lambda: PLAN.read.points[CHEAP_H])) + + # --------------------------------------------------------------------------- # print() if fails: diff --git a/goodwe_controller/test_price.py b/goodwe_controller/test_price.py index fbdf04b..4950afc 100644 --- a/goodwe_controller/test_price.py +++ b/goodwe_controller/test_price.py @@ -1944,6 +1944,363 @@ check("'off' is not refused, on any of the other configurations", {"price_csv_path": ""}))) +# --------------------------------------------------------------------------- # +print() +print("13. TARIFF-14 - the point whose OWN interval covers the instant") +# ⚠️ THE RESOLUTION IS THE WHOLE SECTION, AND ITS ABSENCE IS WHY THE DEFECT +# SHIPPED. Every price fixture in this repository before TARIFF-08 was the +# invented HOURLY row shape, and on an hourly series `s <= u < s + 1 h` and +# "until the next point's start" AGREE AT EVERY INSTANT - so the suite had +# been exercising the one resolution that cannot expose the bug. Section 8 +# counts points on the 23- and 25-hour days and never asks `at()` for a +# quarter-hour. Belgium moved to a 15-minute MTU on 2025-10-01 and the live +# day-ahead feed returns 96 points at 900 s. + +# ⚠️ BRUSSELS MIDNIGHT, NOT UTC MIDNIGHT, so that every wall-clock time in +# a check name below is the time the series really renders - "00:15" and +# "23:45" are what `canonical()` writes, not an offset away from it. +Q0 = datetime(2026, 1, 15, 0, 0, tzinfo=timezone(timedelta(hours=1))) +HOUR = timedelta(hours=1) + + +def spaced(gaps, first=0.10, step=0.01, provisional=None, start=Q0): + """A series whose consecutive gaps are exactly `gaps`, in SECONDS. + + `len(points) == len(gaps) + 1`. Prices ascend by `step`, so a point can be + named by VALUE as well as by identity and a sweep that compared only + prices could still tell two points apart. + """ + n = len(gaps) + 1 + flags = [False] * n if provisional is None else list(provisional) + pts, t = [], start + for i in range(n): + pts.append(PricePoint(brussels(t), round(first + step * i, 6), + bool(flags[i]))) + if i < len(gaps): + t += timedelta(seconds=gaps[i]) + return PriceRead.series(pts, PriceProduct.DAY_AHEAD, "tariff14", start) + + +def gaps_of(s): + u = [p.start.astimezone(UTC) for p in s.points] + return [int((b - a).total_seconds()) for a, b in zip(u, u[1:])] + + +# The three shapes the acceptance criteria name, plus the two degenerate ones. +Q = spaced([900] * 95) # the shipped 96-point Belgian day +H = spaced([3600] * 23) # what every older fixture carried +# ⚠️ FCST-04'S STEADY STATE, NOT AN EDGE CASE. Eight firm quarter-hours, then +# the carried HOURLY points the fill adds for the hours nobody published. The +# seam between them is a 900 s gap and the tail is 3600 s. +MIX = spaced([900] * 8 + [3600] * 3, + provisional=[False] * 9 + [True] * 3) +# The same seam made the END of the series - the last point is the FIRST +# carried hourly one, and the gap behind it is 900 s. This is the fixture the +# tail decision is graded on. +MIX_TAIL = spaced([900] * 8, provisional=[False] * 8 + [True]) +ONE = spaced([]) + +print("13a. the fixtures are not degenerate - asserted before anything else") +check(f"the quarter-hourly fixture is the live shape - 96 points, spacing " + f"{sorted(set(gaps_of(Q)))} s", + len(Q.points) == 96 and set(gaps_of(Q)) == {900}) +check(f"the hourly fixture is 24 points at {sorted(set(gaps_of(H)))} s - and " + f"it is a REGRESSION guard, not a witness: the shipped rule and this " + f"one agree at every instant of it, which is exactly why nothing caught " + f"this", + len(H.points) == 24 and set(gaps_of(H)) == {3600}) +check(f"...while the MIXED fixture really is mixed, spacing " + f"{sorted(set(gaps_of(MIX)))} s - the pair the FCST-04 soak measured in " + f"70 % of samples, so a window derived from ONE global spacing would be " + f"wrong in the NORMAL steady state", + set(gaps_of(MIX)) == {900, 3600} + and set(gaps_of(MIX_TAIL)) == {900} + and gaps_of(MIX)[-1] == 3600 and gaps_of(MIX_TAIL)[-1] == 900) +check("...and every point of every fixture carries a DISTINCT price, so 'the " + "covering point' names one point and a wrong answer cannot coincide " + "with the right one", + all(len({p.eur_per_kwh for p in s.points}) == len(s.points) + for s in (Q, H, MIX, MIX_TAIL))) +check("...and the mixed fixtures carry BOTH firm and carried points, which is " + "what makes them FCST-04's output rather than a spacing exercise", + all(any(p.provisional for p in s.points) + and any(not p.provisional for p in s.points) + for s in (MIX, MIX_TAIL))) + +print() +print("13b. the reading the shipped scan got wrong, stated as literals") +# ⚠️ ASSERTED AGAINST TYPED-OUT INDICES AND PRICES, not against a window this +# file recomputed by price.py's own rule. The sweep below is exhaustive and +# the sweep is derived; these four are the anchors that keep it honest. +_q15 = derived(lambda: Q.at(Q0 + timedelta(minutes=15))) +check(f"at(00:15) on the 900 s series is the 00:15 point at 0.11 EUR/kWh - the " + f"shipped scan answered the 00:00 point at 0.10, because the :00 " + f"point's HOUR had not run out yet (saw {_q15!r})", + _q15 is Q.points[1] and derived(lambda: _q15.eur_per_kwh) == 0.11) +_q195 = derived(lambda: Q.at(Q0 + timedelta(minutes=195))) +check(f"at(03:15) is the 03:15 point at 0.23 - the shipped scan answered " + f"02:30 at 0.20 here, and that is the SECOND failure mode: from the " + f"second hour on the first point still inside its own hour wins, so the " + f"error COMPOUNDS with depth into the series (saw {_q195!r})", + _q195 is Q.points[13] and derived(lambda: _q195.eur_per_kwh) == 0.23) +check("at(00:30) and at(00:45) are their own quarters too, so the answer is " + "not merely 'one point later'", + derived(lambda: Q.at(Q0 + timedelta(minutes=30))) is Q.points[2] + and derived(lambda: Q.at(Q0 + timedelta(minutes=45))) is Q.points[3]) +check("on the MIXED series the quarter-hourly half answers per QUARTER and " + "the carried hourly half answers per HOUR, in one series - at(00:15) is " + "the 00:15 point and at(02:30) is the 02:00 one", + derived(lambda: MIX.at(Q0 + timedelta(minutes=15))) is MIX.points[1] + and derived(lambda: MIX.at(Q0 + timedelta(minutes=150))) is MIX.points[8] + and derived(lambda: MIX.points[8].start.astimezone(UTC)) == + Q0 + timedelta(hours=2)) + +print() +print("13c. the exhaustive sweep, at 900 s, 3600 s and mixed") + + +def windows(s): + """`(start_utc, width)` per point, computed HERE from the gaps. + + Deliberately NOT read off `_Series._cover`: a check built from the table it + grades compares a thing against itself. The rule is restated - a point runs + to the next start, capped at an hour; the last point copies the width in + front of it; a lone point gets an hour. + """ + u = [p.start.astimezone(UTC) for p in s.points] + out = [] + for i, t in enumerate(u): + if i + 1 < len(u): + g = u[i + 1] - t + elif i: + g = t - u[i - 1] + else: + g = HOUR + out.append((t, g if g < HOUR else HOUR)) + return out + + +def sweep(s): + """Every probe whose covering point is unambiguous, and what `at()` said.""" + wrong = [] + for i, (t, w) in enumerate(windows(s)): + for off in (timedelta(0), w / 2, w - timedelta(seconds=1)): + got = derived(lambda: s.at(t + off)) + if got is not s.points[i]: + wrong.append(f"{(t + off).isoformat()} -> {got!r}, " + f"wanted {s.points[i].eur_per_kwh}") + return wrong + + +for _name, _s in (("900 s", Q), ("3600 s", H), ("mixed 900/3600", MIX), + ("mixed, 900 s tail", MIX_TAIL)): + _w = derived(lambda s=_s: sweep(s)) + check(f"{_name}: all {3 * len(_s.points)} probes inside the series return " + f"the point whose OWN interval covers them " + f"({_w if _w != [] else 'no misses'})", _w == []) + +print() +print("13d. the one-hour cap, which is what keeps FCST-04's holes holes") +# ⚠️ NOT DECORATION. FCST-04 leaves an hour OUT when nothing published it and +# yesterday has no value either, and price_forecast.py's module note names +# `at()` raising Absence for it as the behaviour that sends the caller to the +# static profile (§11.4 for that hour). Uncapped, "until the next point's +# start" would stretch the point before a three-hour hole across the whole +# hole - filling holes with a stale price, which is the defect this module +# exists to refuse, introduced by the fix for a different one. +HOLE = PriceRead.series( + [PricePoint(brussels(Q0 + timedelta(hours=i)), 0.10 + i / 100.0, False) + for i in (0, 1, 4, 5)], + PriceProduct.DAY_AHEAD, "tariff14", Q0) +_in_hole = [Q0 + timedelta(hours=2), Q0 + timedelta(hours=2, minutes=30), + Q0 + timedelta(hours=3), Q0 + timedelta(hours=3, minutes=59)] +check(f"a three-hour hole stays a hole: all {len(_in_hole)} instants inside it " + f"raise Absence, while the point BEFORE it still covers its own hour and " + f"the point after it answers from its own start", + all(raises(lambda u=u: HOLE.at(u), Absence) for u in _in_hole) + and derived(lambda: HOLE.at(Q0 + timedelta(hours=1, minutes=59))) + is HOLE.points[1] + and derived(lambda: HOLE.at(Q0 + timedelta(hours=4))) is HOLE.points[2]) +check(f"...and the cap is stated once, as MAX_POINT_SPAN = " + f"{_price.MAX_POINT_SPAN}, rather than spelled into the lookup", + _price.MAX_POINT_SPAN == HOUR) +# ⚠️ AND THE OTHER HALF OF THE HOLE, WHICH IS THE ONE §8.3 READS. FCST-04 +# fills a whole hour nobody published with ONE carried point, so a +# quarter-hourly day can carry an hourly carried point in the middle of it. +# The instant inside that hour is covered by the CARRIED point - and the +# shipped scan answered the FIRM quarter 52 minutes earlier, because that was +# the first point still inside its own hour. `_act()`'s `here.provisional` +# guard and `price_forecast._walk()`'s `series.at(u).provisional` both read the +# flag off whatever that lookup returned, so the flag was FIRM on an hour +# nobody published. +SANDWICH = PriceRead.series( + [PricePoint(brussels(Q0 + timedelta(minutes=15 * i)), + round(0.10 + i / 1000.0, 6), False) for i in range(44)] + + [PricePoint(brussels(Q0 + timedelta(hours=11)), 0.500, True)] + + [PricePoint(brussels(Q0 + timedelta(hours=12, minutes=15 * i)), + round(0.20 + i / 1000.0, 6), False) for i in range(4)], + PriceProduct.DAY_AHEAD, "tariff14", Q0) +_sand_t = Q0 + timedelta(hours=11, minutes=37) + + +def shipped_at(s, u): + """The rule TARIFF-14 replaced, as the COUNTERFACTUAL only - nothing is + graded against it. It is here so the fixture can be SHOWN to separate the + two rules; a check whose data cannot reach the failure it names grades + nothing, and on an hourly series these two agree at every instant.""" + uu = u.astimezone(UTC) + for p in s.points: + st = p.start.astimezone(UTC) + if st <= uu < st + HOUR: + return p + return None + + +check("the sandwich fixture is FCST-04's real output - 44 firm quarters, ONE " + "carried hourly point filling the hour nobody published, then firm " + "quarters again", + len(SANDWICH.points) == 49 + and SANDWICH.points[44].provisional is True + and sum(1 for p in SANDWICH.points if p.provisional) == 1) +_sand_now = derived(lambda: SANDWICH.at(_sand_t)) +_sand_old = derived(lambda: shipped_at(SANDWICH, _sand_t)) +check(f"an instant inside the carried hour is covered by the CARRIED point, " + f"so §8.3's flag is read off the hour it belongs to. The shipped scan " + f"answered the FIRM quarter 52 minutes earlier " + f"({derived(lambda: _sand_old.start.isoformat())}, provisional=" + f"{derived(lambda: _sand_old.provisional)}), which is a carried hour " + f"wearing a published hour's flag", + _sand_now is SANDWICH.points[44] + and derived(lambda: _sand_now.provisional) is True + and _sand_old is SANDWICH.points[43] + and derived(lambda: _sand_old.provisional) is False) + +print() +print("13e. the LAST point's window - decided and documented, not inherited") +# There is no next start and §9.3 says nothing about how wide the final +# interval is, so THE LAST POINT IS GIVEN THE SAME SPAN AS THE POINT BEFORE IT +# (one hour when it is the only point). The alternative - a flat one-hour tail +# - is defensible for an hourly series and wrong for the live one: it answers +# 00:30 TOMORROW with today's last price, which is reaching for a neighbour +# across a day boundary. +_ql = Q.points[-1].start.astimezone(UTC) +_hl = H.points[-1].start.astimezone(UTC) +_ml = MIX_TAIL.points[-1].start.astimezone(UTC) +check("the 900 s day's last point covers its own QUARTER: at(+14:59) answers " + "and at(+15:00) raises, so today's 23:45 price cannot be returned for " + "00:30 tomorrow", + derived(lambda: Q.at(_ql + timedelta(minutes=14, seconds=59))) + is Q.points[-1] + and raises(lambda: Q.at(_ql + timedelta(minutes=15)), Absence)) +check("the hourly series' last point still covers an HOUR, exactly as it " + "shipped - no static or hourly site changes behaviour", + derived(lambda: H.at(_hl + timedelta(minutes=59))) is H.points[-1] + and raises(lambda: H.at(_hl + HOUR), Absence)) +check("the MIXED series ending at the firm/carried seam takes the 900 s in " + "front of it, so its last 45 minutes raise Absence. That is the SAFE " + "direction and it is CHOSEN: §11.4 sends the caller to the static " + "profile rather than trading on a span the series never demonstrated, " + "and the point is carried, so §8.3 forbids irreversible action on it " + "anyway", + derived(lambda: MIX_TAIL.at(_ml + timedelta(minutes=14, seconds=59))) + is MIX_TAIL.points[-1] + and raises(lambda: MIX_TAIL.at(_ml + timedelta(minutes=15)), Absence) + and MIX_TAIL.points[-1].provisional) +# ⚠️ HABIT 1. Without this the three checks above would all pass under a +# single global tail rule of 15 minutes, or of an hour, depending which two +# you read - so the claim that the tail FOLLOWS THE SERIES is asserted as its +# own check, at one instant where the three fixtures must disagree. +_t30 = [derived(lambda s=s, l=l: s.covers(l + timedelta(minutes=30))) + for s, l in ((Q, _ql), (H, _hl), (MIX_TAIL, _ml))] +check(f"...and those tails are genuinely three readings of one rule and not " + f"one constant: 30 minutes past its own last point the hourly series " + f"answers and both 900 s ones do not {_t30}", + _t30 == [False, True, False]) +check("a series of ONE point has no gap in front of it to copy, so it gets an " + "hour - the shipped behaviour, kept deliberately for the degenerate case", + derived(lambda: ONE.at(Q0 + timedelta(minutes=59, seconds=59))) + is ONE.points[0] + and raises(lambda: ONE.at(Q0 + HOUR), Absence)) + +print() +print("13f. order is established by the lookup, not assumed of the caller") +# ⚠️ `at()` now reads a point's NEIGHBOURS, so it WOULD depend on sortedness. +# The index is sorted at construction so that it cannot. `_validate_raw()` +# refuses out-of-order and duplicate starts on the `load_prices()` path +# (section 9 above), `price_history._parse_points()` re-sorts what it reads off +# disk and FCST-04's fill emits ascending - but `PriceRead.series()` itself has +# never enforced any of it, and an assumption that holds today and is unstated +# is how this class of defect recurs. +_REV = PriceRead.series(tuple(reversed(Q.points)), PriceProduct.DAY_AHEAD, + "tariff14", Q0) +check("the reversed fixture really is reversed, so the check below is not " + "grading two identical series", + _REV.points[0] is Q.points[-1] and _REV.points[-1] is Q.points[0] + and _REV.points != Q.points) +_rev_bad = derived(lambda: [p.start.isoformat() for p in Q.points + if _REV.at(p.start) is not p]) +check(f"...and a series handed its 96 points in REVERSE order answers at() " + f"identically at every one of them " + f"({_rev_bad if _rev_bad != [] else 'no misses'})", + _rev_bad == []) +check("...while the load path refuses an unsorted feed outright, so the two " + "defences are stated rather than one being quietly relied on", + raises(lambda: _price._validate_raw( + [RawPoint(T0 + timedelta(minutes=15), 0.1, False), + RawPoint(T0, 0.2, False)], _caps), SourceUnavailable)) +check("a DUPLICATED start does not raise anywhere - not at construction and " + "not on the 30 s control path. The later list entry takes the real span " + "and the earlier one becomes unreachable through at(); load_prices() " + "refuses the shape outright, and this says what happens to anyone who " + "builds one by hand", + derived(lambda: PriceRead.series( + [PricePoint(brussels(Q0), 0.10, False), + PricePoint(brussels(Q0), 0.20, False), + PricePoint(brussels(Q0 + timedelta(minutes=15)), 0.30, False)], + PriceProduct.DAY_AHEAD, "tariff14", Q0).at(Q0).eur_per_kwh) == 0.20) + +print() +print("13g. covers() shared it; window() did not") +check("covers() shared the defect EXACTLY, because it has no window rule of " + "its own - it asks at() - so the one change fixes both. On the 900 s " + "day it is now True 14:59 past the last point and False 15:00 past it, " + "where an hour-wide window said True for another 45 minutes", + derived(lambda: Q.covers(_ql + timedelta(minutes=14, seconds=59))) is True + and derived(lambda: Q.covers(_ql + timedelta(minutes=15))) is False + and derived(lambda: Q.covers(_ql + timedelta(minutes=30))) is False + and derived(lambda: Q.covers(Q0 + timedelta(minutes=15))) is True) +check("window() did NOT share it, and the reason is worth a check rather than " + "a rediscovery: it grades a point's START against a half-open range and " + "never asks how long a point lasts, so it carries no resolution " + "assumption at all. 00:07:30 .. 00:22:30 holds exactly the 00:15 point " + "- while at(00:07:30) is the 00:00 one, which is what makes it a " + "DIFFERENT question and not a coverage query", + derived(lambda: Q.window(Q0 + timedelta(minutes=7, seconds=30), + Q0 + timedelta(minutes=22, seconds=30))) + == (Q.points[1],) + and derived(lambda: Q.at(Q0 + timedelta(minutes=7, seconds=30))) + is Q.points[0]) + +print() +print("13h. the 'no data is not free' contract, unchanged") +check("at() answers with a PricePoint or raises Absence - never None, at any " + "resolution, so `series.at(t) or fallback` cannot become a price of zero", + all(isinstance(derived(lambda s=s: s.at(s.points[0].start)), PricePoint) + for s in (Q, H, MIX, MIX_TAIL, ONE)) + and raises(lambda: Q.at(Q0 - timedelta(seconds=1)), Absence) + and raises(lambda: Q.at(Q0 + timedelta(days=2)), Absence)) +_absence_words = _reason(lambda: Q.at(Q0 + timedelta(days=2))) +check(f"...and the refusal still names the range the series runs, so an " + f"operator can tell 'outside the horizon' from 'into a hole'", + Q.points[0].start.isoformat() in _absence_words + and Q.points[-1].start.isoformat() in _absence_words) +check("...and it never invents a neighbour: the instant one second before the " + "series starts is refused, not answered with the first point", + raises(lambda: Q.at(Q0 - timedelta(seconds=1)), Absence) + and raises(lambda: MIX.at(Q0 - timedelta(seconds=1)), Absence) + and raises(lambda: H.at(Q0 - timedelta(seconds=1)), Absence)) + + # --------------------------------------------------------------------------- # print() if fails: