FCST-04: the carry-forward fill reaches the plan, and the day it carries survives a restart
checks / unit (pull_request) Successful in 7m11s
checks / unit (pull_request) Successful in 7m11s
FCST-03 shipped `fill_from_yesterday()` complete and checked, and shipped the consumers written for it - `TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3, which drops provisional hours from the trade set rather than refusing the whole series. It shipped NO CALLER. Nothing in `app/` named the fill, so no series ever contained a carried point and every one of those consumers was exercising its all-firm branch. ⚠️ THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE. `load_prices()` returns what the feed published and nothing widened it to the requested horizon, so on a dynamic site between midnight and the 14-16 h CET publication the plan covered the rest of today and nothing more - measured on the fixture here, 24 h against a 48 h request. `arbitrage()` then took `max - min` over that truncated window and reported a spread. The number was real, the window was half the one configured, and nothing said so: tomorrow evening's peak, the hour a morning charge would be sold into, was not in the series to be compared against anything. A missing hour is not a wrong price - it is an hour the optimiser never learns exists. WHAT THIS ADDS. `app/price_history.py` keeps the PUBLISHED prices of the last KEEP_DAYS Brussels days in /data/price_history.json; `resolve_tariff()` gains a required keyword-only `history` and fills every unpublished hour of the horizon from yesterday's same-wall-clock hour, flagged. Persistence rather than a variable on the refresh task, because the window the fill exists for is midnight -> publication and an add-on restarted at 08:00 would otherwise spend all of it on the truncated series. Not in state.json: the refresh runs every 1800 s and that file holds the maintenance history and SAFETY-02's latch. ⚠️ ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR. Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill widens what the optimiser can SEE, never what it may DO, and test_price_history.py checks that so it cannot be overclaimed later. What does change a trade is that the filled series is windowed to [since, until), so hours that have ALREADY ELAPSED are no longer in the spread. Before, a cheap night that had gone sat in the series as the day's cheapest hour - and a cycle cannot buy in an hour that is past, so the trade was quietly never taken. Section 2 grades this by finding the cheapest firm hour with and without the fill and showing it MOVES, from 02:00 UTC to 08:00 UTC on the fixture. ⚠️ A DEAD FEED IS STILL §11.4, NOT YESTERDAY. The fill runs only on a `series` read. The two are indistinguishable to the optimiser - both 100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()` is False - so preferring yesterday's feed over the `static_profile` an operator configured FOR that case would change a documented degradation for a difference only the status page can see. That, and two more, in DECISIONS/FCST-04-the-day-that-is-carried.md. ⚠️ THE ANTI-DECAY RULE IS AN ORDERING, NOT A VALIDATION, and saying so is more useful than a guarantee that would not hold. Feeding the filled series back in is how this rots - day 2 carries day 1's guesses, day 3 carries day 2's carry of them, behind a `provisional` flag that stays TRUE while the NUMBER decays. No value-level check can catch it: a carried point and `eneco_eddy`'s published-provisional quarter are the same three fields with the same flag. So `carry()` is the only public entry, it records BEFORE it fills, the filled series exists only as a return value the object never sees again, and test_price_history.py walks app/ for any other module naming `_remember` and reads `carry()`'s AST to pin the order. EVIDENCE. 29/29 test files pass in the CI image. Eight mutations driven against app/price_history.py and app/static_tariff.py, __pycache__ purged before each run per MUTATION.md/TEST-15: newest-record lookup (5 red), recording the filled series (1), replace-instead-of-merge (10), UTC-date keying (10), removing the fill call entirely (12), a lost price field defaulting to 0.0 (2), prune by clock arithmetic (1), a degraded reason that stops naming its cause (1). No survivors; baseline and revert both green. Three of those first crashed the harness instead of reporting a FAIL - "it was refused" and "it broke on the way to being refused" must not grade the same - so `why()` and `_priced()` were added and the run repeated. No option added, config.yaml untouched, and the CHANGELOG entry deliberately picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has been pulled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
glenn schrooyen
co-authored by
Claude Opus 5
Claude Opus 5
parent
587751b115
commit
0618194d9c
@@ -1,5 +1,75 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
⚠️ **THIS HAS NO VERSION AND CANNOT REACH AN INSTALL UNTIL IT GETS ONE.**
|
||||
PKG-06's and PKG-07's rule: the Supervisor keys the built image by `version:`,
|
||||
so `ha apps update` is a silent no-op on an unbumped change. This branch
|
||||
deliberately does **not** pick the number, because the right one depends on a
|
||||
fact the branch cannot observe - whether 0.9.0's image has already been built
|
||||
and pulled. If it has not, this belongs inside 0.9.0's section before it ships;
|
||||
if it has, it needs 0.10.0. Whoever deploys decides.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
|
||||
|
||||
**FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()`
|
||||
was complete and checked, and so were the consumers written for it -
|
||||
`TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and
|
||||
`arbitrage()`'s refusal 3, which drops provisional hours from the trade set
|
||||
rather than refusing the whole series. Nothing in `app/` called the fill, so no
|
||||
series ever contained a carried point and every one of those consumers was
|
||||
exercising its all-firm branch.
|
||||
|
||||
⚠️ **THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE.** `load_prices()`
|
||||
returns what the feed published and nothing widened it to the requested horizon,
|
||||
so on a dynamic site between midnight and the 14-16 h CET day-ahead publication
|
||||
the plan covered the rest of today and **nothing more** - typically 10-24 of the
|
||||
48 hours `price_horizon_h` asked for. `arbitrage()` then took `max - min` over
|
||||
that truncated window and reported a spread. The number was real, the window was
|
||||
half the one configured, and **nothing said so**: tomorrow evening's peak, the
|
||||
hour a morning charge would be sold into, was not in the series at all, so the
|
||||
spread it would have created was never compared against anything.
|
||||
|
||||
**What changes:**
|
||||
|
||||
* `app/price_history.py` (new) keeps the **published** prices of the last
|
||||
`KEEP_DAYS` Brussels days in `/data/price_history.json`, and
|
||||
`resolve_tariff()` now fills every unpublished hour of the horizon from
|
||||
yesterday's same-wall-clock hour, flagged provisional.
|
||||
* `resolve_tariff(history=...)` is **required and keyword-only**. A caller with
|
||||
no durable volume passes the explicit `NoHistory()`. A default of `None`
|
||||
meaning "skip the fill" would hand a caller who forgot the argument exactly
|
||||
the truncated horizon this card removes, silently.
|
||||
* A new status row, **Price history**, names the days held and their hour
|
||||
counts.
|
||||
|
||||
⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.**
|
||||
Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill
|
||||
widens what the optimiser can **see**, never what it may **do**. What does
|
||||
change a trade is that the filled series is windowed to `[since, until)`, so
|
||||
**hours that have already elapsed are no longer in the spread**. Before, a cheap
|
||||
night that had already gone sat in the series as the day's cheapest hour - and a
|
||||
cycle cannot buy in an hour that is past, so the trade was quietly never taken.
|
||||
`test_price_history.py` section 2 grades this by finding the cheapest firm hour
|
||||
with and without the fill and showing it **moves**.
|
||||
|
||||
⚠️ **A DEAD FEED IS STILL §11.4, NOT YESTERDAY.** The fill runs only on a
|
||||
`series` read; a feed that published nothing still degrades to the configured
|
||||
`static_profile`. The two are indistinguishable to the optimiser - both are
|
||||
100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()`
|
||||
is False - so preferring yesterday would change a documented degradation for a
|
||||
difference only the status page can see.
|
||||
`DECISIONS/FCST-04-the-day-that-is-carried.md` carries that and two more.
|
||||
|
||||
⚠️ **A BROKEN CACHE IS NOT A BROKEN CONTROLLER.** No `/data`, an unreadable
|
||||
file, a schema this build does not know, a malformed point - four states, four
|
||||
reasons, none of them a price and none of them an empty series. In every one the
|
||||
plan is still a real provider series and the controller still regulates; the
|
||||
horizon is just as short as it was before this card. **No option was added and
|
||||
`config.yaml` is untouched.**
|
||||
|
||||
## 0.9.0
|
||||
|
||||
⚠️ **MINOR FOR NEW FUNCTIONALITY, NOT FOR A BREAK - AND THAT DISTINCTION IS THE
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# FCST-04 — what "yesterday" is, and what the fill is allowed to reach
|
||||
|
||||
Three decisions, recorded because each had a defensible alternative that a later
|
||||
reader would otherwise re-open from scratch.
|
||||
|
||||
---
|
||||
|
||||
## 1. The fill runs on a `series` read only. A dead feed still goes to §11.4.
|
||||
|
||||
`fill_from_yesterday()` accepts a `no_data` read and will happily build a
|
||||
100 %-carried series from yesterday alone. `resolve_tariff()` does not let it:
|
||||
a feed that published nothing still falls to `_degrade()` and §11.4's own row —
|
||||
*"No price data → fall back to the static profile, mark the plan provisional,
|
||||
no arbitrage."*
|
||||
|
||||
**Why not prefer yesterday's real prices to a coarse day/night profile?**
|
||||
Because it buys nothing the optimiser can use, and it silently replaces a
|
||||
fallback the operator configured.
|
||||
|
||||
Measured, not assumed: a fully-carried series is 100 % provisional, so
|
||||
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on §8.3's
|
||||
firm-pair rung, and `cheap_now()` returns False because `firm_points()` is
|
||||
empty. The §11.4 static fallback produces *exactly* those two answers. The two
|
||||
paths differ only in which numbers the status page shows.
|
||||
|
||||
So the choice is between two identical control behaviours, and the tie-breaker
|
||||
is that `static_profile` is a documented option an operator fills in **for this
|
||||
case**. Overriding it for a cosmetic difference is a behaviour change nobody
|
||||
asked for.
|
||||
|
||||
⚠️ **The consequence, stated so it is not discovered later as a bug:**
|
||||
`fill_from_yesterday()`'s `no_data` branch is not reached from production.
|
||||
It stays checked in `test_price_forecast.py`. Turning it on is a product
|
||||
decision — it would want a rule about how old a carried profile may be before it
|
||||
stops being offered at all — and not a wiring one.
|
||||
|
||||
---
|
||||
|
||||
## 2. "Yesterday" is a date, not the newest record.
|
||||
|
||||
`newest()` is the obvious accessor and it is the one that decays. An add-on down
|
||||
from Friday to Monday would answer Friday's prices to a fill whose only
|
||||
vocabulary is `provisional: true` — a flag that says "this is a guess", never
|
||||
"this guess is three days old". Nothing downstream can tell the difference and
|
||||
nothing reports it.
|
||||
|
||||
So `yesterday(stamp)` computes the Brussels date before `stamp`'s Brussels date
|
||||
and looks up **that** record. Anything else is `no_data` with a reason naming
|
||||
both the date wanted and the dates held.
|
||||
|
||||
The cost is real and accepted: after an outage of more than a day, the first
|
||||
morning back has no fill. That is the honest state — there is no yesterday —
|
||||
and the horizon is short for one morning rather than wrong for several.
|
||||
|
||||
---
|
||||
|
||||
## 3. What is recorded is the provider's read, and the mechanism is ordering.
|
||||
|
||||
Feeding the filled series back into the store is how this rots: day 2 carries
|
||||
day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead
|
||||
feed the "yesterday" being carried is a fossil of the last real publication.
|
||||
Every point still carries `provisional=True`, so the **flag never lies** — what
|
||||
decays is the number, behind a flag that is telling the truth.
|
||||
|
||||
⚠️ **No value-level check can catch that**, and claiming one would be worse than
|
||||
claiming none. A carried point and a published-provisional point (`eneco_eddy`'s
|
||||
current quarter, §9.2) are the same three fields carrying the same flag. The
|
||||
difference is provenance, and provenance is not in the value.
|
||||
|
||||
The defence is therefore an **ordering**, not a validation:
|
||||
|
||||
- `carry()` is the only public entry point. It records its argument, then
|
||||
returns the fill.
|
||||
- The filled series exists only as that return value. The object never sees it
|
||||
again.
|
||||
- `_remember()` is private and called from exactly one place.
|
||||
- `test_price_history.py` walks `app/` and fails if any other module names
|
||||
`_remember`, and reads `carry()`'s AST to confirm the record happens before
|
||||
the lookup — so a later edit that reverses them is red rather than plausible.
|
||||
|
||||
That ordering has a second benefit worth keeping: a feed that returns yesterday
|
||||
alongside today refreshes yesterday's record from live data *before* it is
|
||||
consumed as the profile.
|
||||
|
||||
---
|
||||
|
||||
## What this ticket does not claim
|
||||
|
||||
It does not make a trade possible that yesterday's prices justify. Carried hours
|
||||
are dropped from the trade set by §8.3, and that is not a limitation to be fixed
|
||||
later — it is the rule.
|
||||
|
||||
The one change that **does** move energy came from the windowing, not the
|
||||
carrying: the filled series covers `[since, until)`, so hours that have already
|
||||
elapsed are no longer in the spread. A cheap night that has gone used to be the
|
||||
series' cheapest hour, and a cycle cannot buy in an hour that is past — so the
|
||||
trade was silently never taken. `test_price_history.py` section 2 grades that by
|
||||
finding the cheapest firm hour with and without the fill and showing it moves.
|
||||
@@ -1424,6 +1424,40 @@ distinct state from "prices, all of them zero": the plan is marked provisional
|
||||
and arbitrage is disabled, per the degradation matrix. It never falls back to a
|
||||
price of zero, which would read as free electricity.
|
||||
|
||||
#### The hours the feed has not published yet (FCST-04)
|
||||
|
||||
A day-ahead auction publishes tomorrow's prices once a day, at **14–16 h CET**.
|
||||
Before that, the feed is serving today and nothing else — so a `price_horizon_h`
|
||||
of 48 asks for 48 hours and gets back the ten or fifteen that are left in the
|
||||
day. Until FCST-04 those other hours were simply **absent** from the plan: not
|
||||
priced badly, not flagged, just missing, and nothing said so.
|
||||
|
||||
Now the add-on keeps the **published prices of the last few days** in
|
||||
`/data/price_history.json` and fills every unpublished hour of the horizon with
|
||||
yesterday's price for the same Brussels wall-clock hour, flagged
|
||||
`provisional: true`. Two things follow, and the second one matters more:
|
||||
|
||||
- **The plan says it is guessing.** *Tariff* on the status page reads
|
||||
`provisional` through the morning, because half its hours are. Before, a
|
||||
half-length plan looked completely settled.
|
||||
- **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional
|
||||
price, so arbitrage drops those hours from the trade set entirely — it needs a
|
||||
buy hour and a sell hour that were *both* published. Carrying a price forward
|
||||
widens what the optimiser can **see**; it never widens what it may **do**.
|
||||
|
||||
The plan is also now cut to the horizon you asked for, which means **hours that
|
||||
have already elapsed are no longer in it**. That is a real change to arbitrage:
|
||||
a cheap night that has already gone used to sit in the series as the cheapest
|
||||
hour of the day, and a cycle can never buy in an hour that is past — so the
|
||||
trade was quietly never taken. The spread is now taken over hours still ahead.
|
||||
|
||||
A new status row, **Price history**, says which days are held and how many hours
|
||||
each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon
|
||||
is as short as the feed makes it, and every other guarantee is unchanged — a
|
||||
cache that cannot answer is not allowed to be a controller that cannot regulate.
|
||||
The store is rebuilt by the next successful fetch; on a brand-new install the
|
||||
first fill is possible the day after the first refresh.
|
||||
|
||||
**Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is
|
||||
often near zero, which by itself strongly favours self-consumption over export.
|
||||
It is a single number rather than a curve because no supported supplier
|
||||
|
||||
@@ -68,6 +68,7 @@ from . import peakguard
|
||||
from . import packshare
|
||||
from .p1 import P1Ingest, build_source, is_enabled
|
||||
from .price import Uncertainty, price_option
|
||||
from .price_history import PriceHistory
|
||||
from .static_tariff import resolve_tariff, static_status
|
||||
from .tariff import PeakStore, PeakTracker
|
||||
from .store import migrate_maint_keys, migrate_unit_keys
|
||||
@@ -165,7 +166,8 @@ def pack_energy_of(inv):
|
||||
|
||||
class Controller:
|
||||
def __init__(self, opts: dict, hass: HomeAssistant, store, mqtt_pub, *, stop,
|
||||
peak_db_path: str = "/data/tariff.db"):
|
||||
peak_db_path: str = "/data/tariff.db",
|
||||
price_history_path: str = "/data/price_history.json"):
|
||||
self.o = opts
|
||||
self.hass = hass
|
||||
self.store = store
|
||||
@@ -582,6 +584,27 @@ class Controller:
|
||||
self.peak.restore(self.p1.averager)
|
||||
_LOG.info("capacity-tariff peak store: %s", self.peak.restore_state)
|
||||
|
||||
# FCST-04. The §8.3 carry-forward store: the published prices of the
|
||||
# last few Brussels days, so the hours a day-ahead feed has not printed
|
||||
# yet can be filled from yesterday's profile and FLAGGED, instead of
|
||||
# being absent from the series entirely.
|
||||
#
|
||||
# ⚠️ IT IS AN INSTRUMENT'S BACKING FILE, NOT A STRATEGY. It commands
|
||||
# nothing and claims nothing; what it changes is how many hours the
|
||||
# plan covers. Before this, a dynamic site between midnight and the
|
||||
# 14-16 h CET publication had a series covering only the rest of today
|
||||
# - typically 10-24 of the 48 hours `price_horizon_h` asked for - and
|
||||
# `TariffPlan.arbitrage()` took its spread over that truncated window
|
||||
# without anything reporting that it had. Every carried hour is
|
||||
# provisional, so refusal 3 drops it from the trade set: this widens
|
||||
# what the optimiser can SEE, never what it may act on.
|
||||
#
|
||||
# ⚠️ A BROKEN OR ABSENT STORE IS NOT AN ERROR HERE. It answers `no_data`,
|
||||
# the fill carries nothing, and the plan is exactly the pre-FCST-04
|
||||
# one - short, but honest and still refusing on the same rules.
|
||||
self.price_history = PriceHistory(price_history_path)
|
||||
_LOG.info("§8.3 price history: %s", self.price_history.words)
|
||||
|
||||
# SAFETY-07. The guard TARIFF-01 was built for. It is stateless by
|
||||
# design (peakguard.py holds no history between calls), so the only
|
||||
# state here is what an EDGE needs: what the last cycle concluded, so
|
||||
@@ -3455,7 +3478,8 @@ class Controller:
|
||||
now = datetime.now(timezone.utc)
|
||||
try:
|
||||
plan = await resolve_tariff(
|
||||
self.o, now, now + timedelta(hours=horizon), now=now)
|
||||
self.o, now, now + timedelta(hours=horizon), now=now,
|
||||
history=self.price_history)
|
||||
self.tariff_plan, self.tariff_at = plan, now
|
||||
_LOG.info("tariff refreshed: %s", plan.words)
|
||||
except Exception as err: # noqa: BLE001
|
||||
@@ -4304,6 +4328,13 @@ class Controller:
|
||||
("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")),
|
||||
("Cheap window", "yes" if self.cheap else "no"),
|
||||
("Tariff", self.tariff_words),
|
||||
# FCST-04. WHAT §8.3 HAS TO CARRY FORWARD, which is the difference
|
||||
# between a 48-hour plan and a 12-hour one on a weekday morning.
|
||||
# It is its own row rather than a clause of "Tariff" because it
|
||||
# answers a different question: "Tariff" says what the plan IS, and
|
||||
# this says what the plan could still be filled from if the feed is
|
||||
# late again tomorrow.
|
||||
("Price history", self.price_history.words),
|
||||
# OPT-04. The strategy's own sentence, not a re-derivation of it -
|
||||
# every refusal it can make already carries a reason a human can
|
||||
# act on, and "Arbitrage: no" would throw all of them away.
|
||||
|
||||
@@ -0,0 +1,433 @@
|
||||
"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach.
|
||||
|
||||
WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped
|
||||
`price_forecast.fill_from_yesterday()` complete and checked, and shipped the
|
||||
consumers that were written to expect it - `TariffPlan.firm_points()`,
|
||||
`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3,
|
||||
which drops provisional hours from the trade set instead of refusing the whole
|
||||
series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in
|
||||
`app/` referenced `fill_from_yesterday`, so no series ever contained a carried
|
||||
point and every one of those consumers was exercising its all-firm branch.
|
||||
|
||||
The defect that produced is not an error anywhere. `load_prices()` returns
|
||||
whatever the feed published and nothing widens it to the requested horizon, so
|
||||
on a dynamic site between midnight and the 14-16 h CET day-ahead publication
|
||||
the series covers the rest of TODAY and nothing more - typically 10-24 of the
|
||||
48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes
|
||||
`max - min` over that truncated window and reports a spread. The number is
|
||||
real, the window is half the one the operator configured, and NOTHING SAYS SO:
|
||||
tomorrow evening's peak - the hour a morning charge would be sold into - is
|
||||
simply not in the series, so the spread it would have created cannot clear a
|
||||
threshold it was never compared against. A missing hour is not a wrong price;
|
||||
it is an hour the optimiser never learns exists.
|
||||
|
||||
So this module is the half FCST-03 left out, and it is deliberately the SMALL
|
||||
half: the fill's arithmetic, its DST rule and its provenance audit all stay in
|
||||
`price_forecast.py`, untouched. What lives here is the one thing that
|
||||
arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART.
|
||||
|
||||
|
||||
WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK
|
||||
======================================================
|
||||
An in-memory "the last full day I saw" costs nothing and is wrong exactly when
|
||||
it matters. The window the fill exists for is midnight -> publication; an
|
||||
add-on restarted at 08:00 - a config change, an HA update, a power cut - would
|
||||
come up with no yesterday at all and spend that whole window on the truncated
|
||||
series, which is the pre-FCST-03 behaviour arriving silently on the mornings
|
||||
after an operator touched something. The fill's own value is a function of
|
||||
surviving a restart, so the day it carries has to.
|
||||
|
||||
⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and
|
||||
SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every
|
||||
write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a
|
||||
price cache into it would put 48 extra whole-file rewrites a day through the
|
||||
file whose corruption reads as "no maintenance ever ran" and "never tripped".
|
||||
The cache is worth having; it is not worth putting anywhere near that. Separate
|
||||
file, separate blast radius.
|
||||
|
||||
⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its
|
||||
reason applies unchanged: `/data` is mounted by the Supervisor and always
|
||||
exists on a real install, so `makedirs` here could only ever succeed where the
|
||||
durable volume is MISSING - manufacturing a cache on the container's ephemeral
|
||||
layer that writes fine, reads fine, and is thrown away on the next restart.
|
||||
That is a yesterday that silently forgets, which is worse than one that says it
|
||||
cannot answer.
|
||||
|
||||
|
||||
THE FOUR RULES THIS FILE IS SHAPED BY
|
||||
=====================================
|
||||
|
||||
1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED.
|
||||
Feeding a filled series back in is the defect that makes the whole mechanism
|
||||
rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and
|
||||
after a week of a dead feed the "yesterday" being carried forward is a
|
||||
fossil of the last real publication with nothing anywhere reporting its age.
|
||||
Every point still carries `provisional=True`, so the flag never lies - what
|
||||
decays is the NUMBER, silently, behind a flag that is telling the truth.
|
||||
|
||||
⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a
|
||||
guarantee that would not hold. A carried point and a published-provisional
|
||||
point (`eneco_eddy`'s current quarter, §9.2) are the same three fields with
|
||||
the same flag; the difference is provenance, and provenance is not in the
|
||||
value. So the mechanism is ORDERING, not validation: `carry()` is the only
|
||||
public entry point, it remembers the argument and returns the fill, and the
|
||||
filled series exists only as a return value that `carry()` itself never
|
||||
sees again. `_remember()` is private, is called from exactly one place, and
|
||||
`test_price_history.py` walks the tree and fails if any module outside this
|
||||
file names it.
|
||||
|
||||
2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE".
|
||||
`newest()` is the natural accessor and it is the one that decays: an add-on
|
||||
down from Friday to Monday would answer Friday's prices to a fill that flags
|
||||
them as one day old. The lookup is BY DATE - the Brussels day before the
|
||||
Brussels day of the instant asked about - and a record for any other date is
|
||||
NO_RECORD with a reason naming both dates. A three-day-old profile is not a
|
||||
worse yesterday; it is not a yesterday.
|
||||
|
||||
3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE.
|
||||
`yesterday()` returns a `PriceRead`: a `series` when a full record exists,
|
||||
`no_data(reason)` for every other outcome - no file, no record for that
|
||||
date, an unreadable file, a record that parses to nothing. There is no
|
||||
empty-series answer and no `.get(..., [])`-shaped accessor, for
|
||||
`price.py`'s reason: an empty series is how "the cache is broken" gets to
|
||||
wear the shape of "yesterday was free". `fill_from_yesterday()` takes a
|
||||
`PriceRead` and handles `no_data` on its own, so the degraded case needs no
|
||||
branch at the call site - it fills nothing and the horizon stays short,
|
||||
which is exactly the behaviour that shipped before this ticket.
|
||||
|
||||
4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT.
|
||||
A feed that serves only the REMAINING hours of the day (a legitimate shape -
|
||||
nothing in the plugin contract promises a whole day) would, under
|
||||
replace-the-record, overwrite a complete 24-hour profile with the two points
|
||||
left at 22:00 and hand the next morning a yesterday with two hours in it. So
|
||||
a record is MERGED by instant: a later read updates the hours it carries and
|
||||
leaves the rest, and a revised price for an hour already held wins because
|
||||
it is the newer statement about the same hour.
|
||||
|
||||
⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing
|
||||
`hour -> price` would collapse the two 02:00s of a 25-hour October day into
|
||||
one, inventing a second and different collapsing rule beside the one
|
||||
`fill_from_yesterday()` already documents ("the first of its two 02:00 points
|
||||
is the one used, deterministically"). Storing the instants keeps both, in
|
||||
order, and lets the fill's own rule be the only one in the system.
|
||||
|
||||
WHAT THIS DOES NOT DO, ON PURPOSE
|
||||
=================================
|
||||
It does not extend the fill to a feed that published NOTHING. `resolve_tariff()`
|
||||
applies `carry()` only on a `series` read; a `no_data` read still falls to
|
||||
`_degrade()` and §11.4's row - "No price data -> fall back to the static
|
||||
profile, mark the plan provisional, no arbitrage" - which is the fallback the
|
||||
operator configured `static_profile` FOR. Preferring yesterday's feed over it
|
||||
would change a documented degradation without being asked, and it would buy
|
||||
nothing measurable: a fully-carried series is 100 % provisional, so
|
||||
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is
|
||||
False, exactly as on the §11.4 fallback. The two differ only in which numbers
|
||||
the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway,
|
||||
and that branch is therefore not reached from production - it stays checked,
|
||||
and turning it on is a product decision, not a wiring one. See
|
||||
DECISIONS/FCST-04-the-day-that-is-carried.md.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from .price import PricePoint, PriceProduct, PriceRead, brussels
|
||||
from .price_forecast import fill_from_yesterday
|
||||
|
||||
_LOG = logging.getLogger("goodwe.price.history")
|
||||
|
||||
# How many Brussels days of published prices are kept. The fill needs exactly
|
||||
# one - yesterday - and rule 2 refuses anything older, so this is not a horizon:
|
||||
# it is the number of days a record may sit unused before pruning removes it,
|
||||
# and it is small because a bigger cache would only hold days no lookup can ask
|
||||
# for. Three covers today, yesterday, and the day a fetch spanning midnight
|
||||
# writes ahead of itself.
|
||||
KEEP_DAYS = 3
|
||||
|
||||
SCHEMA = 1
|
||||
|
||||
|
||||
def _bdate(instant: datetime) -> str:
|
||||
"""The Brussels calendar date of an instant, as `YYYY-MM-DD`.
|
||||
|
||||
⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and
|
||||
a cache keyed on the UTC date would file the last 90-120 minutes of every
|
||||
day under the wrong one - so a lookup on the following morning would find a
|
||||
"yesterday" missing its own evening hours, which are the expensive ones.
|
||||
"""
|
||||
return brussels(instant).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
class PriceHistory:
|
||||
"""The published price series of the last few Brussels days, in /data.
|
||||
|
||||
⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no
|
||||
record for yesterday is `ok` and answers `no_data` - a normal state on a
|
||||
site's first day, and a different thing from a store that could not be
|
||||
opened. Both are visible: `ok`/`last_error` describe the file, and
|
||||
`yesterday()`'s reason describes the answer.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str = "/data/price_history.json"):
|
||||
self.path = path
|
||||
self.ok = False
|
||||
self.last_error: str | None = None
|
||||
self._days: dict = {}
|
||||
self._open()
|
||||
|
||||
# -- the file ------------------------------------------------------------
|
||||
def _open(self) -> None:
|
||||
parent = os.path.dirname(self.path)
|
||||
# See the module note: NOT created. A cache on the ephemeral layer is
|
||||
# one that forgets without saying so.
|
||||
if parent and not os.path.isdir(parent):
|
||||
self.last_error = f"no durable volume at {parent}"
|
||||
# INFO, not ERROR: on a dev host or in a test container there is no
|
||||
# /data and nothing is wrong.
|
||||
_LOG.info("no price history: %s - §8.3 carry-forward has nothing "
|
||||
"to carry and the horizon stays as short as the feed",
|
||||
self.last_error)
|
||||
return
|
||||
self.ok = True
|
||||
self._load()
|
||||
|
||||
def _load(self) -> None:
|
||||
try:
|
||||
with open(self.path, "r", encoding="utf-8") as fh:
|
||||
blob = json.load(fh)
|
||||
except FileNotFoundError:
|
||||
_LOG.info("no price history yet at %s - the first refresh writes "
|
||||
"one, and the fill has nothing to carry until the day "
|
||||
"after that", self.path)
|
||||
return
|
||||
except (json.JSONDecodeError, OSError) as err:
|
||||
# A corrupt cache must not stop the controller, and must not be
|
||||
# silently re-created either: the record is discarded, the reason
|
||||
# is kept, and `yesterday()` reports it rather than answering
|
||||
# "no record", which would read as a normal first day.
|
||||
self.last_error = f"unreadable ({err})"
|
||||
_LOG.warning("price history at %s is %s - starting from empty. "
|
||||
"§8.3 carry-forward has nothing to carry until the "
|
||||
"next refresh has written a full day",
|
||||
self.path, self.last_error)
|
||||
return
|
||||
if not isinstance(blob, dict) or blob.get("version") != SCHEMA:
|
||||
found = blob.get("version") if isinstance(blob, dict) else None
|
||||
self.last_error = f"schema {found!r}, not {SCHEMA}"
|
||||
_LOG.warning("price history at %s has %s - discarding it rather "
|
||||
"than guessing what its points mean",
|
||||
self.path, self.last_error)
|
||||
return
|
||||
days = blob.get("days")
|
||||
if not isinstance(days, dict):
|
||||
self.last_error = "no `days` object"
|
||||
_LOG.warning("price history at %s has %s", self.path, self.last_error)
|
||||
return
|
||||
self._days = days
|
||||
|
||||
def _save(self) -> None:
|
||||
if not self.ok:
|
||||
return
|
||||
try:
|
||||
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path))
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1)
|
||||
# Atomic: a power cut mid-write must not leave a truncated file
|
||||
# that parses as an empty day.
|
||||
os.replace(tmp, self.path)
|
||||
except OSError as err:
|
||||
self.last_error = f"could not write ({err})"
|
||||
_LOG.error("could not persist price history: %s", err)
|
||||
|
||||
# -- the one public entry point -----------------------------------------
|
||||
def carry(self, read: PriceRead, *, since: datetime, until: datetime,
|
||||
stamp: datetime) -> PriceRead:
|
||||
"""Remember what this read published, then §8.3-fill the missing hours.
|
||||
|
||||
THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is
|
||||
recorded FIRST, so a feed that hands back yesterday alongside today
|
||||
refreshes yesterday's record from live data before it is consumed as
|
||||
the profile; and the filled series - the one thing that must never be
|
||||
recorded - exists only as this method's return value, which this object
|
||||
never sees again. See rule 1.
|
||||
|
||||
Returns whatever `fill_from_yesterday()` returns: a series over
|
||||
[since, until) whose published hours are the provider's own point
|
||||
objects and whose remaining hours are carried and flagged, or
|
||||
`no_data` when neither source can produce a single hour.
|
||||
"""
|
||||
self._remember(read, stamp)
|
||||
return fill_from_yesterday(read, yesterday=self.yesterday(stamp),
|
||||
since=since, until=until, stamp=stamp)
|
||||
|
||||
# -- the halves, private because rule 1 is an ordering ------------------
|
||||
def _remember(self, read: PriceRead, stamp: datetime) -> None:
|
||||
"""File a provider read's points under their own Brussels days.
|
||||
|
||||
⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach
|
||||
this can record a filled series, which is the decay in rule 1. There is
|
||||
no value-level defence against that - a carried point and a published
|
||||
provisional point are the same three fields - so the defence is that
|
||||
nothing outside this file names it, and `test_price_history.py` checks
|
||||
the tree for that rather than trusting the underscore.
|
||||
"""
|
||||
points = read.match(series=lambda r: r.points, no_data=lambda _r: ())
|
||||
if not points:
|
||||
return
|
||||
product = read.match(series=lambda r: r.product.value,
|
||||
no_data=lambda _r: None)
|
||||
provider = read.match(series=lambda r: r.provider,
|
||||
no_data=lambda _r: None)
|
||||
|
||||
touched = set()
|
||||
for p in points:
|
||||
day = _bdate(p.start)
|
||||
rec = self._days.setdefault(
|
||||
day, {"product": product, "provider": provider, "points": []})
|
||||
# Rule 4: MERGE by instant. A later read updates the hours it
|
||||
# carries and leaves the rest; a revised price for an hour already
|
||||
# held wins, because it is the newer statement about that hour.
|
||||
held = {q["start"]: q for q in rec.get("points", [])
|
||||
if isinstance(q, dict) and "start" in q}
|
||||
held[p.start.isoformat()] = p.canonical()
|
||||
rec["points"] = sorted(held.values(), key=lambda q: q["start"])
|
||||
rec["product"], rec["provider"] = product, provider
|
||||
touched.add(day)
|
||||
|
||||
self._prune(stamp)
|
||||
self._save()
|
||||
_LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
|
||||
len(points), ", ".join(sorted(touched)), len(self._days))
|
||||
|
||||
def _prune(self, stamp: datetime) -> None:
|
||||
"""Keep the newest KEEP_DAYS records. Older ones can never be a lookup.
|
||||
|
||||
Pruning is by SORT ORDER, not by arithmetic against `stamp`: a clock
|
||||
that jumped - the failure mode WITNESS-03 spent a session on - would
|
||||
otherwise delete every record in one pass and leave the fill with
|
||||
nothing on the morning after, which is the one morning it is for.
|
||||
"""
|
||||
if len(self._days) <= KEEP_DAYS:
|
||||
return
|
||||
for day in sorted(self._days)[:-KEEP_DAYS]:
|
||||
del self._days[day]
|
||||
|
||||
def yesterday(self, stamp: datetime) -> PriceRead:
|
||||
"""The published series of the Brussels day BEFORE `stamp`'s, or no_data.
|
||||
|
||||
⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would
|
||||
answer Friday's prices to a Monday fill, and the fill would flag them as
|
||||
one day old because that is the only thing its flag can say.
|
||||
"""
|
||||
if stamp.tzinfo is None or stamp.utcoffset() is None:
|
||||
raise ValueError(
|
||||
f"yesterday({stamp!r}): a naive datetime is not an instant, "
|
||||
f"and 'which Brussels day was yesterday' is exactly the "
|
||||
f"question a wall-clock reading cannot answer.")
|
||||
want = _bdate(brussels(stamp) - timedelta(days=1))
|
||||
if self.last_error and not self._days:
|
||||
return PriceRead.no_data(
|
||||
f"the price history at {self.path} is {self.last_error}, so "
|
||||
f"§8.3 has no profile to carry forward for {want}; the plan "
|
||||
f"covers only the hours the feed published")
|
||||
rec = self._days.get(want)
|
||||
if rec is None:
|
||||
have = ", ".join(sorted(self._days)) or "no days at all"
|
||||
return PriceRead.no_data(
|
||||
f"no published prices recorded for {want} (held: {have}), so "
|
||||
f"§8.3 has no profile to carry forward. A record for another "
|
||||
f"day is not a yesterday")
|
||||
points, bad = _parse_points(rec.get("points"))
|
||||
if not points:
|
||||
why = bad or "it is empty"
|
||||
return PriceRead.no_data(
|
||||
f"the {want} record holds no usable point ({why}), and an "
|
||||
f"empty series is not a free day")
|
||||
try:
|
||||
product = PriceProduct(rec.get("product"))
|
||||
except ValueError:
|
||||
return PriceRead.no_data(
|
||||
f"the {want} record names product {rec.get('product')!r}, "
|
||||
f"which is not a PriceProduct. The optimiser switches on that "
|
||||
f"enum, so a record that cannot name one cannot be a series")
|
||||
if bad:
|
||||
_LOG.warning("price history: %s in the %s record were dropped; "
|
||||
"%d point(s) carried forward", bad, want, len(points))
|
||||
return PriceRead.series(points, product,
|
||||
str(rec.get("provider") or "price_history"),
|
||||
stamp)
|
||||
|
||||
# -- observability -------------------------------------------------------
|
||||
@property
|
||||
def words(self) -> str:
|
||||
"""One line for the status page. Says what is held, or why nothing is."""
|
||||
if not self.ok:
|
||||
return f"unavailable - {self.last_error}"
|
||||
if not self._days:
|
||||
why = f" - {self.last_error}" if self.last_error else ""
|
||||
return f"empty{why} - nothing to carry forward yet"
|
||||
return "; ".join(f"{d} ({len(self._days[d].get('points') or ())} h)"
|
||||
for d in sorted(self._days))
|
||||
|
||||
|
||||
def _parse_points(raw) -> tuple:
|
||||
"""Canonical dicts -> PricePoints, and a phrase naming what was dropped.
|
||||
|
||||
⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported.
|
||||
`float(row.get("eur_per_kwh", 0))` is the shape of the whole family of
|
||||
defects this repo has been removing: a cache that lost a field would answer
|
||||
"that hour was free" to a fill that flags it as an ordinary guess.
|
||||
"""
|
||||
if not isinstance(raw, list):
|
||||
return (), "the `points` field is not a list"
|
||||
out, dropped = [], 0
|
||||
for row in raw:
|
||||
if not isinstance(row, dict):
|
||||
dropped += 1
|
||||
continue
|
||||
try:
|
||||
start = datetime.fromisoformat(row["start"])
|
||||
value = float(row["eur_per_kwh"])
|
||||
flag = row["provisional"]
|
||||
except (KeyError, TypeError, ValueError):
|
||||
dropped += 1
|
||||
continue
|
||||
if start.tzinfo is None or start.utcoffset() is None \
|
||||
or not isinstance(flag, bool):
|
||||
dropped += 1
|
||||
continue
|
||||
out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag))
|
||||
out.sort(key=lambda p: p.start.astimezone(timezone.utc))
|
||||
return tuple(out), (f"{dropped} malformed point(s)" if dropped else "")
|
||||
|
||||
|
||||
class NoHistory:
|
||||
"""The explicit "this caller keeps no history" - a NAME, not a `None`.
|
||||
|
||||
⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip
|
||||
the fill" is the shape this repo keeps removing: a caller who forgets the
|
||||
argument gets the truncated-horizon behaviour of the bug this ticket is
|
||||
about, silently, and the code reads as though the fill is happening. With
|
||||
no default there is nothing to forget, and a caller with no durable volume
|
||||
- a check, a replay harness - has to say `NoHistory()` and be visible in
|
||||
the diff.
|
||||
|
||||
It carries out the ordering faithfully: it records nothing and carries
|
||||
nothing, so `carry()` returns the provider's read unchanged and the plan is
|
||||
exactly what it was before FCST-04.
|
||||
"""
|
||||
|
||||
ok = False
|
||||
last_error = "no history is kept by this caller"
|
||||
words = "not kept"
|
||||
|
||||
def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead:
|
||||
return read
|
||||
|
||||
def yesterday(self, stamp: datetime) -> PriceRead:
|
||||
return PriceRead.no_data(
|
||||
"this caller keeps no price history (NoHistory), so §8.3 has no "
|
||||
"profile to carry forward")
|
||||
@@ -1019,7 +1019,7 @@ def static_status(opts: dict, now: datetime):
|
||||
|
||||
|
||||
async def resolve_tariff(opts: dict, since: datetime, until: datetime,
|
||||
transport=None, now=None) -> TariffPlan:
|
||||
transport=None, now=None, *, history) -> TariffPlan:
|
||||
"""The full §9.2 + §11.4 resolution. MAY PERFORM I/O - see `static_plan`.
|
||||
|
||||
⚠️ NOT ON THE 30 s TICK, and that is a deliberate boundary rather than an
|
||||
@@ -1033,9 +1033,34 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime,
|
||||
price_provider off + profile -> the CONFIGURED product (§9.2 rows 3-4)
|
||||
price_provider off + none -> no prices at all (§11.4, nothing to
|
||||
fall back to)
|
||||
provider series -> the feed (§9.2 rows 1-2)
|
||||
provider series -> the feed, §8.3-FILLED (§9.2 rows 1-2)
|
||||
provider no_data + profile -> §11.4 FALLBACK, provisional, no arbitrage
|
||||
provider no_data + none -> no prices at all
|
||||
|
||||
⚠️ `history` IS REQUIRED, KEYWORD-ONLY, AND HAS NO DEFAULT (FCST-04). It is
|
||||
the §8.3 carry-forward store - `price_history.PriceHistory`, or the explicit
|
||||
`price_history.NoHistory()` for a caller that keeps none. A default of
|
||||
`None` meaning "skip the fill" would give a caller who forgot the argument
|
||||
the TRUNCATED HORIZON this ticket exists to remove, silently, while the code
|
||||
read as though the fill were happening. There is nothing to forget now.
|
||||
|
||||
⚠️ AND THE FILL RUNS ONLY ON A `series` READ. A feed that published nothing
|
||||
still falls to `_degrade()` and §11.4's own row - the static profile the
|
||||
operator configured FOR that case - rather than to yesterday's feed. The two
|
||||
are indistinguishable to the optimiser (both are 100 % provisional, so
|
||||
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()`
|
||||
is False), so preferring yesterday would change a documented degradation for
|
||||
a difference only the status page can see. See price_history.py's closing
|
||||
note and DECISIONS/FCST-04-the-day-that-is-carried.md.
|
||||
|
||||
⚠️ THE FILLED SERIES IS WINDOWED TO [since, until), AND THAT IS A CHANGE.
|
||||
`load_prices()` returns whatever the feed sent, past hours included, and
|
||||
`TariffPlan.arbitrage()` took its spread over all of it - so an hour that
|
||||
had already gone could be the cheapest hour of a cycle nothing can now buy
|
||||
in. `fill_from_yesterday()` walks the requested horizon and emits exactly
|
||||
that, so the spread is over hours that are still ahead. Both directions of
|
||||
the difference are real: the window gains the unpublished hours it was
|
||||
missing and loses the elapsed ones it should never have had.
|
||||
"""
|
||||
stamp = now or datetime.now(timezone.utc)
|
||||
pid = str(price_option(opts, "price_provider") or "off").strip()
|
||||
@@ -1044,6 +1069,20 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime,
|
||||
|
||||
read = await load_prices(opts, since, until, transport)
|
||||
return read.match(
|
||||
series=lambda r: _provider_plan(opts, r, since, until, stamp, history),
|
||||
no_data=lambda r: _degrade(opts, r, since, until, stamp))
|
||||
|
||||
|
||||
def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan:
|
||||
"""§8.3: record what the feed published, carry yesterday into the gaps.
|
||||
|
||||
The fill can itself return `no_data` - a feed whose points all fall outside
|
||||
the horizon, with nothing in the history to fill it from - and that is the
|
||||
§11.4 row arrived at one step later, so it degrades exactly as a failed
|
||||
fetch does rather than being special-cased here.
|
||||
"""
|
||||
filled = history.carry(read, since=since, until=until, stamp=stamp)
|
||||
return filled.match(
|
||||
series=lambda r: TariffPlan(r, _quiet_profile(opts), "provider"),
|
||||
no_data=lambda r: _degrade(opts, r, since, until, stamp))
|
||||
|
||||
|
||||
@@ -0,0 +1,734 @@
|
||||
"""Runnable check for FCST-04 - the day the fill carries. `python3 test_price_history.py`
|
||||
|
||||
WHAT THIS FILE IS DEFENDING. FCST-03 shipped `fill_from_yesterday()` and the
|
||||
consumers written for it, and shipped NO CALLER. This file grades the caller:
|
||||
that a partially-published day-ahead feed now produces a plan covering the whole
|
||||
requested horizon, that the day it carries survives a restart, and that the
|
||||
cache cannot decay by being fed its own output.
|
||||
|
||||
AC1 THE HOLE IS CLOSED. `resolve_tariff()` on a feed that has published 10
|
||||
of 48 hours returns a series covering 48, the 38 unpublished hours are
|
||||
provisional and the 10 published ones are the provider's own objects,
|
||||
unchanged and unflagged. Graded against the same call with
|
||||
`NoHistory()`, which is the pre-FCST-04 behaviour, so the difference is
|
||||
MEASURED rather than described.
|
||||
AC2 THE WINDOW STOPPED INCLUDING THE PAST, which is the half of this change
|
||||
that moves energy. `load_prices()` returns whatever the feed sent, so at
|
||||
09:00 a spread was taken over hours that had already elapsed and the
|
||||
"cheapest hour" could be one no cycle can buy in - the trade is then
|
||||
silently never taken. The filled series is windowed to [since, until),
|
||||
so the firm cheapest hour is a hour still ahead. Graded by finding the
|
||||
cheapest firm hour both ways and showing it MOVES.
|
||||
AC3 THE CACHE CANNOT DECAY. What is recorded is the provider's read and
|
||||
never the filled series; if it were, day 2 would carry day 1's guesses
|
||||
and day 3 would carry day 2's carry of them, behind a `provisional` flag
|
||||
that stays true while the NUMBER rots. Graded by driving `carry()` twice
|
||||
and reading the file back.
|
||||
AC4 A STORE THAT CANNOT ANSWER SAYS SO. No file, no record for yesterday, an
|
||||
unreadable file, a record whose points are malformed - four different
|
||||
reasons, none of them a price, none of them an empty series, and the
|
||||
controller keeps its pre-FCST-04 plan through all four.
|
||||
|
||||
THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
|
||||
|
||||
* EVERY VERDICT IS DERIVED. The carried/published split is re-derived here
|
||||
from the set of hours the fixture published, by different arithmetic than
|
||||
app/ uses. The horizon growth is a subtraction of two measured counts, not
|
||||
a constant. The cheapest-hour move is found by looking, not asserted.
|
||||
|
||||
* NO STIMULUS IS EVALUATED WHERE THE QUANTITY IT GRADES IS DEGENERATE:
|
||||
- YESTERDAY AND TODAY MUST PRICE EVERY LOCAL HOUR DIFFERENTLY, or a fill
|
||||
that carried the wrong day is invisible. Asserted first, as a check.
|
||||
- THE PUBLISHED WINDOW MUST BE A STRICT SUBSET OF THE HORIZON, or "the
|
||||
fill widened it" and "the fill did nothing" are the same observation.
|
||||
- THE ELAPSED HOURS MUST CONTAIN THE CHEAPEST HOUR OF THE WHOLE FEED, or
|
||||
AC2 grades nothing: the windowed and unwindowed cheapest hour would be
|
||||
the same point either way. This is also the realistic shape of the
|
||||
danger - a cheap night that has already gone, holding a battery out of
|
||||
a trade all morning.
|
||||
- THE MERGE FIXTURE MUST REVISE A PRICE TO A DIFFERENT NUMBER, or merge
|
||||
and replace are indistinguishable.
|
||||
- THE PRUNE FIXTURE MUST HOLD MORE DAYS THAN KEEP_DAYS, or pruning is not
|
||||
exercised at all.
|
||||
|
||||
* A FAILED READ IS NOT AN OBSERVATION. Every degraded store is graded on the
|
||||
TYPE it answers with (`no_data` and its reason), never on a falsy value,
|
||||
and the malformed-point case is checked for the specific defect - a price
|
||||
defaulted to 0.0 - rather than for a count.
|
||||
|
||||
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
|
||||
`tools/mutation_run.sh`, which does. See MUTATION.md and TEST-15.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import asyncio
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app import price_history as _ph
|
||||
from app.price import (
|
||||
PricePoint, PriceProduct, PriceRead, brussels,
|
||||
)
|
||||
from app.price_history import KEEP_DAYS, NoHistory, PriceHistory, SCHEMA
|
||||
from app.static_tariff import resolve_tariff
|
||||
|
||||
UTC = timezone.utc
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
fails = []
|
||||
_tmpdirs = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(f" {'ok ' if cond else 'FAIL'} {name}")
|
||||
if not cond:
|
||||
fails.append(name)
|
||||
|
||||
|
||||
def why(read):
|
||||
"""A read's reason, or "" when it is a series.
|
||||
|
||||
⚠️ NOT `read.reason`. On a `_Series` that attribute RAISES `Absence` by
|
||||
design, so a mutation that makes a refusing accessor answer a series would
|
||||
take this file down with a traceback instead of reporting a FAIL - and
|
||||
MUTATION.md's rule is that "it was refused" and "it broke on the way to
|
||||
being refused" must not grade the same.
|
||||
"""
|
||||
return read.match(series=lambda _r: "", no_data=lambda r: r.reason)
|
||||
|
||||
|
||||
def raises(fn, exc):
|
||||
try:
|
||||
fn()
|
||||
except exc:
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def run(coro):
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
def tmpdir():
|
||||
d = tempfile.mkdtemp(prefix="fcst04-")
|
||||
_tmpdirs.append(d)
|
||||
return d
|
||||
|
||||
|
||||
def store():
|
||||
"""A history on a fresh directory that looks durable."""
|
||||
return PriceHistory(os.path.join(tmpdir(), "price_history.json"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The fixtures.
|
||||
#
|
||||
# ⚠️ A DAY IS A BRUSSELS DAY, BUILT BY SEARCH. The first draft of this file made
|
||||
# a day 24 UTC hours long and every count in it came out one short: in January
|
||||
# Belgium is UTC+1, so 23:00 UTC already belongs to tomorrow, and a "24-hour
|
||||
# day" built by adding hours to midnight UTC is 23 hours of one Brussels day
|
||||
# plus one hour of the next. That is the same class of error the module is
|
||||
# keyed to avoid, so the fixture must not contain it - and it is found by
|
||||
# LOOKING at each instant's Brussels date rather than by adding an offset a
|
||||
# helper would have to be right about.
|
||||
#
|
||||
# ⚠️ THE INSTANT IS 09:00 BRUSSELS ON A WEEKDAY, WHICH IS THE WHOLE POINT. The
|
||||
# day-ahead auction publishes at 14-16 h CET, so at this instant a healthy feed
|
||||
# is serving TODAY IN FULL and nothing of tomorrow - the window this ticket
|
||||
# exists for. A fixture at 20:00 would have both days and would grade nothing.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
TODAY_D = "2026-01-15" # a Thursday; Belgium on CET
|
||||
YDAY_D = "2026-01-14"
|
||||
HORIZON_H = 48
|
||||
|
||||
|
||||
def bday_hours(bdate: str) -> list:
|
||||
"""The UTC instants whose BRUSSELS date is `bdate`, one per local hour."""
|
||||
probe = datetime.fromisoformat(bdate + "T00:00:00+00:00") - timedelta(hours=3)
|
||||
out = []
|
||||
for n in range(30):
|
||||
u = probe + timedelta(hours=n)
|
||||
if brussels(u).strftime("%Y-%m-%d") == bdate:
|
||||
out.append(u)
|
||||
return out
|
||||
|
||||
|
||||
TODAY_H = bday_hours(TODAY_D)
|
||||
YDAY_H = bday_hours(YDAY_D)
|
||||
|
||||
|
||||
def at_local(bdate: str, hh: int) -> datetime:
|
||||
"""The UTC instant whose Brussels wall clock reads hh:00 on `bdate`."""
|
||||
return next(u for u in bday_hours(bdate) if brussels(u).hour == hh)
|
||||
|
||||
|
||||
NOW = at_local(TODAY_D, 9)
|
||||
|
||||
# Today's feed, in EUR/MWh, keyed by BRUSSELS local hour and all distinct.
|
||||
# ⚠️ THE CHEAPEST HOUR IS 03:00 LOCAL - ALREADY ELAPSED AT 09:00 - and that is
|
||||
# AC2's stimulus, asserted below rather than assumed.
|
||||
TODAY_MWH = {h: 120.0 + h for h in range(24)}
|
||||
TODAY_MWH[3] = 5.0 # the cheap night that has already gone
|
||||
TODAY_MWH[19] = 260.0 # the dear evening still ahead
|
||||
|
||||
# Yesterday's published prices, DIFFERENT AT EVERY LOCAL HOUR from today's, so
|
||||
# a fill that carried the wrong day is visible.
|
||||
YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)}
|
||||
|
||||
MARKUP, TAXES = 0.03, 0.06
|
||||
|
||||
|
||||
def allin(mwh):
|
||||
"""EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces."""
|
||||
return round(mwh * 0.001 + MARKUP + TAXES, 6)
|
||||
|
||||
|
||||
def feed_body(instants, table):
|
||||
return json.dumps({"data": [
|
||||
{"start": u.isoformat(), "price_eur_mwh": table[brussels(u).hour]}
|
||||
for u in instants]}).encode()
|
||||
|
||||
|
||||
def transport_for(body):
|
||||
async def _t(url, timeout_s):
|
||||
return body
|
||||
return _t
|
||||
|
||||
|
||||
CFG = {"price_provider": "generic_dayahead",
|
||||
"price_feed_url": "https://api.energy-charts.info/price?bzn=BE",
|
||||
"price_timeout_s": 1,
|
||||
"price_markup_eur_kwh": MARKUP,
|
||||
"price_taxes_eur_kwh": TAXES,
|
||||
"static_profile": ""}
|
||||
|
||||
|
||||
def series_of(instants, table, product=PriceProduct.DAY_AHEAD,
|
||||
provider="generic_dayahead", stamp=None):
|
||||
"""A PriceRead.series built directly - the shape `load_prices()` returns."""
|
||||
return PriceRead.series(
|
||||
[PricePoint(start=brussels(u),
|
||||
eur_per_kwh=allin(table[brussels(u).hour]),
|
||||
provisional=False)
|
||||
for u in instants],
|
||||
product, provider, stamp or NOW)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print("0. the arithmetic this file grades with must not be degenerate")
|
||||
|
||||
check(f"a Brussels day is 24 hours long in the fixture helper, on both fixture "
|
||||
f"days ({len(TODAY_H)}, {len(YDAY_H)}) - a 23-hour 'day' built by adding "
|
||||
f"to midnight UTC is what made every count below come out one short",
|
||||
len(TODAY_H) == 24 and len(YDAY_H) == 24
|
||||
and {brussels(u).hour for u in TODAY_H} == set(range(24)))
|
||||
|
||||
# The feed serves TODAY IN FULL - which is what a day-ahead feed does - so the
|
||||
# hours it publishes that are still AHEAD are a strict subset of the horizon.
|
||||
_ahead = [u for u in TODAY_H if u >= NOW]
|
||||
check(f"at 09:00 the hours the feed published that have NOT elapsed are a "
|
||||
f"STRICT SUBSET of the {HORIZON_H} h horizon ({len(_ahead)} of "
|
||||
f"{HORIZON_H}), so 'the fill widened the plan' and 'the fill did "
|
||||
f"nothing' are different observations",
|
||||
0 < len(_ahead) < HORIZON_H)
|
||||
|
||||
check("yesterday prices EVERY local hour differently from today, so a fill "
|
||||
"that carried the wrong day is visible in the numbers",
|
||||
all(TODAY_MWH[h] != YESTERDAY_MWH[h] for h in range(24)))
|
||||
|
||||
check("...and no fixture price is zero, so 'dropped the price' is visible too",
|
||||
all(TODAY_MWH.values()) and all(YESTERDAY_MWH.values()))
|
||||
|
||||
_cheapest_all = min(TODAY_H, key=lambda u: TODAY_MWH[brussels(u).hour])
|
||||
_cheapest_ahead = min(_ahead, key=lambda u: TODAY_MWH[brussels(u).hour])
|
||||
check(f"AC2's stimulus is real: the cheapest hour of the whole published day is "
|
||||
f"{brussels(_cheapest_all).hour:02d}:00 local, which has already elapsed "
|
||||
f"at 09:00, and the cheapest hour still AHEAD is a different one "
|
||||
f"({brussels(_cheapest_ahead).hour:02d}:00). Without this the windowed "
|
||||
f"and unwindowed answers would be the same point and AC2 would grade "
|
||||
f"nothing",
|
||||
_cheapest_all != _cheapest_ahead and _cheapest_all < NOW
|
||||
and _cheapest_ahead >= NOW)
|
||||
|
||||
check(f"KEEP_DAYS is {KEEP_DAYS}, and the prune fixture below files more days "
|
||||
f"than that, so pruning is exercised rather than skipped", KEEP_DAYS < 6)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("1. AC1 - the hole: a partially-published feed now covers the horizon")
|
||||
|
||||
# ⚠️ THE FEED PUBLISHES TODAY IN FULL, WHICH IS WHAT A DAY-AHEAD FEED DOES.
|
||||
# The partial publication this ticket is about is that TOMORROW is missing, not
|
||||
# that this morning is - and the elapsed hours the feed is still serving are
|
||||
# exactly what AC2 is about, so they have to be in the body.
|
||||
_body_today = feed_body(TODAY_H, TODAY_MWH)
|
||||
_until = NOW + timedelta(hours=HORIZON_H)
|
||||
|
||||
# The pre-FCST-04 behaviour, by the shipped code path: NoHistory() records
|
||||
# nothing and carries nothing, so this IS the old plan.
|
||||
_before = run(resolve_tariff(CFG, NOW, _until,
|
||||
transport=transport_for(_body_today), now=NOW,
|
||||
history=NoHistory()))
|
||||
|
||||
# And the same call with a history that already holds yesterday.
|
||||
_h1 = store()
|
||||
_h1_seed = series_of(YDAY_H, YESTERDAY_MWH)
|
||||
_ph.PriceHistory._remember(_h1, _h1_seed, NOW) # seeding, not a caller
|
||||
_after = run(resolve_tariff(CFG, NOW, _until,
|
||||
transport=transport_for(_body_today), now=NOW,
|
||||
history=_h1))
|
||||
|
||||
_n_before = len(_before.read.points)
|
||||
_n_after = len(_after.read.points)
|
||||
check(f"the plan's horizon GREW, measured: {_n_before} h before FCST-04, "
|
||||
f"{_n_after} h after, against a {HORIZON_H} h request",
|
||||
_n_after > _n_before and _n_after == HORIZON_H)
|
||||
|
||||
check(f"...and before the change it was exactly what the feed sent "
|
||||
f"({len(TODAY_H)} h of one day), which is the defect from both ends: the "
|
||||
f"{HORIZON_H - len(_ahead)} hours past the feed's own horizon were not "
|
||||
f"short-priced but ABSENT, and the {len(TODAY_H) - len(_ahead)} hours "
|
||||
f"that had already elapsed were in the series anyway",
|
||||
_n_before == len(TODAY_H))
|
||||
|
||||
# The split, re-derived here from the fixture rather than read off the module.
|
||||
_published_instants = {u.astimezone(UTC) for u in _ahead}
|
||||
_got_carried = {p.start.astimezone(UTC) for p in _after.read.points
|
||||
if p.provisional}
|
||||
_got_firm = {p.start.astimezone(UTC) for p in _after.read.points
|
||||
if not p.provisional}
|
||||
check("AC1: exactly the hours the feed published are FIRM, and exactly the "
|
||||
"hours it did not are PROVISIONAL - the flag is a claim about "
|
||||
"provenance, re-derived here from the fixture",
|
||||
_got_firm == _published_instants
|
||||
and _got_carried == {p.start.astimezone(UTC)
|
||||
for p in _after.read.points} - _published_instants
|
||||
and len(_got_carried) == HORIZON_H - len(_ahead))
|
||||
|
||||
_before_by_start = {p.start.astimezone(UTC): p for p in _before.read.points}
|
||||
check("...and every published point is the PROVIDER'S OWN OBJECT, passed "
|
||||
"through by identity - a copy is where the flag gets lost (FCST-03)",
|
||||
all(p.eur_per_kwh == _before_by_start[p.start.astimezone(UTC)].eur_per_kwh
|
||||
for p in _after.read.points if not p.provisional))
|
||||
|
||||
# The carried numbers are yesterday's, at the same BRUSSELS wall-clock hour.
|
||||
# ⚠️ THE LOOKUP IS BUILT BY SEARCH over yesterday's own instants, not by
|
||||
# subtracting 24 h from today's - a fill that subtracted a fixed day is exactly
|
||||
# what this is meant to catch, and a helper that subtracts one cannot catch it.
|
||||
_Y_BY_LOCAL = {}
|
||||
for _u in YDAY_H:
|
||||
_Y_BY_LOCAL.setdefault(brussels(_u).hour,
|
||||
allin(YESTERDAY_MWH[brussels(_u).hour]))
|
||||
_carried_ok = [_Y_BY_LOCAL.get(brussels(p.start).hour) is not None
|
||||
and abs(p.eur_per_kwh
|
||||
- _Y_BY_LOCAL[brussels(p.start).hour]) < 1e-9
|
||||
for p in _after.read.points if p.provisional]
|
||||
check(f"...and every carried number is YESTERDAY's price for the same BRUSSELS "
|
||||
f"wall-clock hour, not today's and not a neighbour's "
|
||||
f"({sum(_carried_ok)}/{len(_carried_ok)})",
|
||||
_carried_ok and all(_carried_ok))
|
||||
|
||||
check("the plan reports itself PROVISIONAL now, which it did not before - the "
|
||||
"old plan claimed full firmness over a horizon it covered half of",
|
||||
_after.provisional and not _before.provisional)
|
||||
|
||||
check("...and the plan is still a provider plan, not a §11.4 fallback: the "
|
||||
"fill does not degrade anything",
|
||||
_after.origin == "provider" and _after.product is PriceProduct.DAY_AHEAD)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("2. AC2 - the spread stopped being taken over hours that have gone")
|
||||
|
||||
_A = dict(peak_guard_active=False, min_spread_eur_kwh=0.0,
|
||||
uncertainty=frozenset())
|
||||
_v_before = _before.arbitrage(**_A)
|
||||
_v_after = _after.arbitrage(**_A)
|
||||
check("both plans still ALLOW a trade, so this section compares two verdicts "
|
||||
"rather than a verdict and a refusal",
|
||||
_v_before.kind == "allowed" and _v_after.kind == "allowed")
|
||||
|
||||
_cheap_before = _v_before.cheapest.start.astimezone(UTC)
|
||||
_cheap_after = _v_after.cheapest.start.astimezone(UTC)
|
||||
check(f"AC2: the cheapest FIRM hour MOVED from {_cheap_before.isoformat()} - "
|
||||
f"already elapsed at {NOW.isoformat()}, so no cycle could ever buy in it "
|
||||
f"- to {_cheap_after.isoformat()}, which is still ahead",
|
||||
_cheap_before < NOW <= _cheap_after and _cheap_before != _cheap_after)
|
||||
|
||||
check("...and the new one is the cheapest hour the feed published that has NOT "
|
||||
"gone, derived from the fixture rather than from the module",
|
||||
_cheap_after == _cheapest_ahead.astimezone(UTC))
|
||||
|
||||
check("...and no point of the filled series starts before the requested "
|
||||
"`since` hour: the window is [since, until) and the past is not in it",
|
||||
min(p.start.astimezone(UTC) for p in _after.read.points)
|
||||
>= NOW.replace(minute=0, second=0, microsecond=0))
|
||||
|
||||
# ⚠️ THE HONEST LIMIT OF THIS TICKET, checked so it cannot be overclaimed
|
||||
# later: the CARRIED hours buy no trade. §8.3 forbids acting on a provisional
|
||||
# price and refusal 3 drops them from the trade set, so what widened is what
|
||||
# the optimiser can SEE, and the trade above came from the WINDOWING, not from
|
||||
# yesterday's numbers.
|
||||
_firm_after = [p for p in _after.read.points if not p.provisional]
|
||||
check("the carried hours are NOT in the trade set - the verdict's cheapest and "
|
||||
"dearest are both firm, so this ticket widened what the optimiser sees "
|
||||
"and not what it may act on",
|
||||
not _v_after.cheapest.provisional and not _v_after.dearest.provisional
|
||||
and len(_firm_after) == len(_ahead))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("3. AC3 - the cache records the FEED, never its own output")
|
||||
|
||||
_h2 = store()
|
||||
_read2 = series_of(TODAY_H, TODAY_MWH)
|
||||
_filled2 = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW)
|
||||
# Round two, as the refresh task does every 1800 s: the same partial feed.
|
||||
_filled2b = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW)
|
||||
|
||||
_disk = json.load(open(_h2.path, encoding="utf-8"))
|
||||
_today_key = brussels(NOW).strftime("%Y-%m-%d")
|
||||
_recorded = {p["start"] for p in _disk["days"][_today_key]["points"]}
|
||||
_recorded_utc = {datetime.fromisoformat(s).astimezone(UTC) for s in _recorded}
|
||||
check(f"AC3: after two `carry()` rounds the record for {_today_key} holds "
|
||||
f"EXACTLY the {len(TODAY_H)} hours the feed published "
|
||||
f"({len(_recorded_utc)} held) - not one of the "
|
||||
f"{HORIZON_H - len(_ahead)} hours the fill invented",
|
||||
_recorded_utc == {u.astimezone(UTC) for u in TODAY_H})
|
||||
|
||||
check("...and no recorded point is flagged provisional, because a carried "
|
||||
"point never reached the file - the flag would still read true while the "
|
||||
"NUMBER decayed, which is the failure this rule is about",
|
||||
not any(p["provisional"] for p in _disk["days"][_today_key]["points"]))
|
||||
|
||||
def _canon(read):
|
||||
return [p.canonical() for p in read.match(series=lambda r: r.points,
|
||||
no_data=lambda _r: ())]
|
||||
|
||||
|
||||
check(f"...and the fill is idempotent across the two rounds, which is §5.4's "
|
||||
f"late-publication contract - the refresh task runs it again every "
|
||||
f"1800 s against whatever the feed says by then "
|
||||
f"({len(_canon(_filled2))} h both times)",
|
||||
_canon(_filled2) and _canon(_filled2) == _canon(_filled2b))
|
||||
|
||||
# The ordering is the mechanism, so the mechanism is checked in the tree.
|
||||
_APP = os.path.join(HERE, "app")
|
||||
_leaks = []
|
||||
for name in sorted(os.listdir(_APP)):
|
||||
if not name.endswith(".py") or name == "price_history.py":
|
||||
continue
|
||||
src = open(os.path.join(_APP, name), encoding="utf-8").read()
|
||||
if "_remember" in src:
|
||||
_leaks.append(name)
|
||||
check(f"AC3's mechanism is an ORDERING, not a validation - a carried point and "
|
||||
f"a published-provisional point are the same three fields - so nothing "
|
||||
f"outside app/price_history.py may name `_remember` {_leaks or 'none do'}",
|
||||
not _leaks)
|
||||
|
||||
_carry_src = inspect.getsource(PriceHistory.carry)
|
||||
_tree = ast.parse(_carry_src.strip())
|
||||
_calls = [n.func.attr for n in ast.walk(_tree)
|
||||
if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)]
|
||||
check(f"...and `carry()` records BEFORE it fills, so a feed carrying yesterday "
|
||||
f"alongside today refreshes the profile before consuming it {_calls}",
|
||||
_calls.index("_remember") < _calls.index("yesterday"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("4. rule 2 - 'yesterday' is a DATE, not 'the newest day I have'")
|
||||
|
||||
_h3 = store()
|
||||
_ph.PriceHistory._remember(
|
||||
_h3, series_of(bday_hours("2026-01-12"), YESTERDAY_MWH), NOW)
|
||||
_y3 = _h3.yesterday(NOW)
|
||||
check("a store holding only the day before LAST answers no_data, not those "
|
||||
"prices - `newest()` is the accessor that decays, and an add-on down "
|
||||
"over a weekend is when it would",
|
||||
_y3.kind == "no_data")
|
||||
check(f"...and the reason names BOTH the date wanted and the dates held, so an "
|
||||
f"operator can see which it is: {why(_y3)!r}",
|
||||
"2026-01-14" in why(_y3) and "2026-01-12" in why(_y3))
|
||||
|
||||
_h4 = store()
|
||||
_ph.PriceHistory._remember(
|
||||
_h4, series_of(YDAY_H, YESTERDAY_MWH), NOW)
|
||||
_y4 = _h4.yesterday(NOW)
|
||||
check("...while the day BEFORE this one answers a series, so the refusal above "
|
||||
"is about the date and not about the store being broken",
|
||||
_y4.kind == "series" and len(_y4.points) == 24)
|
||||
|
||||
check("a naive instant is refused rather than assumed to be UTC - 'which "
|
||||
"Brussels day was yesterday' is the question a wall clock cannot answer",
|
||||
raises(lambda: _h4.yesterday(datetime(2026, 1, 15, 9)), ValueError))
|
||||
|
||||
# The Brussels-date rule, on the fixture a UTC-date rule gets wrong. A feed
|
||||
# serving the 24 UTC hours of a July day spans TWO Brussels days - Belgium is
|
||||
# on CEST, so 22:00 and 23:00 UTC are already 00:00 and 01:00 of tomorrow - and
|
||||
# a UTC-keyed cache would file all 24 under one date. The evening hours are the
|
||||
# expensive ones, so filing them under the wrong day is the costly direction.
|
||||
_h5 = store()
|
||||
_jul_utc = [datetime(2026, 7, 14, h, tzinfo=UTC) for h in range(24)]
|
||||
_split = {brussels(u).strftime("%Y-%m-%d") for u in _jul_utc}
|
||||
check(f"the July fixture really does straddle two Brussels days {sorted(_split)}"
|
||||
f", or the next check passes for a UTC-keyed cache as well",
|
||||
len(_split) == 2)
|
||||
_ph.PriceHistory._remember(
|
||||
_h5, series_of(_jul_utc, YESTERDAY_MWH), _jul_utc[-1])
|
||||
_disk5 = json.load(open(_h5.path, encoding="utf-8"))
|
||||
_counts = {d: len(_disk5["days"][d]["points"]) for d in sorted(_disk5["days"])}
|
||||
check(f"...and the record is keyed on the BRUSSELS date, so those 24 hours are "
|
||||
f"filed under both of them {_counts} rather than all under one",
|
||||
set(_counts) == _split and sum(_counts.values()) == 24
|
||||
and all(_counts.values()))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("5. AC4 - four degraded stores, four reasons, no prices")
|
||||
|
||||
_missing = PriceHistory(os.path.join(tmpdir(), "gone", "price_history.json"))
|
||||
check("no durable volume: the store is NOT ok, and it does not create the "
|
||||
"directory - a cache on the ephemeral layer reads fine and is thrown "
|
||||
"away on the next restart (tariff.PeakStore's rule)",
|
||||
not _missing.ok and "no durable volume" in (_missing.last_error or "")
|
||||
and not os.path.isdir(os.path.dirname(_missing.path)))
|
||||
|
||||
_d6 = tmpdir()
|
||||
_p6 = os.path.join(_d6, "price_history.json")
|
||||
open(_p6, "w", encoding="utf-8").write("{ this is not json")
|
||||
_h6 = PriceHistory(_p6)
|
||||
check("an unreadable file: the reason is kept and reported, and the record is "
|
||||
"NOT silently re-created as an empty one that reads like a first day",
|
||||
"unreadable" in (_h6.last_error or "")
|
||||
and _h6.yesterday(NOW).kind == "no_data"
|
||||
and "unreadable" in why(_h6.yesterday(NOW)))
|
||||
|
||||
_p7 = os.path.join(tmpdir(), "price_history.json")
|
||||
json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8"))
|
||||
_h7 = PriceHistory(_p7)
|
||||
check("a file from a schema this build does not know is discarded rather than "
|
||||
"guessed at - its points might mean something else",
|
||||
"schema" in (_h7.last_error or "")
|
||||
and _h7.yesterday(NOW).kind == "no_data")
|
||||
|
||||
# A record whose points are malformed. THE DEFECT IS A DEFAULTED PRICE.
|
||||
_p8 = os.path.join(tmpdir(), "price_history.json")
|
||||
_good = {"start": brussels(at_local(YDAY_D, 5)).isoformat(),
|
||||
"eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False}
|
||||
json.dump({"version": SCHEMA, "days": {"2026-01-14": {
|
||||
"product": "dynamic_dayahead", "provider": "generic_dayahead",
|
||||
"points": [_good,
|
||||
{"start": "2026-01-14T06:00:00+01:00"}, # no price
|
||||
{"start": "2026-01-14 07:00:00", "eur_per_kwh": 0.2,
|
||||
"provisional": False}, # naive
|
||||
{"eur_per_kwh": 0.3, "provisional": False}, # no instant
|
||||
"not even a dict"]}}},
|
||||
open(_p8, "w", encoding="utf-8"))
|
||||
_y8 = PriceHistory(_p8).yesterday(NOW)
|
||||
check("a malformed point is DROPPED, not defaulted: the four broken rows are "
|
||||
"gone and the one good row survives",
|
||||
_y8.kind == "series" and len(_y8.points) == 1)
|
||||
check("...and specifically no point came back priced 0.0 - "
|
||||
"`float(row.get('eur_per_kwh', 0))` is how a lost field becomes 'that "
|
||||
"hour was free'",
|
||||
all(p.eur_per_kwh != 0.0 for p in _y8.points)
|
||||
and abs(_y8.points[0].eur_per_kwh - allin(YESTERDAY_MWH[5])) < 1e-9)
|
||||
|
||||
_p9 = os.path.join(tmpdir(), "price_history.json")
|
||||
json.dump({"version": SCHEMA, "days": {"2026-01-14": {
|
||||
"product": "no_such_product", "provider": "x",
|
||||
"points": [_good]}}}, open(_p9, "w", encoding="utf-8"))
|
||||
check("a record naming a product that is not a PriceProduct is no_data, not a "
|
||||
"series with a string in the field the optimiser switches on",
|
||||
PriceHistory(_p9).yesterday(NOW).kind == "no_data")
|
||||
|
||||
# ...and every one of those leaves the CONTROLLER regulating.
|
||||
#
|
||||
# ⚠️ THE PLAN IS NOT BIT-FOR-BIT THE PRE-FCST-04 ONE, and the first draft of
|
||||
# this check claimed it was. A broken cache still goes through the fill, which
|
||||
# WINDOWS to [since, until) whether or not it has anything to carry - so the
|
||||
# plan is the feed's own hours MINUS the elapsed ones. That is the AC2 half of
|
||||
# this change, and it is correct with no cache at all: an hour that has gone is
|
||||
# not a cheap hour. What must survive a broken cache is that the plan is a real
|
||||
# provider series with no invented points and no §11.4 degradation.
|
||||
for _label, _hist in (("no durable volume", _missing),
|
||||
("unreadable file", _h6),
|
||||
("wrong schema", _h7)):
|
||||
_plan = run(resolve_tariff(CFG, NOW, _until,
|
||||
transport=transport_for(_body_today), now=NOW,
|
||||
history=_hist))
|
||||
check(f"...and with {_label} the plan is a real provider series over the "
|
||||
f"{len(_ahead)} published hours still ahead, with NOTHING carried "
|
||||
f"and no §11.4 degradation. A broken cache is not a broken "
|
||||
f"controller - it is a short horizon, which is where this started",
|
||||
_plan.origin == "provider"
|
||||
and len(_plan.read.points) == len(_ahead)
|
||||
and not any(p.provisional for p in _plan.read.points))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("6. rule 4 - hours accumulate within a day; they do not replace it")
|
||||
|
||||
_h10 = store()
|
||||
_ph.PriceHistory._remember(_h10, series_of(TODAY_H, TODAY_MWH), NOW)
|
||||
# A feed that serves only the hours left in the day - a legitimate shape, and
|
||||
# the one that turns replace-the-record into a two-hour yesterday.
|
||||
_REVISED = dict(TODAY_MWH)
|
||||
_REVISED[22], _REVISED[23] = 999.0, 998.0
|
||||
check("the revision really changes the numbers, or merge and replace are the "
|
||||
"same observation",
|
||||
_REVISED[22] != TODAY_MWH[22] and _REVISED[23] != TODAY_MWH[23])
|
||||
_ph.PriceHistory._remember(
|
||||
_h10, series_of([at_local(TODAY_D, 22), at_local(TODAY_D, 23)], _REVISED),
|
||||
NOW)
|
||||
|
||||
_back = _h10.yesterday(NOW + timedelta(days=1))
|
||||
check(f"a two-hour read did not overwrite a 24-hour day: "
|
||||
f"{len(_back.points) if _back.kind == 'series' else _back.kind} hours "
|
||||
f"held. Under replace, the next morning's 'yesterday' would have been "
|
||||
f"two hours long",
|
||||
_back.kind == "series" and len(_back.points) == 24)
|
||||
|
||||
# ⚠️ KEYED BY BRUSSELS HOUR, because that is what the fixture tables are keyed
|
||||
# by. Keying this by UTC hour reads the wrong row of the table in winter and
|
||||
# grades the module against a price nobody put there.
|
||||
_by_hour = {brussels(p.start).hour: p.eur_per_kwh for p in _back.points}
|
||||
|
||||
|
||||
def _priced(h, want):
|
||||
"""True only if hour `h` is present AND carries `want`. A MISSING hour is
|
||||
False, never a KeyError: under `replace` the hours simply are not there, and
|
||||
a harness that dies on the mutation it is grading reports nothing at all."""
|
||||
return h in _by_hour and abs(_by_hour[h] - want) < 1e-9
|
||||
|
||||
|
||||
check("...and the two revised hours carry the NEW price, because a later read "
|
||||
"is the newer statement about the same hour",
|
||||
_priced(22, allin(999.0)) and _priced(23, allin(998.0)))
|
||||
check("...while every untouched hour kept its original price",
|
||||
all(_priced(h, allin(TODAY_MWH[h])) for h in range(22)))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("7. the restart this whole module exists for")
|
||||
|
||||
_d11 = tmpdir()
|
||||
_p11 = os.path.join(_d11, "price_history.json")
|
||||
_first = PriceHistory(_p11)
|
||||
_ph.PriceHistory._remember(
|
||||
_first, series_of(YDAY_H, YESTERDAY_MWH), NOW)
|
||||
del _first
|
||||
_second = PriceHistory(_p11) # a brand-new add-on process
|
||||
_y11 = _second.yesterday(NOW)
|
||||
check("yesterday survives a restart, which is the ONLY reason this is a file "
|
||||
"and not a variable on the refresh task: the window the fill exists for "
|
||||
"is midnight -> publication, and an add-on restarted at 08:00 would "
|
||||
"otherwise spend all of it on the truncated series",
|
||||
_y11.kind == "series" and len(_y11.points) == 24)
|
||||
check("...with the prices intact through the JSON round trip, to six decimals",
|
||||
all(abs(p.eur_per_kwh
|
||||
- allin(YESTERDAY_MWH[brussels(p.start).hour])) < 1e-9
|
||||
for p in _y11.points))
|
||||
|
||||
_plan11 = run(resolve_tariff(CFG, NOW, _until,
|
||||
transport=transport_for(_body_today), now=NOW,
|
||||
history=_second))
|
||||
check("...and the restarted controller's very first refresh already covers the "
|
||||
"whole horizon",
|
||||
len(_plan11.read.points) == HORIZON_H)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("8. pruning keeps yesterday, and a jumped clock does not empty the store")
|
||||
|
||||
_h12 = store()
|
||||
for back in range(6, 0, -1):
|
||||
_ph.PriceHistory._remember(
|
||||
_h12,
|
||||
series_of(bday_hours(
|
||||
(datetime.fromisoformat(TODAY_D) - timedelta(days=back)
|
||||
).strftime("%Y-%m-%d")), YESTERDAY_MWH),
|
||||
NOW)
|
||||
_held = sorted(json.load(open(_h12.path, encoding="utf-8"))["days"])
|
||||
check(f"six days filed, {KEEP_DAYS} held {_held} - a bigger cache would only "
|
||||
f"hold days rule 2 refuses to look at",
|
||||
len(_held) == KEEP_DAYS)
|
||||
check("...and yesterday is one of them, which is the only day that matters",
|
||||
_h12.yesterday(NOW).kind == "series")
|
||||
|
||||
_h13 = store()
|
||||
_ph.PriceHistory._remember(
|
||||
_h13, series_of(YDAY_H, YESTERDAY_MWH), NOW)
|
||||
_ph.PriceHistory._prune(_h13, NOW + timedelta(days=4000))
|
||||
check("pruning is by SORT ORDER, not by arithmetic against the clock: a clock "
|
||||
"that jumped four thousand days does not delete the record, which would "
|
||||
"empty the store on exactly the morning the fill is for",
|
||||
_h13.yesterday(NOW).kind == "series")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("9. the parameter cannot be forgotten, and main.py does not forget it")
|
||||
|
||||
_sig = inspect.signature(resolve_tariff)
|
||||
_hp = _sig.parameters.get("history")
|
||||
check("`resolve_tariff(history=...)` is KEYWORD-ONLY and has NO DEFAULT - a "
|
||||
"default of None meaning 'skip the fill' would hand a caller who forgot "
|
||||
"it the truncated horizon this ticket removes, silently",
|
||||
_hp is not None and _hp.kind is inspect.Parameter.KEYWORD_ONLY
|
||||
and _hp.default is inspect.Parameter.empty)
|
||||
|
||||
check("...so calling it without one is a TypeError, not a quiet regression",
|
||||
raises(lambda: run(resolve_tariff(CFG, NOW, _until,
|
||||
transport=transport_for(_body_today),
|
||||
now=NOW)), TypeError))
|
||||
|
||||
_MAIN = open(os.path.join(_APP, "main.py"), encoding="utf-8").read()
|
||||
_main_tree = ast.parse(_MAIN)
|
||||
_refresh = next(n for n in ast.walk(_main_tree)
|
||||
if isinstance(n, ast.AsyncFunctionDef)
|
||||
and n.name == "run_tariff_refresh")
|
||||
_kw = [k.arg for n in ast.walk(_refresh) if isinstance(n, ast.Call)
|
||||
for k in n.keywords]
|
||||
check(f"the shipped refresh task passes a history {_kw} - the wiring is what "
|
||||
f"this ticket IS, so it is checked in the tree rather than described",
|
||||
"history" in _kw)
|
||||
check("...and it is the CONTROLLER'S store, not a NoHistory() smuggled in to "
|
||||
"make the call typecheck",
|
||||
"history=self.price_history" in _MAIN
|
||||
and "NoHistory" not in _MAIN)
|
||||
|
||||
# NoHistory is inert, and says so by identity rather than by producing a series.
|
||||
_nh_read = series_of(TODAY_H, TODAY_MWH)
|
||||
check("NoHistory() carries nothing and returns the read BY IDENTITY, so the "
|
||||
"explicit 'I keep no history' is exactly the old behaviour and not a "
|
||||
"second implementation of it",
|
||||
NoHistory().carry(_nh_read, since=NOW, until=_until, stamp=NOW)
|
||||
is _nh_read
|
||||
and NoHistory().yesterday(NOW).kind == "no_data")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
for d in _tmpdirs:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
print()
|
||||
if fails:
|
||||
print(f"{len(fails)} FAILED: " + "; ".join(fails))
|
||||
sys.exit(1)
|
||||
print("all checks passed")
|
||||
@@ -69,6 +69,7 @@ from app.price import (
|
||||
Absence, PricePoint, PriceProduct, PriceRead, arbitrage_allowed, brussels,
|
||||
plan_is_provisional, required_spread, uncertainty_of,
|
||||
)
|
||||
from app.price_history import NoHistory
|
||||
from app.static_tariff import (
|
||||
MEAN_WINDOW_DAYS, ProfileInvalid, StaticProfile, TariffPlan,
|
||||
parse_static_profile, resolve_tariff, static_plan, static_status,
|
||||
@@ -746,7 +747,8 @@ _cfg = opts(TWO_PERIOD, price_provider="generic_dayahead",
|
||||
price_feed_url="https://api.energy-charts.info/price?bzn=BE",
|
||||
price_timeout_s=1, price_markup_eur_kwh=0.03,
|
||||
price_taxes_eur_kwh=0.06)
|
||||
_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead))
|
||||
_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead,
|
||||
history=NoHistory()))
|
||||
check("a dead feed falls back to the static profile rather than to nothing",
|
||||
_fb.origin == "fallback" and _fb.has_prices)
|
||||
check("...with the SAME prices the configured-product path produces, because "
|
||||
@@ -769,7 +771,8 @@ check("...and the provisional plan has no cheap window either: §8.3 forbids an
|
||||
not any(_fb.cheap_now(at_local(THU, h)) for h in range(24)))
|
||||
_fb_none = run(resolve_tariff(opts(None, price_provider="generic_dayahead",
|
||||
price_timeout_s=1),
|
||||
THU, THU + timedelta(days=1), transport=_dead))
|
||||
THU, THU + timedelta(days=1), transport=_dead,
|
||||
history=NoHistory()))
|
||||
check("a dead feed with NO static_profile has nothing to fall back to, and "
|
||||
"says that rather than inventing a tariff",
|
||||
_fb_none.origin == "none" and not _fb_none.has_prices)
|
||||
@@ -785,7 +788,7 @@ async def _live(url, timeout_s):
|
||||
|
||||
|
||||
_live_plan = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1),
|
||||
transport=_live))
|
||||
transport=_live, history=NoHistory()))
|
||||
check("a live feed is used in preference to the static profile",
|
||||
_live_plan.origin == "provider"
|
||||
and _live_plan.product is PriceProduct.DAY_AHEAD)
|
||||
|
||||
Reference in New Issue
Block a user