diff --git a/goodwe_controller/DECISIONS/TARIFF-02-price-provider-plugins.md b/goodwe_controller/DECISIONS/TARIFF-02-price-provider-plugins.md new file mode 100644 index 0000000..d032440 --- /dev/null +++ b/goodwe_controller/DECISIONS/TARIFF-02-price-provider-plugins.md @@ -0,0 +1,273 @@ +# TARIFF-02 - price provider plugins, and the canonical shape (with later additions) + +**Status: SHIPPED.** Tests: `test_price.py`, `test_price_service.py`. Written +down by MAINT-20 from the long comments of `app/price.py` so the module can stay +short. Related: [[TARIFF-07]], [[TARIFF-09]], [[TARIFF-10]], [[TARIFF-14]], +[[TARIFF-15]], [[PRICE-01]]. + +## Where it lives + +specs.md 2.2 puts this at `ems/ingest/prices/`; that layout is not being built. +Owner decision 2026-08-30: everything ships in `goodwe_controller/app/`. The +behaviour in specs.md 9.3 / 8.3 / 11.4 is unchanged and authoritative. + +## The canonical format (9.3) + +`{"prices": [{"start": "2026-01-15T00:00:00+01:00", "eur_per_kwh": 0.1834, +"provisional": false}]}`. `start` is tz-aware at the Brussels offset for that +instant, the price is EUR/kWh and all-in, `provisional` is per point (8.3: no +irreversible action on a provisional price). Three fields; a fourth would change +the wire format every store, fixture and captured body is written against. + +## Three rules the module is shaped by + +1. **Double-counting markup is refused, given a correct declaration.** A plugin + returns `RawPoint`s with the figure its source published and declares in + `capabilities().sources` which of {ENERGY, MARKUP, TAXES} it already holds. + `assemble()` (framework only) adds the complement from `price_markup_eur_kwh` + / `price_taxes_eur_kwh`. A plugin cannot reach those two numbers by any route: + `PluginOptions` raises `FrameworkOption`, and review of PR #16 wrote a + double-counting plugin three more ways (`opts._o`, `price_option()`, and the + public `PRICE_OPTION_DEFAULTS`), so `register()` also refuses any plugin + whose own code mentions a framework key (`code_leaks()`). The guarantee is + exactly this: the complement is added once *given a correct `sources`*. A + plugin declaring `{ENERGY}` for an all-in feed comes out one markup high and + no code can catch that. test_price.py checks that a plugin's docstring lines + (`SOURCES:` / `The framework ADDS:`) agree with its declaration, not that + either is true. `code_leaks()` strips docstrings before searching (they must + name the components) and fails open on missing source (a frozen build must + still start; CI asserts source is found for every shipped plugin). +2. **The product type is declared, not inferred from the id.** `capabilities(). + product` is a `PriceProduct`, rides on the series, and `volatility_regime()` + reads it (8.3: real-time products widen thresholds). A plugin id appears only + in `PROVIDERS`, `RETIRED_PROVIDERS` and its own class (AC2). No shipped plugin + declares `REALTIME` since [[TARIFF-07]]; `_WIDE`, `Uncertainty.REALTIME_PRODUCT` + and `WIDEN_FACTOR`'s 2.5 are kept, and test_price.py section 10 declares + REALTIME from a class outside this file to keep them live. +3. **"No price data" is a type, not an empty list.** 11.4: no data means static + profile, provisional plan, no arbitrage; all-zero prices mean arbitrage is + merely pointless. `PriceRead.series()` refuses an empty list; + `PriceRead.no_data(reason)` carries `.reason` and nothing else, so + `min(read.points)`, `for p in read.points` and `.get("points", [])` raise + `Absence`; `.match(*, series, no_data)` takes both handlers as required + keywords. Same shape as `sim/riglock.py` after TEST-04 (read its `Absence` + block before changing this). Two ways past it survive and are not worth code: + `vars(read).get("points", [])`, and `bool(read)` being True for both + variants (a falsy `no_data` would turn `if read:` into a guard by accident; + use `.match`). + +## Timezone without `zoneinfo` + +The Alpine image has no `tzdata`, so `ZoneInfo("Europe/Brussels")` would raise +on the client's Pi. Every instant is UTC and rendered at its fixed offset, so +the October 02:00-03:00 hour that happens twice is two instants (`02:00+02:00` +and `02:00+01:00`); a naive-local-hour key or a 24-slot array is wrong on both +transition days (test_price.py counts points on both 2026 transitions). The EU +rule: CEST from 01:00 UTC on the last Sunday of March to 01:00 UTC on the last +Sunday of October. Review of PR #16 matched it against `zoneinfo` over 149 040 +hourly instants 2020-2036 with zero mismatches. Residual risk is political (EU +abolishes the change): one edit in `brussels_offset()`. + +## The feed range (TARIFF-09) + +`generic_dayahead.fetch()` used to ignore `since`/`until` and request the bare +URL, so the feed answered ONE Brussels day and 8.3 dropped everything past today +as provisional: at 09:00 over a 48 h horizon 15 firm and 33 carried, at 23:00 1 +firm and 47 carried. `feed_days()` returns the Brussels calendar days (first, +last, both inclusive; the API takes dates) that the half-open `[since, until)` +touches: Brussels, never UTC (22:30 UTC in summer is already tomorrow); `until` +exclusive, so local midnight belongs to the day before; an empty or inverted +window collapses to one day. `api.energy-charts.info` capture: +`scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json`. +`dated_feed_url()` refuses a base that already has a query string (appending +would give `?bzn=BE?bzn=BE&start=...`). An operator's `price_feed_url` override +is used verbatim and loses the range, on purpose: rewriting an operator's URL +reorders and re-escapes it, and both providers declare a second host +(`web-api.tp.entsoe.eu`) whose parameters are not `start`/`end`. The cost is +logged at fetch time, naming the horizon consequence. The URL rule is shared by +`generic_dayahead` and `custom_formula` (TARIFF-15), taking `plugin_id` as an +argument so AC2 holds. The "range is used" promise is not mechanically enforced +(a rule would be satisfied by an invented parameter name); test_price.py pins +which plugins consume the range as a literal roster. + +## Absence and the covering window + +See [[TARIFF-14]] for the cover table: a point covers from its own start to the +next point's start, never more than an hour; the last point gets the span of the +one before; sorted at construction. `covers()` delegates to `at()` (a second copy +of the rule is a second place to be wrong); `window_points()` grades only the +`start` against `[since, until)` and is not a coverage query. `at()` is a linear +scan over about 192 entries on the 30 s path; a bisect is not worth a second +way to be wrong. A duplicated start (not reachable through `load_prices()`) +gives the earlier entry a zero-width window. + +## Taxes components (TARIFF-18) + +`price_taxes_eur_kwh`'s default (0.06 before) is the sum of +`FLEMISH_TAXES_COMPONENTS_EUR_KWH`, each row one line of the Eneco residential +card for Flanders, September 2026, including 6 % VAT, in EUR/kWh; +test_price.py section 1b re-adds them. It is per DSO: the distribution row runs +5,28 c (Midden-Vlaanderen) to 6,70 c (West), so the all-in Flemish total runs +about 0.117 to 0.131; a site outside Antwerpen types its own. The default sits +near the low end on purpose: the scalar cancels in `max - min`, and its one +decision is the 17 negative-price guard in arbitrage.py, which tests the +absolute all-in sell price. Too high carries a genuinely negative hour above +zero (export into paid-for disposal); too low only declines a trade that would +have paid. `energiebijdrage` is a row of zero: from 1 August 2026 it is merged +into the bijzondere accijns, and the zero row makes the double-count a visible +decision. Moving a shipped default changes which configs start: TARIFF-17's rung +in validate.py refuses `price_taxes_profile` together with a non-default scalar, +comparing against the default (the Supervisor merges every schema key into the +stored blob, 70 of 71 keys on the reference install, CHANGELOG 0.9.0), so a +superseded shipped default (0.06) must still read as untouched, and the rung's +sentence names both figures. + +## One in-code mirror of option defaults (CFG-03) + +Option defaults live in config.yaml's `options:` and `schema:` and main.py's +`.get()` fallbacks. `PRICE_OPTION_DEFAULTS` is the single in-code mirror for the +price options: validate.py imports it, main.py has no `o.get("price_...", X)` +fallbacks, no dataclass carries a copy, and test_price.py section 1 reads +config.yaml and asserts equality key for key. `static_profile` (TARIFF-03), +`price_taxes_profile` (TARIFF-17), `price_formula` (TARIFF-15), +`price_export_formula` (TARIFF-16) and the OPT-04 keys live there for the same +reason, not in their own modules (a second dict would be a fourth location). +Details: + +- The tables / formulas are in `FRAMEWORK_KEYS`: a plugin reading one could add + taxes a second time per hour, or fold an injection curve into a consumption + price. `BANNED_IN_PLUGIN_CODE` derives from that set. +- `arbitrage_soc_floor` / `arbitrage_soc_ceiling` are not the maintenance keys + (a balancing cycle is not a trading reserve). +- `cycle_cost_threshold_eur` is 0.10 because 10.1's name says so, not because it + makes anything trade: on Belgian day/night numbers the static spread is about + 0.08, so a static-tariff site does not trade at this default (test_arbitrage.py + section 7). Whether 0.10, or FCST-03's 2.5 / 1.5, is right is an owner + decision. +- `price_service_token` is a secret (`SECRET_OPTIONS`); `scrub()` removes secret + values from any text built from an error. + +## Widening (FCST-03) + +One path, a set of active uncertainty sources (real-time product 8.3, no weather +data 11.4). The combination rule is `max`, not a product (2.5 x 1.5 would give a +threshold nothing on a Belgian tariff clears). The factors are policy defaults, +not derived (no arbitrage ever ran); they differ so `max`, `sum` and `product` +are distinguishable in a check. `widen_factor()` refuses a string (a silent 1.0 +would report a widening that is not applied). + +## Injection price (AC4, TARIFF-16) + +`export_price_eur_kwh()` is a separate scalar. It was said to be so because +nothing dynamic exists to route; that was false (Eneco publishes `0,1 x BELPEX-H +- 1,188`, ENGIE `(B x eSpot_15) - A`). The true reason: an injection curve has +its own coefficients and VAT treatment (injection is VAT-exempt, consumption is +6 %). TARIFF-16 added `price_export_formula`, read by `export_formula()` through +the same parser as the consumption leg (`parse_export_formula()` is +`parse_price_formula()` with its option name moved). `export_price_at()` falls +through to the scalar and ignores the index, correctly. A negative answer is a +real answer (Eneco's formula is below zero for any index under 11,88 EUR/MWh); +nothing floors it. Nothing under `app/` calls either reader to take a decision: +arbitrage prices the sell side as displaced import. A malformed leg is refused at +startup by validate.py; `taxes_profile()` and `export_formula()` still handle it +loudly at run time. `static_tariff` is imported lazily in `taxes_profile()` +because it imports this module at module scope. + +## Taxes at each point's own instant (TARIFF-17) + +`taxes_at()` takes a scalar or a period table (duck-typed on `price_at`, because +price.py must not import static_tariff.py). A non-flat component matters: ORES +bills `dag 13,27 / nacht 7,39` c, a 5,88 c gap coinciding with the cheap spot +hours, and flattening it understates the spread. `assemble()` must evaluate it +per point; hoisting the lookup out of the loop collapses the feature back into a +level (test_price.py section 24 grades which hour is cheapest, since a +spread-magnitude assertion passes on the hoisted version). On a scalar, `0.0 + +markup` is exactly `markup` in IEEE 754, so scalar sites get identical bytes. +The table wins over the scalar; validate.py refuses configs where both speak. + +## Retired providers (TARIFF-07) + +`RETIRED_PROVIDERS` exists so `eneco_eddy` and friends are not refused as typos: +they were shipped, the operator chose them, and every failed fetch fell back +silently to `static_profile`. No alias to `generic_dayahead`: they published +retail prices (`B x spot + A`) and it reads wholesale; an alias would invent the +site's two coefficients (behind the klantenzone login; CREG tariff-card pages +404, VREG unreachable). Each value is the id-specific half of the refusal (the +endpoint, what it answered on 2026-09-13, why no replacement can be built); +validate.py owns the framing. `register()` refuses at import to register a +retired id again, because `pid in PROVIDERS` going true silences the named +refusal; re-shipping is a decision recorded by removing it from +`RETIRED_PROVIDERS` in the same commit. See [[TARIFF-07]]. + +## Parsers (TARIFF-08) + +The only parser is columnar: `https://api.energy-charts.info/price?bzn=BE` gives +parallel `unix_seconds` / `price` arrays plus `unit`. The earlier row parser read +`doc["data"]`, a shape no feed ever published, graded by fixtures written from +the same guess, so every fetch ended `SourceUnavailable` and 11.4 fell back to +static silently. Two shapes, two functions: a parser that sniffs can decide +wrong, and a wrong-but-parseable response becomes a price. +`_points_from_columnar_json()` refuses: + +- unequal arrays (`zip()` would truncate the evening peak, and a series missing + the expensive hours is traded on while no-data switches arbitrage off); +- a unit other than the declared one (EUR/kWh under the same keys would be 1000 + times too small and `MAX_PLAUSIBLE_EUR_KWH` bounds only the large side); +- naive time: `unix_seconds` are UTC (`fromtimestamp(t, timezone.utc)`). + Rendering at the Brussels offset is `assemble()`'s, once, for every plugin. + +A `null` price is dropped and logged (the pairing is reliable, the source said +it published none; `_Series.at()` answers the hole with `Absence`); +`scenarios/fetch_epex_be.py` dropped 0 nulls in 12 months to 2026-08-31. The +feed is quarter-hourly since Belgium's 15-minute MTU (2025-10-01); nothing +assumes either granularity (test_price.py section 2b replays both). +`_validate_raw()` accepts negative prices (windy-Sunday hours are the year's best +charging; 17 keeps the regression) and rejects NaN, infinity, an implausible +magnitude, naive datetimes, and duplicate or out-of-order starts. + +## Transport and plugin details + +- `check_host()` splits the scheme before parsing the host. The first version + stripped `len("https://")`, which on `http://` ate two characters of the host + (`pi.energy-charts.info`), so a non-TLS feed failed the allow-list and the check + "plain http is refused" passed even with the TLS rule deleted. Found by + mutation in review of PR #16. +- The default transport imports aiohttp lazily so `manual_csv` works with no HTTP + stack. With `headers` (price_service only) it answers in the safe words of + `status_words()` and follows no redirects (the host is fixed; a redirect would + take the token somewhere the allow-list never saw). +- `manual_csv`: the CSV is all-in (the plugin cannot know what a typed number + contains); `load_prices()` warns when markup / taxes are set non-zero under a + plugin that sources them. No transport, aiohttp, network or tz database: it is + the offline escape hatch and the TEST-01 replay ingest path. +- `custom_formula` (TARIFF-15) declares `{ENERGY, MARKUP}`, not `ALL_IN`: a + card's `Tariefformule` is the energy component only (Eneco tabulates network + tariffs separately; ENGIE excludes network, transport, levies, excise), so + `ALL_IN` would drop about 12 c/kWh. The risk left is an operator folding + network charges into `offset`; the card self-check in custom_formula.py refuses + that, and grades the coefficients against the card, not the card against + reality. No expression evaluation: the string is settable over HTTP through + the ingress config page, so `eval` would be RCE in a root container holding + `SUPERVISOR_TOKEN`. It reuses `generic_dayahead`'s feed, URL and parser. It + keeps `scale=1.0`, so its `RawPoint`s carry EUR/MWh in a field called + `eur_per_kwh`: contained (destructured on the next line, never leaves the + method; `_validate_raw()` would refuse an escape). +- `startup_preview()` is a hook (TARIFF-15), not a line in `load_prices()`: only + the plugin knows what went into its arithmetic, and the index beside the + all-in figure makes a factor of 1000 obvious. It must not raise into the control + path; the id is recorded before the hook runs so a failing preview is not + retried every 30 minutes. +- `load_prices()` returns, never raises, except `FrameworkOption` (a plugin trying + to read the markup; turning it into "no price data" would hide the defect). +- `price_provider_id()` is one spelling of the selected id because the same + expression was written at three call sites (TARIFF-10). validate.py keeps its + own reader on purpose: it validates the raw blob. +- `price_provider_selected()` (TARIFF-10): the background refresh used to start + only when `arbitrage_enabled` (ships false) was on, so a configured site + fetched nothing (40 h soak: zero feed requests, `present: false` on all 802 + samples). It is not a trading switch: `arbitrage.enabled()` answers that. +- `price_service` (PRICE-01): `FALLBACKS` and `KEEP_LAST` are empty/False for + every earlier plugin, so `price_chain.py` is not used for them. + `TOKEN_REFUSED` is the one phrase the fallback chain reads to tell "the owner + must act" from "the server is down". `covers` lists complete days only; an hour + missing inside a day is a hole, not a free hour.