MAINT-20 part 1: DECISIONS note for the price provider plugins (TARIFF-02)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
glenn schrooyen
co-authored by
Claude
Claude
parent
96ca66fc38
commit
194efac98d
@@ -0,0 +1,273 @@
|
||||
# TARIFF-02 - price provider plugins, and the canonical shape (with later additions)
|
||||
|
||||
**Status: SHIPPED.** Tests: `test_price.py`, `test_price_service.py`. Written
|
||||
down by MAINT-20 from the long comments of `app/price.py` so the module can stay
|
||||
short. Related: [[TARIFF-07]], [[TARIFF-09]], [[TARIFF-10]], [[TARIFF-14]],
|
||||
[[TARIFF-15]], [[PRICE-01]].
|
||||
|
||||
## Where it lives
|
||||
|
||||
specs.md 2.2 puts this at `ems/ingest/prices/`; that layout is not being built.
|
||||
Owner decision 2026-08-30: everything ships in `goodwe_controller/app/`. The
|
||||
behaviour in specs.md 9.3 / 8.3 / 11.4 is unchanged and authoritative.
|
||||
|
||||
## The canonical format (9.3)
|
||||
|
||||
`{"prices": [{"start": "2026-01-15T00:00:00+01:00", "eur_per_kwh": 0.1834,
|
||||
"provisional": false}]}`. `start` is tz-aware at the Brussels offset for that
|
||||
instant, the price is EUR/kWh and all-in, `provisional` is per point (8.3: no
|
||||
irreversible action on a provisional price). Three fields; a fourth would change
|
||||
the wire format every store, fixture and captured body is written against.
|
||||
|
||||
## Three rules the module is shaped by
|
||||
|
||||
1. **Double-counting markup is refused, given a correct declaration.** A plugin
|
||||
returns `RawPoint`s with the figure its source published and declares in
|
||||
`capabilities().sources` which of {ENERGY, MARKUP, TAXES} it already holds.
|
||||
`assemble()` (framework only) adds the complement from `price_markup_eur_kwh`
|
||||
/ `price_taxes_eur_kwh`. A plugin cannot reach those two numbers by any route:
|
||||
`PluginOptions` raises `FrameworkOption`, and review of PR #16 wrote a
|
||||
double-counting plugin three more ways (`opts._o`, `price_option()`, and the
|
||||
public `PRICE_OPTION_DEFAULTS`), so `register()` also refuses any plugin
|
||||
whose own code mentions a framework key (`code_leaks()`). The guarantee is
|
||||
exactly this: the complement is added once *given a correct `sources`*. A
|
||||
plugin declaring `{ENERGY}` for an all-in feed comes out one markup high and
|
||||
no code can catch that. test_price.py checks that a plugin's docstring lines
|
||||
(`SOURCES:` / `The framework ADDS:`) agree with its declaration, not that
|
||||
either is true. `code_leaks()` strips docstrings before searching (they must
|
||||
name the components) and fails open on missing source (a frozen build must
|
||||
still start; CI asserts source is found for every shipped plugin).
|
||||
2. **The product type is declared, not inferred from the id.** `capabilities().
|
||||
product` is a `PriceProduct`, rides on the series, and `volatility_regime()`
|
||||
reads it (8.3: real-time products widen thresholds). A plugin id appears only
|
||||
in `PROVIDERS`, `RETIRED_PROVIDERS` and its own class (AC2). No shipped plugin
|
||||
declares `REALTIME` since [[TARIFF-07]]; `_WIDE`, `Uncertainty.REALTIME_PRODUCT`
|
||||
and `WIDEN_FACTOR`'s 2.5 are kept, and test_price.py section 10 declares
|
||||
REALTIME from a class outside this file to keep them live.
|
||||
3. **"No price data" is a type, not an empty list.** 11.4: no data means static
|
||||
profile, provisional plan, no arbitrage; all-zero prices mean arbitrage is
|
||||
merely pointless. `PriceRead.series()` refuses an empty list;
|
||||
`PriceRead.no_data(reason)` carries `.reason` and nothing else, so
|
||||
`min(read.points)`, `for p in read.points` and `.get("points", [])` raise
|
||||
`Absence`; `.match(*, series, no_data)` takes both handlers as required
|
||||
keywords. Same shape as `sim/riglock.py` after TEST-04 (read its `Absence`
|
||||
block before changing this). Two ways past it survive and are not worth code:
|
||||
`vars(read).get("points", [])`, and `bool(read)` being True for both
|
||||
variants (a falsy `no_data` would turn `if read:` into a guard by accident;
|
||||
use `.match`).
|
||||
|
||||
## Timezone without `zoneinfo`
|
||||
|
||||
The Alpine image has no `tzdata`, so `ZoneInfo("Europe/Brussels")` would raise
|
||||
on the client's Pi. Every instant is UTC and rendered at its fixed offset, so
|
||||
the October 02:00-03:00 hour that happens twice is two instants (`02:00+02:00`
|
||||
and `02:00+01:00`); a naive-local-hour key or a 24-slot array is wrong on both
|
||||
transition days (test_price.py counts points on both 2026 transitions). The EU
|
||||
rule: CEST from 01:00 UTC on the last Sunday of March to 01:00 UTC on the last
|
||||
Sunday of October. Review of PR #16 matched it against `zoneinfo` over 149 040
|
||||
hourly instants 2020-2036 with zero mismatches. Residual risk is political (EU
|
||||
abolishes the change): one edit in `brussels_offset()`.
|
||||
|
||||
## The feed range (TARIFF-09)
|
||||
|
||||
`generic_dayahead.fetch()` used to ignore `since`/`until` and request the bare
|
||||
URL, so the feed answered ONE Brussels day and 8.3 dropped everything past today
|
||||
as provisional: at 09:00 over a 48 h horizon 15 firm and 33 carried, at 23:00 1
|
||||
firm and 47 carried. `feed_days()` returns the Brussels calendar days (first,
|
||||
last, both inclusive; the API takes dates) that the half-open `[since, until)`
|
||||
touches: Brussels, never UTC (22:30 UTC in summer is already tomorrow); `until`
|
||||
exclusive, so local midnight belongs to the day before; an empty or inverted
|
||||
window collapses to one day. `api.energy-charts.info` capture:
|
||||
`scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json`.
|
||||
`dated_feed_url()` refuses a base that already has a query string (appending
|
||||
would give `?bzn=BE?bzn=BE&start=...`). An operator's `price_feed_url` override
|
||||
is used verbatim and loses the range, on purpose: rewriting an operator's URL
|
||||
reorders and re-escapes it, and both providers declare a second host
|
||||
(`web-api.tp.entsoe.eu`) whose parameters are not `start`/`end`. The cost is
|
||||
logged at fetch time, naming the horizon consequence. The URL rule is shared by
|
||||
`generic_dayahead` and `custom_formula` (TARIFF-15), taking `plugin_id` as an
|
||||
argument so AC2 holds. The "range is used" promise is not mechanically enforced
|
||||
(a rule would be satisfied by an invented parameter name); test_price.py pins
|
||||
which plugins consume the range as a literal roster.
|
||||
|
||||
## Absence and the covering window
|
||||
|
||||
See [[TARIFF-14]] for the cover table: a point covers from its own start to the
|
||||
next point's start, never more than an hour; the last point gets the span of the
|
||||
one before; sorted at construction. `covers()` delegates to `at()` (a second copy
|
||||
of the rule is a second place to be wrong); `window_points()` grades only the
|
||||
`start` against `[since, until)` and is not a coverage query. `at()` is a linear
|
||||
scan over about 192 entries on the 30 s path; a bisect is not worth a second
|
||||
way to be wrong. A duplicated start (not reachable through `load_prices()`)
|
||||
gives the earlier entry a zero-width window.
|
||||
|
||||
## Taxes components (TARIFF-18)
|
||||
|
||||
`price_taxes_eur_kwh`'s default (0.06 before) is the sum of
|
||||
`FLEMISH_TAXES_COMPONENTS_EUR_KWH`, each row one line of the Eneco residential
|
||||
card for Flanders, September 2026, including 6 % VAT, in EUR/kWh;
|
||||
test_price.py section 1b re-adds them. It is per DSO: the distribution row runs
|
||||
5,28 c (Midden-Vlaanderen) to 6,70 c (West), so the all-in Flemish total runs
|
||||
about 0.117 to 0.131; a site outside Antwerpen types its own. The default sits
|
||||
near the low end on purpose: the scalar cancels in `max - min`, and its one
|
||||
decision is the 17 negative-price guard in arbitrage.py, which tests the
|
||||
absolute all-in sell price. Too high carries a genuinely negative hour above
|
||||
zero (export into paid-for disposal); too low only declines a trade that would
|
||||
have paid. `energiebijdrage` is a row of zero: from 1 August 2026 it is merged
|
||||
into the bijzondere accijns, and the zero row makes the double-count a visible
|
||||
decision. Moving a shipped default changes which configs start: TARIFF-17's rung
|
||||
in validate.py refuses `price_taxes_profile` together with a non-default scalar,
|
||||
comparing against the default (the Supervisor merges every schema key into the
|
||||
stored blob, 70 of 71 keys on the reference install, CHANGELOG 0.9.0), so a
|
||||
superseded shipped default (0.06) must still read as untouched, and the rung's
|
||||
sentence names both figures.
|
||||
|
||||
## One in-code mirror of option defaults (CFG-03)
|
||||
|
||||
Option defaults live in config.yaml's `options:` and `schema:` and main.py's
|
||||
`.get()` fallbacks. `PRICE_OPTION_DEFAULTS` is the single in-code mirror for the
|
||||
price options: validate.py imports it, main.py has no `o.get("price_...", X)`
|
||||
fallbacks, no dataclass carries a copy, and test_price.py section 1 reads
|
||||
config.yaml and asserts equality key for key. `static_profile` (TARIFF-03),
|
||||
`price_taxes_profile` (TARIFF-17), `price_formula` (TARIFF-15),
|
||||
`price_export_formula` (TARIFF-16) and the OPT-04 keys live there for the same
|
||||
reason, not in their own modules (a second dict would be a fourth location).
|
||||
Details:
|
||||
|
||||
- The tables / formulas are in `FRAMEWORK_KEYS`: a plugin reading one could add
|
||||
taxes a second time per hour, or fold an injection curve into a consumption
|
||||
price. `BANNED_IN_PLUGIN_CODE` derives from that set.
|
||||
- `arbitrage_soc_floor` / `arbitrage_soc_ceiling` are not the maintenance keys
|
||||
(a balancing cycle is not a trading reserve).
|
||||
- `cycle_cost_threshold_eur` is 0.10 because 10.1's name says so, not because it
|
||||
makes anything trade: on Belgian day/night numbers the static spread is about
|
||||
0.08, so a static-tariff site does not trade at this default (test_arbitrage.py
|
||||
section 7). Whether 0.10, or FCST-03's 2.5 / 1.5, is right is an owner
|
||||
decision.
|
||||
- `price_service_token` is a secret (`SECRET_OPTIONS`); `scrub()` removes secret
|
||||
values from any text built from an error.
|
||||
|
||||
## Widening (FCST-03)
|
||||
|
||||
One path, a set of active uncertainty sources (real-time product 8.3, no weather
|
||||
data 11.4). The combination rule is `max`, not a product (2.5 x 1.5 would give a
|
||||
threshold nothing on a Belgian tariff clears). The factors are policy defaults,
|
||||
not derived (no arbitrage ever ran); they differ so `max`, `sum` and `product`
|
||||
are distinguishable in a check. `widen_factor()` refuses a string (a silent 1.0
|
||||
would report a widening that is not applied).
|
||||
|
||||
## Injection price (AC4, TARIFF-16)
|
||||
|
||||
`export_price_eur_kwh()` is a separate scalar. It was said to be so because
|
||||
nothing dynamic exists to route; that was false (Eneco publishes `0,1 x BELPEX-H
|
||||
- 1,188`, ENGIE `(B x eSpot_15) - A`). The true reason: an injection curve has
|
||||
its own coefficients and VAT treatment (injection is VAT-exempt, consumption is
|
||||
6 %). TARIFF-16 added `price_export_formula`, read by `export_formula()` through
|
||||
the same parser as the consumption leg (`parse_export_formula()` is
|
||||
`parse_price_formula()` with its option name moved). `export_price_at()` falls
|
||||
through to the scalar and ignores the index, correctly. A negative answer is a
|
||||
real answer (Eneco's formula is below zero for any index under 11,88 EUR/MWh);
|
||||
nothing floors it. Nothing under `app/` calls either reader to take a decision:
|
||||
arbitrage prices the sell side as displaced import. A malformed leg is refused at
|
||||
startup by validate.py; `taxes_profile()` and `export_formula()` still handle it
|
||||
loudly at run time. `static_tariff` is imported lazily in `taxes_profile()`
|
||||
because it imports this module at module scope.
|
||||
|
||||
## Taxes at each point's own instant (TARIFF-17)
|
||||
|
||||
`taxes_at()` takes a scalar or a period table (duck-typed on `price_at`, because
|
||||
price.py must not import static_tariff.py). A non-flat component matters: ORES
|
||||
bills `dag 13,27 / nacht 7,39` c, a 5,88 c gap coinciding with the cheap spot
|
||||
hours, and flattening it understates the spread. `assemble()` must evaluate it
|
||||
per point; hoisting the lookup out of the loop collapses the feature back into a
|
||||
level (test_price.py section 24 grades which hour is cheapest, since a
|
||||
spread-magnitude assertion passes on the hoisted version). On a scalar, `0.0 +
|
||||
markup` is exactly `markup` in IEEE 754, so scalar sites get identical bytes.
|
||||
The table wins over the scalar; validate.py refuses configs where both speak.
|
||||
|
||||
## Retired providers (TARIFF-07)
|
||||
|
||||
`RETIRED_PROVIDERS` exists so `eneco_eddy` and friends are not refused as typos:
|
||||
they were shipped, the operator chose them, and every failed fetch fell back
|
||||
silently to `static_profile`. No alias to `generic_dayahead`: they published
|
||||
retail prices (`B x spot + A`) and it reads wholesale; an alias would invent the
|
||||
site's two coefficients (behind the klantenzone login; CREG tariff-card pages
|
||||
404, VREG unreachable). Each value is the id-specific half of the refusal (the
|
||||
endpoint, what it answered on 2026-09-13, why no replacement can be built);
|
||||
validate.py owns the framing. `register()` refuses at import to register a
|
||||
retired id again, because `pid in PROVIDERS` going true silences the named
|
||||
refusal; re-shipping is a decision recorded by removing it from
|
||||
`RETIRED_PROVIDERS` in the same commit. See [[TARIFF-07]].
|
||||
|
||||
## Parsers (TARIFF-08)
|
||||
|
||||
The only parser is columnar: `https://api.energy-charts.info/price?bzn=BE` gives
|
||||
parallel `unix_seconds` / `price` arrays plus `unit`. The earlier row parser read
|
||||
`doc["data"]`, a shape no feed ever published, graded by fixtures written from
|
||||
the same guess, so every fetch ended `SourceUnavailable` and 11.4 fell back to
|
||||
static silently. Two shapes, two functions: a parser that sniffs can decide
|
||||
wrong, and a wrong-but-parseable response becomes a price.
|
||||
`_points_from_columnar_json()` refuses:
|
||||
|
||||
- unequal arrays (`zip()` would truncate the evening peak, and a series missing
|
||||
the expensive hours is traded on while no-data switches arbitrage off);
|
||||
- a unit other than the declared one (EUR/kWh under the same keys would be 1000
|
||||
times too small and `MAX_PLAUSIBLE_EUR_KWH` bounds only the large side);
|
||||
- naive time: `unix_seconds` are UTC (`fromtimestamp(t, timezone.utc)`).
|
||||
Rendering at the Brussels offset is `assemble()`'s, once, for every plugin.
|
||||
|
||||
A `null` price is dropped and logged (the pairing is reliable, the source said
|
||||
it published none; `_Series.at()` answers the hole with `Absence`);
|
||||
`scenarios/fetch_epex_be.py` dropped 0 nulls in 12 months to 2026-08-31. The
|
||||
feed is quarter-hourly since Belgium's 15-minute MTU (2025-10-01); nothing
|
||||
assumes either granularity (test_price.py section 2b replays both).
|
||||
`_validate_raw()` accepts negative prices (windy-Sunday hours are the year's best
|
||||
charging; 17 keeps the regression) and rejects NaN, infinity, an implausible
|
||||
magnitude, naive datetimes, and duplicate or out-of-order starts.
|
||||
|
||||
## Transport and plugin details
|
||||
|
||||
- `check_host()` splits the scheme before parsing the host. The first version
|
||||
stripped `len("https://")`, which on `http://` ate two characters of the host
|
||||
(`pi.energy-charts.info`), so a non-TLS feed failed the allow-list and the check
|
||||
"plain http is refused" passed even with the TLS rule deleted. Found by
|
||||
mutation in review of PR #16.
|
||||
- The default transport imports aiohttp lazily so `manual_csv` works with no HTTP
|
||||
stack. With `headers` (price_service only) it answers in the safe words of
|
||||
`status_words()` and follows no redirects (the host is fixed; a redirect would
|
||||
take the token somewhere the allow-list never saw).
|
||||
- `manual_csv`: the CSV is all-in (the plugin cannot know what a typed number
|
||||
contains); `load_prices()` warns when markup / taxes are set non-zero under a
|
||||
plugin that sources them. No transport, aiohttp, network or tz database: it is
|
||||
the offline escape hatch and the TEST-01 replay ingest path.
|
||||
- `custom_formula` (TARIFF-15) declares `{ENERGY, MARKUP}`, not `ALL_IN`: a
|
||||
card's `Tariefformule` is the energy component only (Eneco tabulates network
|
||||
tariffs separately; ENGIE excludes network, transport, levies, excise), so
|
||||
`ALL_IN` would drop about 12 c/kWh. The risk left is an operator folding
|
||||
network charges into `offset`; the card self-check in custom_formula.py refuses
|
||||
that, and grades the coefficients against the card, not the card against
|
||||
reality. No expression evaluation: the string is settable over HTTP through
|
||||
the ingress config page, so `eval` would be RCE in a root container holding
|
||||
`SUPERVISOR_TOKEN`. It reuses `generic_dayahead`'s feed, URL and parser. It
|
||||
keeps `scale=1.0`, so its `RawPoint`s carry EUR/MWh in a field called
|
||||
`eur_per_kwh`: contained (destructured on the next line, never leaves the
|
||||
method; `_validate_raw()` would refuse an escape).
|
||||
- `startup_preview()` is a hook (TARIFF-15), not a line in `load_prices()`: only
|
||||
the plugin knows what went into its arithmetic, and the index beside the
|
||||
all-in figure makes a factor of 1000 obvious. It must not raise into the control
|
||||
path; the id is recorded before the hook runs so a failing preview is not
|
||||
retried every 30 minutes.
|
||||
- `load_prices()` returns, never raises, except `FrameworkOption` (a plugin trying
|
||||
to read the markup; turning it into "no price data" would hide the defect).
|
||||
- `price_provider_id()` is one spelling of the selected id because the same
|
||||
expression was written at three call sites (TARIFF-10). validate.py keeps its
|
||||
own reader on purpose: it validates the raw blob.
|
||||
- `price_provider_selected()` (TARIFF-10): the background refresh used to start
|
||||
only when `arbitrage_enabled` (ships false) was on, so a configured site
|
||||
fetched nothing (40 h soak: zero feed requests, `present: false` on all 802
|
||||
samples). It is not a trading switch: `arbitrage.enabled()` answers that.
|
||||
- `price_service` (PRICE-01): `FALLBACKS` and `KEEP_LAST` are empty/False for
|
||||
every earlier plugin, so `price_chain.py` is not used for them.
|
||||
`TOKEN_REFUSED` is the one phrase the fallback chain reads to tell "the owner
|
||||
must act" from "the server is down". `covers` lists complete days only; an hour
|
||||
missing inside a day is a hole, not a free hour.
|
||||
Reference in New Issue
Block a user