# TARIFF-02 - price provider plugins, and the canonical shape (with later additions)

**Status: SHIPPED.** Tests: `test_price.py`, `test_price_service.py`. Written
down by MAINT-20 from the long comments of `app/price.py` so the module can stay
short. Related: [[TARIFF-07]], [[TARIFF-09]], [[TARIFF-10]], [[TARIFF-14]],
[[TARIFF-15]], [[PRICE-01]].

## Where it lives

specs.md 2.2 puts this at `ems/ingest/prices/`; that layout is not being built.
Owner decision 2026-08-30: everything ships in `goodwe_controller/app/`. The
behaviour in specs.md 9.3 / 8.3 / 11.4 is unchanged and authoritative.

## The canonical format (9.3)

`{"prices": [{"start": "2026-01-15T00:00:00+01:00", "eur_per_kwh": 0.1834,
"provisional": false}]}`. `start` is tz-aware at the Brussels offset for that
instant, the price is EUR/kWh and all-in, `provisional` is per point (8.3: no
irreversible action on a provisional price). Three fields; a fourth would change
the wire format every store, fixture and captured body is written against.

## Three rules the module is shaped by

1. **Double-counting markup is refused, given a correct declaration.** A plugin
   returns `RawPoint`s with the figure its source published and declares in
   `capabilities().sources` which of {ENERGY, MARKUP, TAXES} it already holds.
   `assemble()` (framework only) adds the complement from `price_markup_eur_kwh`
   / `price_taxes_eur_kwh`. A plugin cannot reach those two numbers by any route:
   `PluginOptions` raises `FrameworkOption`, and review of PR #16 wrote a
   double-counting plugin three more ways (`opts._o`, `price_option()`, and the
   public `PRICE_OPTION_DEFAULTS`), so `register()` also refuses any plugin
   whose own code mentions a framework key (`code_leaks()`). The guarantee is
   exactly this: the complement is added once *given a correct `sources`*. A
   plugin declaring `{ENERGY}` for an all-in feed comes out one markup high and
   no code can catch that. test_price.py checks that a plugin's docstring lines
   (`SOURCES:` / `The framework ADDS:`) agree with its declaration, not that
   either is true. `code_leaks()` strips docstrings before searching (they must
   name the components) and fails open on missing source (a frozen build must
   still start; CI asserts source is found for every shipped plugin).
2. **The product type is declared, not inferred from the id.** `capabilities().
   product` is a `PriceProduct`, rides on the series, and `volatility_regime()`
   reads it (8.3: real-time products widen thresholds). A plugin id appears only
   in `PROVIDERS`, `RETIRED_PROVIDERS` and its own class (AC2). No shipped plugin
   declares `REALTIME` since [[TARIFF-07]]; `_WIDE`, `Uncertainty.REALTIME_PRODUCT`
   and `WIDEN_FACTOR`'s 2.5 are kept, and test_price.py section 10 declares
   REALTIME from a class outside this file to keep them live.
3. **"No price data" is a type, not an empty list.** 11.4: no data means static
   profile, provisional plan, no arbitrage; all-zero prices mean arbitrage is
   merely pointless. `PriceRead.series()` refuses an empty list;
   `PriceRead.no_data(reason)` carries `.reason` and nothing else, so
   `min(read.points)`, `for p in read.points` and `.get("points", [])` raise
   `Absence`; `.match(*, series, no_data)` takes both handlers as required
   keywords. Same shape as `sim/riglock.py` after TEST-04 (read its `Absence`
   block before changing this). Two ways past it survive and are not worth code:
   `vars(read).get("points", [])`, and `bool(read)` being True for both
   variants (a falsy `no_data` would turn `if read:` into a guard by accident;
   use `.match`).

## Timezone without `zoneinfo`

The Alpine image has no `tzdata`, so `ZoneInfo("Europe/Brussels")` would raise
on the client's Pi. Every instant is UTC and rendered at its fixed offset, so
the October 02:00-03:00 hour that happens twice is two instants (`02:00+02:00`
and `02:00+01:00`); a naive-local-hour key or a 24-slot array is wrong on both
transition days (test_price.py counts points on both 2026 transitions). The EU
rule: CEST from 01:00 UTC on the last Sunday of March to 01:00 UTC on the last
Sunday of October. Review of PR #16 matched it against `zoneinfo` over 149 040
hourly instants 2020-2036 with zero mismatches. Residual risk is political (EU
abolishes the change): one edit in `brussels_offset()`.

## The feed range (TARIFF-09)

`generic_dayahead.fetch()` used to ignore `since`/`until` and request the bare
URL, so the feed answered ONE Brussels day and 8.3 dropped everything past today
as provisional: at 09:00 over a 48 h horizon 15 firm and 33 carried, at 23:00 1
firm and 47 carried. `feed_days()` returns the Brussels calendar days (first,
last, both inclusive; the API takes dates) that the half-open `[since, until)`
touches: Brussels, never UTC (22:30 UTC in summer is already tomorrow); `until`
exclusive, so local midnight belongs to the day before; an empty or inverted
window collapses to one day. `api.energy-charts.info` capture:
`scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json`.
`dated_feed_url()` refuses a base that already has a query string (appending
would give `?bzn=BE?bzn=BE&start=...`). An operator's `price_feed_url` override
is used verbatim and loses the range, on purpose: rewriting an operator's URL
reorders and re-escapes it, and both providers declare a second host
(`web-api.tp.entsoe.eu`) whose parameters are not `start`/`end`. The cost is
logged at fetch time, naming the horizon consequence. The URL rule is shared by
`generic_dayahead` and `custom_formula` (TARIFF-15), taking `plugin_id` as an
argument so AC2 holds. The "range is used" promise is not mechanically enforced
(a rule would be satisfied by an invented parameter name); test_price.py pins
which plugins consume the range as a literal roster.

## Absence and the covering window

See [[TARIFF-14]] for the cover table: a point covers from its own start to the
next point's start, never more than an hour; the last point gets the span of the
one before; sorted at construction. `covers()` delegates to `at()` (a second copy
of the rule is a second place to be wrong); `window_points()` grades only the
`start` against `[since, until)` and is not a coverage query. `at()` is a linear
scan over about 192 entries on the 30 s path; a bisect is not worth a second
way to be wrong. A duplicated start (not reachable through `load_prices()`)
gives the earlier entry a zero-width window.

## Taxes components (TARIFF-18)

`price_taxes_eur_kwh`'s default (0.06 before) is the sum of
`FLEMISH_TAXES_COMPONENTS_EUR_KWH`, each row one line of the Eneco residential
card for Flanders, September 2026, including 6 % VAT, in EUR/kWh;
test_price.py section 1b re-adds them. It is per DSO: the distribution row runs
5,28 c (Midden-Vlaanderen) to 6,70 c (West), so the all-in Flemish total runs
about 0.117 to 0.131; a site outside Antwerpen types its own. The default sits
near the low end on purpose: the scalar cancels in `max - min`, and its one
decision is the 17 negative-price guard in arbitrage.py, which tests the
absolute all-in sell price. Too high carries a genuinely negative hour above
zero (export into paid-for disposal); too low only declines a trade that would
have paid. `energiebijdrage` is a row of zero: from 1 August 2026 it is merged
into the bijzondere accijns, and the zero row makes the double-count a visible
decision. Moving a shipped default changes which configs start: TARIFF-17's rung
in validate.py refuses `price_taxes_profile` together with a non-default scalar,
comparing against the default (the Supervisor merges every schema key into the
stored blob, 70 of 71 keys on the reference install, CHANGELOG 0.9.0), so a
superseded shipped default (0.06) must still read as untouched, and the rung's
sentence names both figures.

## One in-code mirror of option defaults (CFG-03)

Option defaults live in config.yaml's `options:` and `schema:` and main.py's
`.get()` fallbacks. `PRICE_OPTION_DEFAULTS` is the single in-code mirror for the
price options: validate.py imports it, main.py has no `o.get("price_...", X)`
fallbacks, no dataclass carries a copy, and test_price.py section 1 reads
config.yaml and asserts equality key for key. `static_profile` (TARIFF-03),
`price_taxes_profile` (TARIFF-17), `price_formula` (TARIFF-15),
`price_export_formula` (TARIFF-16) and the OPT-04 keys live there for the same
reason, not in their own modules (a second dict would be a fourth location).
Details:

- The tables / formulas are in `FRAMEWORK_KEYS`: a plugin reading one could add
  taxes a second time per hour, or fold an injection curve into a consumption
  price. `BANNED_IN_PLUGIN_CODE` derives from that set.
- `arbitrage_soc_floor` / `arbitrage_soc_ceiling` are not the maintenance keys
  (a balancing cycle is not a trading reserve).
- `cycle_cost_threshold_eur` is 0.10 because 10.1's name says so, not because it
  makes anything trade: on Belgian day/night numbers the static spread is about
  0.08, so a static-tariff site does not trade at this default (test_arbitrage.py
  section 7). Whether 0.10, or FCST-03's 2.5 / 1.5, is right is an owner
  decision.
- `price_service_token` is a secret (`SECRET_OPTIONS`); `scrub()` removes secret
  values from any text built from an error.

## Widening (FCST-03)

One path, a set of active uncertainty sources (real-time product 8.3, no weather
data 11.4). The combination rule is `max`, not a product (2.5 x 1.5 would give a
threshold nothing on a Belgian tariff clears). The factors are policy defaults,
not derived (no arbitrage ever ran); they differ so `max`, `sum` and `product`
are distinguishable in a check. `widen_factor()` refuses a string (a silent 1.0
would report a widening that is not applied).

## Injection price (AC4, TARIFF-16)

`export_price_eur_kwh()` is a separate scalar. It was said to be so because
nothing dynamic exists to route; that was false (Eneco publishes `0,1 x BELPEX-H
- 1,188`, ENGIE `(B x eSpot_15) - A`). The true reason: an injection curve has
its own coefficients and VAT treatment (injection is VAT-exempt, consumption is
6 %). TARIFF-16 added `price_export_formula`, read by `export_formula()` through
the same parser as the consumption leg (`parse_export_formula()` is
`parse_price_formula()` with its option name moved). `export_price_at()` falls
through to the scalar and ignores the index, correctly. A negative answer is a
real answer (Eneco's formula is below zero for any index under 11,88 EUR/MWh);
nothing floors it. Nothing under `app/` calls either reader to take a decision:
arbitrage prices the sell side as displaced import. A malformed leg is refused at
startup by validate.py; `taxes_profile()` and `export_formula()` still handle it
loudly at run time. `static_tariff` is imported lazily in `taxes_profile()`
because it imports this module at module scope.

## Taxes at each point's own instant (TARIFF-17)

`taxes_at()` takes a scalar or a period table (duck-typed on `price_at`, because
price.py must not import static_tariff.py). A non-flat component matters: ORES
bills `dag 13,27 / nacht 7,39` c, a 5,88 c gap coinciding with the cheap spot
hours, and flattening it understates the spread. `assemble()` must evaluate it
per point; hoisting the lookup out of the loop collapses the feature back into a
level (test_price.py section 24 grades which hour is cheapest, since a
spread-magnitude assertion passes on the hoisted version). On a scalar, `0.0 +
markup` is exactly `markup` in IEEE 754, so scalar sites get identical bytes.
The table wins over the scalar; validate.py refuses configs where both speak.

## Retired providers (TARIFF-07)

`RETIRED_PROVIDERS` exists so `eneco_eddy` and friends are not refused as typos:
they were shipped, the operator chose them, and every failed fetch fell back
silently to `static_profile`. No alias to `generic_dayahead`: they published
retail prices (`B x spot + A`) and it reads wholesale; an alias would invent the
site's two coefficients (behind the klantenzone login; CREG tariff-card pages
404, VREG unreachable). Each value is the id-specific half of the refusal (the
endpoint, what it answered on 2026-09-13, why no replacement can be built);
validate.py owns the framing. `register()` refuses at import to register a
retired id again, because `pid in PROVIDERS` going true silences the named
refusal; re-shipping is a decision recorded by removing it from
`RETIRED_PROVIDERS` in the same commit. See [[TARIFF-07]].

## Parsers (TARIFF-08)

The only parser is columnar: `https://api.energy-charts.info/price?bzn=BE` gives
parallel `unix_seconds` / `price` arrays plus `unit`. The earlier row parser read
`doc["data"]`, a shape no feed ever published, graded by fixtures written from
the same guess, so every fetch ended `SourceUnavailable` and 11.4 fell back to
static silently. Two shapes, two functions: a parser that sniffs can decide
wrong, and a wrong-but-parseable response becomes a price.
`_points_from_columnar_json()` refuses:

- unequal arrays (`zip()` would truncate the evening peak, and a series missing
  the expensive hours is traded on while no-data switches arbitrage off);
- a unit other than the declared one (EUR/kWh under the same keys would be 1000
  times too small and `MAX_PLAUSIBLE_EUR_KWH` bounds only the large side);
- naive time: `unix_seconds` are UTC (`fromtimestamp(t, timezone.utc)`).
  Rendering at the Brussels offset is `assemble()`'s, once, for every plugin.

A `null` price is dropped and logged (the pairing is reliable, the source said
it published none; `_Series.at()` answers the hole with `Absence`);
`scenarios/fetch_epex_be.py` dropped 0 nulls in 12 months to 2026-08-31. The
feed is quarter-hourly since Belgium's 15-minute MTU (2025-10-01); nothing
assumes either granularity (test_price.py section 2b replays both).
`_validate_raw()` accepts negative prices (windy-Sunday hours are the year's best
charging; 17 keeps the regression) and rejects NaN, infinity, an implausible
magnitude, naive datetimes, and duplicate or out-of-order starts.

## Transport and plugin details

- `check_host()` splits the scheme before parsing the host. The first version
  stripped `len("https://")`, which on `http://` ate two characters of the host
  (`pi.energy-charts.info`), so a non-TLS feed failed the allow-list and the check
  "plain http is refused" passed even with the TLS rule deleted. Found by
  mutation in review of PR #16.
- The default transport imports aiohttp lazily so `manual_csv` works with no HTTP
  stack. With `headers` (price_service only) it answers in the safe words of
  `status_words()` and follows no redirects (the host is fixed; a redirect would
  take the token somewhere the allow-list never saw).
- `manual_csv`: the CSV is all-in (the plugin cannot know what a typed number
  contains); `load_prices()` warns when markup / taxes are set non-zero under a
  plugin that sources them. No transport, aiohttp, network or tz database: it is
  the offline escape hatch and the TEST-01 replay ingest path.
- `custom_formula` (TARIFF-15) declares `{ENERGY, MARKUP}`, not `ALL_IN`: a
  card's `Tariefformule` is the energy component only (Eneco tabulates network
  tariffs separately; ENGIE excludes network, transport, levies, excise), so
  `ALL_IN` would drop about 12 c/kWh. The risk left is an operator folding
  network charges into `offset`; the card self-check in custom_formula.py refuses
  that, and grades the coefficients against the card, not the card against
  reality. No expression evaluation: the string is settable over HTTP through
  the ingress config page, so `eval` would be RCE in a root container holding
  `SUPERVISOR_TOKEN`. It reuses `generic_dayahead`'s feed, URL and parser. It
  keeps `scale=1.0`, so its `RawPoint`s carry EUR/MWh in a field called
  `eur_per_kwh`: contained (destructured on the next line, never leaves the
  method; `_validate_raw()` would refuse an escape).
- `startup_preview()` is a hook (TARIFF-15), not a line in `load_prices()`: only
  the plugin knows what went into its arithmetic, and the index beside the
  all-in figure makes a factor of 1000 obvious. It must not raise into the control
  path; the id is recorded before the hook runs so a failing preview is not
  retried every 30 minutes.
- `load_prices()` returns, never raises, except `FrameworkOption` (a plugin trying
  to read the markup; turning it into "no price data" would hide the defect).
- `price_provider_id()` is one spelling of the selected id because the same
  expression was written at three call sites (TARIFF-10). validate.py keeps its
  own reader on purpose: it validates the raw blob.
- `price_provider_selected()` (TARIFF-10): the background refresh used to start
  only when `arbitrage_enabled` (ships false) was on, so a configured site
  fetched nothing (40 h soak: zero feed requests, `present: false` on all 802
  samples). It is not a trading switch: `arbitrage.enabled()` answers that.
- `price_service` (PRICE-01): `FALLBACKS` and `KEEP_LAST` are empty/False for
  every earlier plugin, so `price_chain.py` is not used for them.
  `TOKEN_REFUSED` is the one phrase the fallback chain reads to tell "the owner
  must act" from "the server is down". `covers` lists complete days only; an hour
  missing inside a day is a hole, not a free hour.