goodwe-addon/goodwe_controller/DECISIONS/TARIFF-02-price-provider-plugins.md

TARIFF-02 - price provider plugins, and the canonical shape (with later additions)

Status: SHIPPED. Tests: test_price.py, test_price_service.py. Written down by MAINT-20 from the long comments of app/price.py so the module can stay short. Related: TARIFF-07, TARIFF-09, TARIFF-10, TARIFF-14, TARIFF-15, PRICE-01.

Where it lives

specs.md 2.2 puts this at ems/ingest/prices/; that layout is not being built. Owner decision 2026-08-30: everything ships in goodwe_controller/app/. The behaviour in specs.md 9.3 / 8.3 / 11.4 is unchanged and authoritative.

The canonical format (9.3)

{"prices": [{"start": "2026-01-15T00:00:00+01:00", "eur_per_kwh": 0.1834, "provisional": false}]}. start is tz-aware at the Brussels offset for that instant, the price is EUR/kWh and all-in, provisional is per point (8.3: no irreversible action on a provisional price). Three fields; a fourth would change the wire format every store, fixture and captured body is written against.

Three rules the module is shaped by

1. Double-counting markup is refused, given a correct declaration. A plugin returns RawPoints with the figure its source published and declares in capabilities().sources which of {ENERGY, MARKUP, TAXES} it already holds. assemble() (framework only) adds the complement from price_markup_eur_kwh / price_taxes_eur_kwh. A plugin cannot reach those two numbers by any route: PluginOptions raises FrameworkOption, and review of PR #16 wrote a double-counting plugin three more ways (opts._o, price_option(), and the public PRICE_OPTION_DEFAULTS), so register() also refuses any plugin whose own code mentions a framework key (code_leaks()). The guarantee is exactly this: the complement is added once given a correct sources. A plugin declaring {ENERGY} for an all-in feed comes out one markup high and no code can catch that. test_price.py checks that a plugin's docstring lines (SOURCES: / The framework ADDS:) agree with its declaration, not that either is true. code_leaks() strips docstrings before searching (they must name the components) and fails open on missing source (a frozen build must still start; CI asserts source is found for every shipped plugin).
2. The product type is declared, not inferred from the id. capabilities(). product is a PriceProduct, rides on the series, and volatility_regime() reads it (8.3: real-time products widen thresholds). A plugin id appears only in PROVIDERS, RETIRED_PROVIDERS and its own class (AC2). No shipped plugin declares REALTIME since TARIFF-07; _WIDE, Uncertainty.REALTIME_PRODUCT and WIDEN_FACTOR's 2.5 are kept, and test_price.py section 10 declares REALTIME from a class outside this file to keep them live.
3. "No price data" is a type, not an empty list. 11.4: no data means static profile, provisional plan, no arbitrage; all-zero prices mean arbitrage is merely pointless. PriceRead.series() refuses an empty list; PriceRead.no_data(reason) carries .reason and nothing else, so min(read.points), for p in read.points and .get("points", []) raise Absence; .match(*, series, no_data) takes both handlers as required keywords. Same shape as sim/riglock.py after TEST-04 (read its Absence block before changing this). Two ways past it survive and are not worth code: vars(read).get("points", []), and bool(read) being True for both variants (a falsy no_data would turn if read: into a guard by accident; use .match).

Timezone without zoneinfo

The Alpine image has no tzdata, so ZoneInfo("Europe/Brussels") would raise on the client's Pi. Every instant is UTC and rendered at its fixed offset, so the October 02:00-03:00 hour that happens twice is two instants (02:00+02:00 and 02:00+01:00); a naive-local-hour key or a 24-slot array is wrong on both transition days (test_price.py counts points on both 2026 transitions). The EU rule: CEST from 01:00 UTC on the last Sunday of March to 01:00 UTC on the last Sunday of October. Review of PR #16 matched it against zoneinfo over 149 040 hourly instants 2020-2036 with zero mismatches. Residual risk is political (EU abolishes the change): one edit in brussels_offset().

The feed range (TARIFF-09)

generic_dayahead.fetch() used to ignore since/until and request the bare URL, so the feed answered ONE Brussels day and 8.3 dropped everything past today as provisional: at 09:00 over a 48 h horizon 15 firm and 33 carried, at 23:00 1 firm and 47 carried. feed_days() returns the Brussels calendar days (first, last, both inclusive; the API takes dates) that the half-open [since, until) touches: Brussels, never UTC (22:30 UTC in summer is already tomorrow); until exclusive, so local midnight belongs to the day before; an empty or inverted window collapses to one day. api.energy-charts.info capture: scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json. dated_feed_url() refuses a base that already has a query string (appending would give ?bzn=BE?bzn=BE&start=...). An operator's price_feed_url override is used verbatim and loses the range, on purpose: rewriting an operator's URL reorders and re-escapes it, and both providers declare a second host (web-api.tp.entsoe.eu) whose parameters are not start/end. The cost is logged at fetch time, naming the horizon consequence. The URL rule is shared by generic_dayahead and custom_formula (TARIFF-15), taking plugin_id as an argument so AC2 holds. The "range is used" promise is not mechanically enforced (a rule would be satisfied by an invented parameter name); test_price.py pins which plugins consume the range as a literal roster.

Absence and the covering window

See TARIFF-14 for the cover table: a point covers from its own start to the next point's start, never more than an hour; the last point gets the span of the one before; sorted at construction. covers() delegates to at() (a second copy of the rule is a second place to be wrong); window_points() grades only the start against [since, until) and is not a coverage query. at() is a linear scan over about 192 entries on the 30 s path; a bisect is not worth a second way to be wrong. A duplicated start (not reachable through load_prices()) gives the earlier entry a zero-width window.

Taxes components (TARIFF-18)

price_taxes_eur_kwh's default (0.06 before) is the sum of FLEMISH_TAXES_COMPONENTS_EUR_KWH, each row one line of the Eneco residential card for Flanders, September 2026, including 6 % VAT, in EUR/kWh; test_price.py section 1b re-adds them. It is per DSO: the distribution row runs 5,28 c (Midden-Vlaanderen) to 6,70 c (West), so the all-in Flemish total runs about 0.117 to 0.131; a site outside Antwerpen types its own. The default sits near the low end on purpose: the scalar cancels in max - min, and its one decision is the 17 negative-price guard in arbitrage.py, which tests the absolute all-in sell price. Too high carries a genuinely negative hour above zero (export into paid-for disposal); too low only declines a trade that would have paid. energiebijdrage is a row of zero: from 1 August 2026 it is merged into the bijzondere accijns, and the zero row makes the double-count a visible decision. Moving a shipped default changes which configs start: TARIFF-17's rung in validate.py refuses price_taxes_profile together with a non-default scalar, comparing against the default (the Supervisor merges every schema key into the stored blob, 70 of 71 keys on the reference install, CHANGELOG 0.9.0), so a superseded shipped default (0.06) must still read as untouched, and the rung's sentence names both figures.

One in-code mirror of option defaults (CFG-03)

Option defaults live in config.yaml's options: and schema: and main.py's .get() fallbacks. PRICE_OPTION_DEFAULTS is the single in-code mirror for the price options: validate.py imports it, main.py has no o.get("price_...", X) fallbacks, no dataclass carries a copy, and test_price.py section 1 reads config.yaml and asserts equality key for key. static_profile (TARIFF-03), price_taxes_profile (TARIFF-17), price_formula (TARIFF-15), price_export_formula (TARIFF-16) and the OPT-04 keys live there for the same reason, not in their own modules (a second dict would be a fourth location). Details:

• The tables / formulas are in FRAMEWORK_KEYS: a plugin reading one could add taxes a second time per hour, or fold an injection curve into a consumption price. BANNED_IN_PLUGIN_CODE derives from that set.
• arbitrage_soc_floor / arbitrage_soc_ceiling are not the maintenance keys (a balancing cycle is not a trading reserve).
• cycle_cost_threshold_eur is 0.10 because 10.1's name says so, not because it makes anything trade: on Belgian day/night numbers the static spread is about 0.08, so a static-tariff site does not trade at this default (test_arbitrage.py section 7). Whether 0.10, or FCST-03's 2.5 / 1.5, is right is an owner decision.
• price_service_token is a secret (SECRET_OPTIONS); scrub() removes secret values from any text built from an error.

Widening (FCST-03)

One path, a set of active uncertainty sources (real-time product 8.3, no weather data 11.4). The combination rule is max, not a product (2.5 x 1.5 would give a threshold nothing on a Belgian tariff clears). The factors are policy defaults, not derived (no arbitrage ever ran); they differ so max, sum and product are distinguishable in a check. widen_factor() refuses a string (a silent 1.0 would report a widening that is not applied).

Injection price (AC4, TARIFF-16)

export_price_eur_kwh() is a separate scalar. It was said to be so because nothing dynamic exists to route; that was false (Eneco publishes `0,1 x BELPEX-H

• 1,188, ENGIE (B x eSpot_15) - A). The true reason: an injection curve has its own coefficients and VAT treatment (injection is VAT-exempt, consumption is 6 %). TARIFF-16 added price_export_formula, read by export_formula() through the same parser as the consumption leg (parse_export_formula()isparse_price_formula()with its option name moved).export_price_at()falls through to the scalar and ignores the index, correctly. A negative answer is a real answer (Eneco's formula is below zero for any index under 11,88 EUR/MWh); nothing floors it. Nothing underapp/calls either reader to take a decision: arbitrage prices the sell side as displaced import. A malformed leg is refused at startup by validate.py;taxes_profile()andexport_formula()still handle it loudly at run time.static_tariffis imported lazily intaxes_profile()` because it imports this module at module scope.

Taxes at each point's own instant (TARIFF-17)

taxes_at() takes a scalar or a period table (duck-typed on price_at, because price.py must not import static_tariff.py). A non-flat component matters: ORES bills dag 13,27 / nacht 7,39 c, a 5,88 c gap coinciding with the cheap spot hours, and flattening it understates the spread. assemble() must evaluate it per point; hoisting the lookup out of the loop collapses the feature back into a level (test_price.py section 24 grades which hour is cheapest, since a spread-magnitude assertion passes on the hoisted version). On a scalar, 0.0 + markup is exactly markup in IEEE 754, so scalar sites get identical bytes. The table wins over the scalar; validate.py refuses configs where both speak.

Retired providers (TARIFF-07)

RETIRED_PROVIDERS exists so eneco_eddy and friends are not refused as typos: they were shipped, the operator chose them, and every failed fetch fell back silently to static_profile. No alias to generic_dayahead: they published retail prices (B x spot + A) and it reads wholesale; an alias would invent the site's two coefficients (behind the klantenzone login; CREG tariff-card pages 404, VREG unreachable). Each value is the id-specific half of the refusal (the endpoint, what it answered on 2026-09-13, why no replacement can be built); validate.py owns the framing. register() refuses at import to register a retired id again, because pid in PROVIDERS going true silences the named refusal; re-shipping is a decision recorded by removing it from RETIRED_PROVIDERS in the same commit. See TARIFF-07.

Parsers (TARIFF-08)

The only parser is columnar: https://api.energy-charts.info/price?bzn=BE gives parallel unix_seconds / price arrays plus unit. The earlier row parser read doc["data"], a shape no feed ever published, graded by fixtures written from the same guess, so every fetch ended SourceUnavailable and 11.4 fell back to static silently. Two shapes, two functions: a parser that sniffs can decide wrong, and a wrong-but-parseable response becomes a price. _points_from_columnar_json() refuses:

• unequal arrays (zip() would truncate the evening peak, and a series missing the expensive hours is traded on while no-data switches arbitrage off);
• a unit other than the declared one (EUR/kWh under the same keys would be 1000 times too small and MAX_PLAUSIBLE_EUR_KWH bounds only the large side);
• naive time: unix_seconds are UTC (fromtimestamp(t, timezone.utc)). Rendering at the Brussels offset is assemble()'s, once, for every plugin.

A null price is dropped and logged (the pairing is reliable, the source said it published none; _Series.at() answers the hole with Absence); scenarios/fetch_epex_be.py dropped 0 nulls in 12 months to 2026-08-31. The feed is quarter-hourly since Belgium's 15-minute MTU (2025-10-01); nothing assumes either granularity (test_price.py section 2b replays both). _validate_raw() accepts negative prices (windy-Sunday hours are the year's best charging; 17 keeps the regression) and rejects NaN, infinity, an implausible magnitude, naive datetimes, and duplicate or out-of-order starts.

Transport and plugin details

• check_host() splits the scheme before parsing the host. The first version stripped len("https://"), which on http:// ate two characters of the host (pi.energy-charts.info), so a non-TLS feed failed the allow-list and the check "plain http is refused" passed even with the TLS rule deleted. Found by mutation in review of PR #16.
• The default transport imports aiohttp lazily so manual_csv works with no HTTP stack. With headers (price_service only) it answers in the safe words of status_words() and follows no redirects (the host is fixed; a redirect would take the token somewhere the allow-list never saw).
• manual_csv: the CSV is all-in (the plugin cannot know what a typed number contains); load_prices() warns when markup / taxes are set non-zero under a plugin that sources them. No transport, aiohttp, network or tz database: it is the offline escape hatch and the TEST-01 replay ingest path.
• custom_formula (TARIFF-15) declares {ENERGY, MARKUP}, not ALL_IN: a card's Tariefformule is the energy component only (Eneco tabulates network tariffs separately; ENGIE excludes network, transport, levies, excise), so ALL_IN would drop about 12 c/kWh. The risk left is an operator folding network charges into offset; the card self-check in custom_formula.py refuses that, and grades the coefficients against the card, not the card against reality. No expression evaluation: the string is settable over HTTP through the ingress config page, so eval would be RCE in a root container holding SUPERVISOR_TOKEN. It reuses generic_dayahead's feed, URL and parser. It keeps scale=1.0, so its RawPoints carry EUR/MWh in a field called eur_per_kwh: contained (destructured on the next line, never leaves the method; _validate_raw() would refuse an escape).
• startup_preview() is a hook (TARIFF-15), not a line in load_prices(): only the plugin knows what went into its arithmetic, and the index beside the all-in figure makes a factor of 1000 obvious. It must not raise into the control path; the id is recorded before the hook runs so a failing preview is not retried every 30 minutes.
• load_prices() returns, never raises, except FrameworkOption (a plugin trying to read the markup; turning it into "no price data" would hide the defect).
• price_provider_id() is one spelling of the selected id because the same expression was written at three call sites (TARIFF-10). validate.py keeps its own reader on purpose: it validates the raw blob.
• price_provider_selected() (TARIFF-10): the background refresh used to start only when arbitrage_enabled (ships false) was on, so a configured site fetched nothing (40 h soak: zero feed requests, present: false on all 802 samples). It is not a trading switch: arbitrage.enabled() answers that.
• price_service (PRICE-01): FALLBACKS and KEEP_LAST are empty/False for every earlier plugin, so price_chain.py is not used for them. TOKEN_REFUSED is the one phrase the fallback chain reads to tell "the owner must act" from "the server is down". covers lists complete days only; an hour missing inside a day is a hole, not a free hour.