From eda6d0aed104851210ecdddc9c89c047c119112b Mon Sep 17 00:00:00 2001 From: glenn schrooyen Date: Sun, 1 Nov 2026 01:17:46 +0100 Subject: [PATCH] MAINT-20 part 1: custom_formula.py comments cut; new TARIFF-16 note Co-Authored-By: Claude --- .../DECISIONS/TARIFF-16-the-injection-leg.md | 102 +++ goodwe_controller/app/custom_formula.py | 610 +++--------------- 2 files changed, 194 insertions(+), 518 deletions(-) create mode 100644 goodwe_controller/DECISIONS/TARIFF-16-the-injection-leg.md diff --git a/goodwe_controller/DECISIONS/TARIFF-16-the-injection-leg.md b/goodwe_controller/DECISIONS/TARIFF-16-the-injection-leg.md new file mode 100644 index 0000000..9dee873 --- /dev/null +++ b/goodwe_controller/DECISIONS/TARIFF-16-the-injection-leg.md @@ -0,0 +1,102 @@ +# TARIFF-16 - the injection leg of a dynamic contract (`price_export_formula`) + +**Status: SHIPPED.** Tests: `test_price.py` (17 series). Written down by +MAINT-20 from `app/custom_formula.py` and `app/price.py`. Follows [[TARIFF-15]], +whose parser it reuses. + +## Why a formula, not the scalar + +`price.py` modelled injection as one scalar. The market publishes a line a +scalar cannot hold: + + Eneco Zon & Wind Dynamisch 0,1 x BELPEX-H - 1,188 c EUR/kWh + ENGIE Dynamic (B x eSpot_15) - A + Aspiravi (non-dynamic) 0,07 x Belpex - 2 + +1. Its own factor (Eneco 0,102 on consumption, 0,100 on injection; Aspiravi 0,07 + against about 0,10): not the consumption `B` with a sign flip. +2. A negative offset (`- 1,188`, `- 2`); `price_markup_eur_kwh: float(0,1)` + cannot express one. +3. VAT-exempt ("Injectietarieven zijn vrijgesteld van btw") against 6 % on + consumption: representable only because `vat_pct` is per spec. +4. It goes negative with no floor: Eneco's is below zero for any BELPEX-H under + 11,88 EUR/MWh, the hours a PV household exports in. You can be charged to + export. + +`price_export_formula` is `custom_formula`'s parser with one argument moved +(`option`, a row of `_LEGS`), as TARIFF-17 did for `parse_static_profile()`. A +second parser would be a second `_finite()`, unknown-key rule, unit table and +`to_resolution()`, diverging first on the 23- and 25-hour days. Every earlier +call site passes no `option` and reads byte-identical sentences. The flat scalar +`export_price_eur_kwh()` is untouched and still answers when the option is "". + +## What it does not affect + +`arbitrage.py` does not read the injection price at all (9.3: near-zero Belgian +injection; it prices the sell side as displaced import), so no trade today turns +on this number and it is not a wrong-trade fix. It changes what an operator is +told and what a future export-aware strategy (OPT-06) rests on. + +## The card cannot always grade it + +Eneco's injection formula is printed with no price beside it. The `card` block +is required on the consumption leg and optional here (`FormulaSpec.card_graded` +is how callers ask; `card_discrepancy()` raises on an ungraded spec rather than +returning 0.0). The self-check runs wherever a printed estimate exists, and +validate.py says at startup that a leg is ungraded. Inventing an estimate would +grade the arithmetic against itself (MUTATION.md habit 4). + +## `implausible_injection()`: bounds on the coefficients, not the price + +The consumption band (`implausible_all_in()`) cannot be reused: it adds +`price_taxes_eur_kwh`, a consumption component, and its floor `energy >= 0.02` +would refuse the real market's negative hours. But that floor catches a whole +card read 100x too small, which the card check cannot see (it cancels). A band +spanning zero cannot catch 0.00132 EUR/kWh, so the bound moves onto the +coefficients, index-free and sign-free: + +- pass-through `factor x UNITS[units] / 0.001` (one EUR/MWh is 0.001 EUR/kWh): + 1.00 for Eneco's injection leg, 0.70 for Aspiravi's; 100x off reads 0.01 or 100; +- offset magnitude, on the absolute value (1,188 c = 0.01188; a hundredfold error + is 1.188 EUR/kWh). + +Two rungs can both answer one stimulus, so they are graded on their reasons and +mutated together (MUTATION.md, TEST-16); test_price.py 17d builds fixtures so +each is the only one firing. It bounds the shape of the coefficients, never +their truth. + +## The consumption band's asymmetry (`implausible_all_in()`) + +The low end is graded on the energy component and the high end on the all-in +figure. `price_taxes_eur_kwh` ships at 0.11736 (TARIFF-18; 0.06 before), nearly +six times the low bound, so a card read 100x too small (0.0016 EUR/kWh) is +carried over the floor by the flat tax addend alone and would pass a band applied +to the all-in total. Raising the default made the trap wider. The ceiling is on +the all-in figure because that is what the site is billed. It grades the card's +own point, not every hour: live EPEX hours may fall outside (a negative hour is +the best charging opportunity of the year, 17 keeps a regression case). It is +called from validate.py only, since it reads the framework-owned taxes number. + +## Smaller rules + +- `CARD_TOLERANCE` is 0.01 in the card's units, the last digit both real cards + print (Eneco 15,55 against 15.5471; ENGIE 7,522 against 7.5218). Every error + class it exists for is at least two orders of magnitude bigger. +- The card age warning (about 6 months) is never a refusal: ENGIE freezes a + contract's coefficients, so an old card can be exactly right. +- `units` is required with no default (a default is the factor-1000 class). + A zero factor is a fixed tariff in a dynamic one's clothes (every spread 0, + arbitrage never fires; 14's warning box), so the operator is pointed at the + first-class way to say one price. A negative factor inverts the market (charge + into the evening peak). `vat_pct: 0` is legitimate (injection is exempt; some + cards are quoted VAT-inclusive); the bound exists for the 106 / 1.06 / 6 + confusion. An unknown key is refused, not ignored, so a user who types "expr" + is not priced on the fields they also typed. A non-finite factor would make + every price NaN; `True` is an `int` and would price the day at the index. +- `to_resolution()`: the bucket is a UTC hour, exact for Belgium (CET and CEST + are whole-hour offsets, so keying by UTC cannot collide the two 02:00s). It + decides how many points are emitted, not how long one covers ([[TARIFF-14]]). An + incomplete hour is averaged over what is there, not dropped or zero-filled. +- The startup preview prints the index that went in beside the price that came + out (`61.96 -> 0.13522`): every error this module refuses is invisible on the + status page, and a factor of 1000 is one glance there. diff --git a/goodwe_controller/app/custom_formula.py b/goodwe_controller/app/custom_formula.py index b233295..e6f88c8 100644 --- a/goodwe_controller/app/custom_formula.py +++ b/goodwe_controller/app/custom_formula.py @@ -1,174 +1,18 @@ """TARIFF-15 - the customer's own tariff card, as two fixed coefficients. -WHAT THIS IS FOR, in one sentence: a Belgian dynamic contract prices energy as -`B x index + A` with coefficients that exist in exactly one place - the tariff -card in the customer's klantenzone - so the customer types those two numbers -and the add-on does every unit conversion and the VAT multiplication itself. +A Belgian dynamic contract prices energy as `B x index + A` with coefficients +that exist only on the customer's card. The customer types those two numbers; +the add-on does every unit conversion and the VAT multiplication. Fixed +coefficients, no expression evaluation of any kind (the string is settable over +HTTP, and an expression cannot be graded against the card). An unknown key is +refused, never ignored. validate.py refuses to start unless the formula +reproduces the card's printed estimate within CARD_TOLERANCE: that grades the +coefficients against the card, not the card against reality. -WHY A FORMULA AND NOT FOURTEEN MORE PLUGINS (§9.2, TARIFF-07) -============================================================= -There is NO machine-readable source of Belgian tariff coefficients anywhere. -CREG's consumer tariff-card pages 404, CREG Scan is a JS-only wizard, the -annual report carries the supplier list and no per-product coefficients, and -VREG was unreachable. Every coefficient found in research came off a supplier's -own PDF, one of them image-only. So there is nothing to scrape and a -per-supplier plugin would have no source for its numbers - fourteen plugins -would be fourteen copies of the same invention, which is the defect TARIFF-07 -withdrew three plugins for. THE CUSTOMER IS THE ONLY PARTY WHO CAN SEE THEIR -OWN COEFFICIENTS, and the card they read them from is the only artefact that -can grade them. - -THE SHAPE IS THE MARKET'S PUBLISHED ARITHMETIC, NOT OUR APPROXIMATION -==================================================================== -Both suppliers whose 2026 cards were readable print the formula, the reference -index they assumed AND the resulting price, so the arithmetic below is -reproducible against something they published rather than against our own -guess: - - Eneco (0.102 x 133.9905 + 1.00) x 1.06 = 15.5471 card prints 15,55 - ENGIE (0.100 x 61.96 + 0.90) x 1.06 = 7.5218 card prints 7,522 - -Two independent suppliers, reproduced to the last printed digit, in -c EUR/kWh. test_price.py section 13 runs both through the SHIPPED code. - -⚠️ THE DOUBLE-COUNTING MECHANISM, AND WHY THIS DESIGN IS SAFE AT ALL -==================================================================== -This provider declares `sources = {ENERGY, MARKUP}` and NOT `ALL_IN`. A Belgian -card's `Tariefformule` line is THE ENERGY COMPONENT ONLY: Eneco puts network -tariffs in one table and levies in a second, and ENGIE states in so many words -that the formula "does not include fixed network/distribution costs, transport, -grid levies, or excise duties". Declaring ALL_IN would lie on behalf of every -user who typed what their card says, and would silently drop ~12 c EUR/kWh. - -price.py already mechanises that a PLUGIN cannot double-count (`PluginOptions`, -`FRAMEWORK_KEYS`, `code_leaks()`). This provider introduces a risk none of those -can see, because it is not in the code: THE OPERATOR double-counting, by folding -their network charges into `offset` while `price_taxes_eur_kwh` carries them as -well. `code_leaks()` scans code. This is config. - -So the card is the grader: - - At startup validate.py computes - `(factor x card.index_eur_per_mwh + offset) x (1 + vat_pct/100)` and - REFUSES TO START unless it equals `card.expect_incl_vat` to within - CARD_TOLERANCE, naming the discrepancy. - -An `offset` inflated by network charges CANNOT PASS: the card's printed estimate -is by construction the energy component alone, so adding ~12 c EUR/kWh moves -15.55 to ~27 and the add-on refuses. The same check catches VAT omitted, VAT -applied twice, a transposed digit, and a UNIT MISMATCH BETWEEN THE TWO SIDES - -coefficients read off the formula line while the expected price is typed in -another convention, which is the factor-10/100/1000 class as it actually -happens. - -⚠️ THE ONE UNIT ERROR THE CARD CHECK CANNOT SEE IS THE ONE APPLIED TO BOTH -SIDES AT ONCE, and saying so is the difference between a guard and a slogan: -scaling the coefficients AND the expected price by the same 100 cancels out of -a comparison between them. That is what `implausible_all_in()` is for, and it -is a SEPARATE refusal on a SEPARATE quantity - the card's own point, all-in, -against the band a Belgian retail tariff can occupy. - -⚠️ AND THE LIMIT, IN THE SAME BREATH AS THE CLAIM, exactly as price.py's trap 1 -states its own: THE CHECK GRADES THE COEFFICIENTS AGAINST THE CARD, NOT THE CARD -AGAINST REALITY. A consistent-but-wrong pair - both numbers copied off the wrong -product, or off last year's card - reproduces its own printed price perfectly -and passes. Nothing here or anywhere else can catch that; what is mechanised is -that the two coefficients and the printed estimate AGREE, which is the whole of -the unit/VAT/decimal-point class and none of the "wrong contract" class. - -⚠️ FIXED COEFFICIENTS. NO EXPRESSION EVALUATION OF ANY KIND. -============================================================ -All 24 surveyed Belgian products, both injection legs and aWATTar AT fit -`B x index + A`, so an expression buys nothing the market needs. Against that: - - * the add-on serves `POST config/save` -> `POST /addons/self/options` on its - own ingress page, so this string is SETTABLE OVER HTTP. `eval` on it is - remote code execution in a root container holding SUPERVISOR_TOKEN, with - `homeassistant_api` and `hassio_api`, driving an inverter over RS485. - (`eval(s, {"__builtins__": {}})` is not a sandbox and never was.) - * ⚠️ AND THE DECIDING ARGUMENT IS NOT SECURITY - AN EXPRESSION IS - UNVERIFIABLE. Two coefficients can be graded against the card's printed - estimate. An arbitrary expression cannot be graded against anything, so it - makes the invisible-wrong-coefficient problem STRICTLY WORSE, which is the - exact failure this provider exists to prevent. - -Future shapes become NAMED FIELDS - `clamp_lo`, `factor_negative`, -`offset_bands` - each individually refusable and each individually gradeable -against a card. Never a grammar. `parse_price_formula()` therefore REFUSES AN -UNKNOWN KEY rather than ignoring it, so a user who types `"expr"` is told it -does not exist instead of being silently priced on the fields they also typed. -test_price.py section 13f walks this module's AST and fails if `eval`, `exec`, -`compile`, `__import__` or `literal_eval` ever appears in it. - -WHERE THE NUMBERS COME FROM AND WHERE THEY GO -============================================= -`index` names the reference series. `epex_be` is the EPEX BE day-ahead curve in -EUR/MWh, which is what `generic_dayahead` already fetches - same feed, same -`brussels_date_range()` request, same columnar parser, same declared egress -hosts. No new parser and no new failure mode. - -`resolution` says which series the contract is written against: - - * `quarter_hourly` - the 96 points the feed publishes since Belgium's - 15-minute MTU (2025-10-01), emitted as they are; - * `hourly` - the four quarters of each hour averaged into one point. - -⚠️ NEITHER IS AN HOURLY ASSUMPTION IN `_Series.at()`'s SENSE (TARIFF-14). The -hourly option changes how many POINTS are emitted and their spacing; the window -each point covers is still derived from its neighbours by `_cover_table()`, and -nothing here tells anything downstream how long a point lasts. A feed that is -already hourly averages one member per hour, which is the identity - so the -option cannot silently assume the granularity it was handed. - -`units` is NOT bureaucracy. Two suppliers publish the same shape in different -units and the market aggregator prints both conventions in one column, so the -add-on has to be told which one the card is in - and the unit conversion is the -add-on's job precisely because it is one of the two steps that silently costs a -factor of 10, 100 or 1000. The other is VAT, which is also the add-on's. - -TARIFF-16 - THE SECOND LEG, AND WHY IT IS THIS PARSER AND NOT A SECOND ONE -========================================================================== -A Belgian dynamic contract has TWO published lines, not one, and the second is -what the site is PAID for export: - - Eneco Zon & Wind Dynamisch 0,1 x BELPEX-H - 1,188 c EUR/kWh - ENGIE Dynamic (B x eSpot_15) - A - Aspiravi (non-dynamic) 0,07 x Belpex - 2 - -Same `B x index + A` shape - which is why `price_export_formula` is this -module's parser with ONE argument moved (`option`), exactly as TARIFF-17 made -`parse_static_profile()`'s option name a parameter rather than forking the -period grammar. What differs between the two legs is FOUR numbers and a -handful of sentences, and all of them are rows of `_LEGS`: - - * its OWN factor - Eneco 0,102 on consumption against 0,100 on injection, - Aspiravi 0,07 against ~0,10. NOT the consumption `B` with a sign flip; - * a NEGATIVE offset, which `factor`/`offset` already allow and which - `price_markup_eur_kwh: float(0,1)` cannot express at all; - * `vat_pct: 0` - "Injectietarieven zijn vrijgesteld van btw" - against 6 on - consumption, which is a SIX PERCENT ASYMMETRY between the buy leg and the - sell leg and is representable only because `vat_pct` is PER SPEC; - * a price that goes BELOW ZERO with no floor, in the hours a PV household - exports. `implausible_injection()` is the band that accepts that and still - refuses an implausible one; `implausible_all_in()` is NOT it, and its - docstring says why. - -⚠️ AND THE CARD CANNOT ALWAYS GRADE IT. Eneco's injection formula is printed -with NO price beside it, so on that card there is nothing to reproduce. The -`card` block is therefore OPTIONAL on this leg and REQUIRED on the other, the -self-check runs wherever there is a printed estimate, and validate.py says at -startup when a leg is ungraded. Inventing an estimate to grade against would -be grading arithmetic against itself, which is the defect MUTATION.md habit 4 -is about; TARIFF-15 demonstrated two tariffs end to end rather than three for -the same reason. - -⚠️ NOTHING TRADES ON THIS TODAY, AND SAYING SO IS PART OF SHIPPING IT. -`arbitrage.py` does not read `export_price_eur_kwh` at all - its own comment -says so, citing §9.3's near-zero Belgian injection price - and both hours it -compares are all-in CONSUMPTION prices. So this is not a wrong-trade defect -and must not be described as one. What it changes is what an operator is told, -and what a future export-aware strategy would rest on. -""" +The provider declares `{ENERGY, MARKUP}`, never `ALL_IN` (a card's formula is +the energy component only). TARIFF-16 added the injection leg through this same +parser. DECISIONS/TARIFF-15-the-card-is-the-grader.md and +DECISIONS/TARIFF-16-the-injection-leg.md.""" import json import math @@ -183,13 +27,11 @@ class FormulaInvalid(Exception): validate.py at startup - never at 03:00 on a fetch.""" -# --------------------------------------------------------------------------- # -# The enumerations. Every one of these is a CLOSED set, and that is the point: -# an unknown value is refused by name rather than guessed at. -# --------------------------------------------------------------------------- # +# The enumerations. Each is a CLOSED set: an unknown value is refused by name, +# not guessed. -# unit -> how many EUR/kWh one of it is. ⚠️ THE WHOLE FACTOR-1000 CLASS LIVES -# HERE, so the table is tiny, explicit, and has no "default". +# unit -> EUR/kWh per one of it. The whole factor-1000 class lives here: tiny, +# explicit, no default. UNITS: dict = { "c_eur_per_kwh": 0.01, # eurocent per kWh - Eneco and ENGIE print this "eur_per_kwh": 1.0, # some aggregators print this @@ -198,54 +40,42 @@ UNITS: dict = { RESOLUTIONS: tuple = ("hourly", "quarter_hourly") -# index id -> the human name of the series it refers to. One entry today; a -# second index is a new row here plus a fetch path, not a new grammar. +# index id -> the series it names. A second index is a new row here plus a +# fetch path. INDEXES: dict = { "epex_be": "Belgian day-ahead price, EUR/MWh", } -# ⚠️ 0.01 IN THE CARD'S OWN UNITS, WHICH IS THE LAST DIGIT BOTH REAL CARDS -# PRINT. Eneco prints 15,55 against a computed 15.5471 (0.0029 out) and ENGIE -# prints 7,522 against a computed 7.5218 (0.0002 out), so the tolerance has to -# absorb the card's own rounding and nothing more. Every error class this check -# exists for - a factor of 10, of 100, of 1000, a missing or doubled 1.06, a -# transposed digit, ~12 c EUR/kWh of network charge folded into `offset` - is -# at least two orders of magnitude bigger than this. +# 0.01 in the card's own units, the last digit both real cards print (Eneco +# 15,55 vs 15.5471; ENGIE 7,522 vs 7.5218): the card's rounding and nothing +# more. +# DECISIONS/TARIFF-16-the-injection-leg.md. CARD_TOLERANCE = 0.01 -# The all-in plausibility band, EUR/kWh. `MAX_PLAUSIBLE_EUR_KWH` in price.py -# bounds only prices that are too LARGE; this is the other end, and it is the -# one the factor-1000 error falls off. +# The all-in plausibility band, EUR/kWh: the low end, where a factor-1000 error +# falls off. PLAUSIBLE_LO_EUR_KWH = 0.02 PLAUSIBLE_HI_EUR_KWH = 1.00 -# ~6 months. ⚠️ A WARNING, NEVER A REFUSAL: ENGIE freezes a contract's -# coefficients for its duration, so a card two years old can be exactly right -# and a site refused over one would be a site that cannot start. +# ~6 months. A WARNING, never a refusal: ENGIE freezes a contract's +# coefficients. CARD_STALE_DAYS = 183 _CARD_KEYS = ("month", "index_eur_per_mwh", "expect_incl_vat") _TOP_KEYS = ("index", "resolution", "factor", "offset", "units", "vat_pct", "card") -# --------------------------------------------------------------------------- # # TARIFF-16 - the SECOND leg, and the one parameter that carries it -# --------------------------------------------------------------------------- # EXPORT_FORMULA_OPTION = "price_export_formula" -# ⚠️ THE PASS-THROUGH FRACTION, WHICH IS WHY THE NEXT TWO BOUNDS CAN BE -# INDEX-FREE. One EUR/MWh IS 0.001 EUR/kWh, so `factor x UNITS[units] / 0.001` -# is the dimensionless share of the wholesale price the contract passes -# through - 1.00 for Eneco's injection leg (0,100 c EUR/kWh per EUR/MWh), 0.70 -# for Aspiravi's (0,07). A coefficient typed a hundred times too small or too -# large moves this by a hundred, whatever the index and whatever the sign of -# the price. +# The pass-through fraction, `factor x UNITS[units] / 0.001`: index-free and +# sign-free, so a coefficient typed 100x off moves it by 100 whatever the +# price. EUR_KWH_PER_EUR_MWH = 0.001 -# The two plausibility bounds for an INJECTION leg. ⚠️ NEITHER LOOKS AT THE -# SIGN OF THE PRICE, AND THAT IS THE WHOLE POINT - see -# `implausible_injection()`. +# The two plausibility bounds for an INJECTION leg; neither looks at the sign +# of the price. PASSTHROUGH_LO = 0.10 # 10% of the wholesale price PASSTHROUGH_HI = 10.0 # 1000% of it INJECTION_OFFSET_MAX_EUR_KWH = 0.20 @@ -254,18 +84,8 @@ INJECTION_OFFSET_MAX_EUR_KWH = 0.20 @dataclass(frozen=True) class _Leg: """What ONE option's coefficients price, as sentences its refusals read. - - ⚠️ TARIFF-16 MADE `option` A PARAMETER RATHER THAN FORKING THE PARSER, - exactly as TARIFF-17 made `option`/`means` parameters of - `parse_static_profile()` rather than forking the period grammar. A second - copy of this parser would be a second `_finite()`, a second unknown-key - rule, a second unit table and a second `to_resolution()` - and the two - would diverge first on the 23- and 25-hour days, where nobody looks. - Everything that differs between the consumption leg and the injection leg - is a STRING in this table; the arithmetic, the enumerations, the card - comparison and the resolution rule are ONE implementation with one set of - checks over it. - """ + `option` is a parameter, not a fork of the parser (TARIFF-16, as TARIFF-17 did + for `parse_static_profile()`).""" option: str what: str # "consumption" / "injection", for a sentence @@ -316,12 +136,8 @@ _LEGS = { @dataclass(frozen=True) class FormulaSpec: - """One contract's published arithmetic. SEVEN numbers and no code. - - ⚠️ EVERY FIELD IS A FIXED, NAMED SCALAR. There is deliberately no field - that holds an expression, a callable, a lambda or a template, and there is - no place to put one: `parse_price_formula()` refuses an unrecognised key. - """ + """One contract's published arithmetic: SEVEN numbers and no code. Every field is + a fixed named scalar; `parse_price_formula()` refuses an unrecognised key.""" index: str resolution: str @@ -332,8 +148,8 @@ class FormulaSpec: card_month: str = "" # "YYYY-MM", as the card is dated card_index_eur_per_mwh: float = None card_expect_incl_vat: float = None # in `units`, as the card prints it - # TARIFF-16. Which OPTION this spec was read from, so a refusal, a log line - # and `card_graded` name the leg the operator actually typed. + # TARIFF-16. Which OPTION this spec was read from, so refusals name the leg + # typed. option: str = "price_formula" @property @@ -342,24 +158,14 @@ class FormulaSpec: @property def card_graded(self) -> bool: - """Is there a printed estimate to grade these coefficients against? - - ⚠️ FALSE IS A REAL, SHIPPED STATE ON THE INJECTION LEG and is reported - rather than defaulted: Eneco's injection formula is printed with no - price beside it. Never False on `price_formula`, where `_card()` - refuses an absent block. - """ + """Is there a printed estimate to grade these coefficients against? False is a + real state on the injection leg (Eneco prints none); never False on `price_formula`.""" return self.card_expect_incl_vat is not None @property def passthrough(self) -> float: - """The dimensionless share of the wholesale price this passes through. - - `factor x UNITS[units] / 0.001`. 1.00 for Eneco's injection leg, 0.70 - for Aspiravi's. ⚠️ INDEX-FREE AND SIGN-FREE, which is what lets - `implausible_injection()` bound a leg whose price legitimately goes - below zero. - """ + """The dimensionless share of the wholesale price this passes through. Index-free + and sign-free, which lets `implausible_injection()` bound a leg that goes negative.""" return self.factor * self.unit_to_eur_kwh / EUR_KWH_PER_EUR_MWH @property @@ -368,46 +174,27 @@ class FormulaSpec: return self.offset * self.unit_to_eur_kwh -# --------------------------------------------------------------------------- # # The arithmetic. Two functions, and the SECOND one is the contract. -# --------------------------------------------------------------------------- # def in_card_units(spec: FormulaSpec, index_eur_per_mwh: float) -> float: - """`(factor x index + offset) x (1 + vat_pct/100)`, in the card's units. - - This is the line the card prints, spelled exactly as the card spells it, - and it is the only place the coefficients are combined. The card - self-check and the price path both go through here, so a mutation to it - cannot move the price without also moving the self-check. - """ + """`(factor x index + offset) x (1 + vat_pct/100)`, in the card's units: the line + the card prints. The only place the coefficients are combined, so the card + check and the price path cannot diverge.""" return ((spec.factor * float(index_eur_per_mwh) + spec.offset) * (1.0 + spec.vat_pct / 100.0)) def price_eur_kwh(spec: FormulaSpec, index_eur_per_mwh: float) -> float: - """The energy+markup price for one index value, in EUR/kWh. - - ⚠️ THE UNIT CONVERSION IS THE ADD-ON'S JOB AND IT HAPPENS HERE, ONCE. The - user types what their card prints, in the card's own units; every step - from there to EUR/kWh is code, because those steps are exactly the ones - that silently cost a factor of 10, 100 or 1000 when a human does them. - """ + """The energy+markup price for one index value, in EUR/kWh. The unit conversion + happens here, once: those steps are the ones that cost a factor of 10, 100 or 1000.""" return in_card_units(spec, index_eur_per_mwh) * spec.unit_to_eur_kwh def card_discrepancy(spec: FormulaSpec) -> float: - """How far the coefficients land from the card's own printed estimate. - - In the card's units, signed: positive means the formula computes MORE than - the card prints, which is the direction an `offset` carrying network - charges moves it. - - ⚠️ RAISES ON A SPEC WITH NO CARD RATHER THAN RETURNING 0.0 (TARIFF-16). An - ungraded injection leg has no printed estimate, and a discrepancy of zero - is exactly what "the coefficients reproduce the card" looks like - so a - silent zero here would report the strongest possible result for the one - state in which nothing was compared at all. - """ + """How far the coefficients land from the card's printed estimate, in the card's + units, signed: positive means MORE than the card prints (the direction an + `offset` carrying network charges moves it). Raises on a spec with no card + rather than returning 0.0 (TARIFF-16).""" if not spec.card_graded: raise FormulaInvalid( f"{spec.option} has no card block, so there is no printed " @@ -417,18 +204,10 @@ def card_discrepancy(spec: FormulaSpec) -> float: - spec.card_expect_incl_vat -# --------------------------------------------------------------------------- # # Reading the option -# --------------------------------------------------------------------------- # def _finite(value, field: str, option: str = "price_formula") -> float: - """A real, finite number - or a refusal naming the field. - - ⚠️ SAME CLASS AS `_validate_raw()`'s NaN RULE, one layer earlier. A NaN - factor makes every price NaN, every comparison False, and an arbitrage - strategy that silently never fires; `True` is an `int` in Python and would - price the whole day at the index. - """ + """A real, finite number, or a refusal naming the field (NaN would price every hour NaN; `True` is an `int`).""" if isinstance(value, bool) or not isinstance(value, (int, float)): raise FormulaInvalid( f"{option}.{field} is {value!r}, which is not a number. Write a " @@ -441,15 +220,9 @@ def _finite(value, field: str, option: str = "price_formula") -> float: def _card(raw, leg: _Leg = None) -> tuple: - """The `card` block: the estimate the coefficients are graded against. - - ⚠️ REQUIRED ON THE CONSUMPTION LEG AND OPTIONAL ON THE INJECTION ONE, and - the asymmetry is a fact about the CARDS rather than a relaxed rule. See - `_LEGS[EXPORT_FORMULA_OPTION].card_note`: Eneco's injection formula is - printed with no price beside it, so on that card there is nothing to grade - against. `("", None, None)` is the ungraded answer and - `FormulaSpec.card_graded` is how every caller asks. - """ + """The `card` block: the estimate the coefficients are graded against. Required on + the consumption leg, optional on the injection one because Eneco prints none + (`("", None, None)`; `FormulaSpec.card_graded`).""" if leg is None: leg = _LEGS["price_formula"] if raw is None: @@ -491,31 +264,8 @@ def _month_date(month: str): def parse_price_formula(value, *, option: str = "price_formula"): - """`price_formula` -> a `FormulaSpec`, or None when unset. - - ⚠️ `option` IS TARIFF-16'S WHOLE REUSE, and it is TARIFF-17's move. It - selects a row of `_LEGS` and nothing else: the enumerations, the - arithmetic, `to_resolution()`, the unknown-key rule and the card - comparison are shared, and every pre-existing call site - which passes no - `option` at all - lands on the consumption row and reads byte-identical - sentences. `parse_export_formula()` is a two-line wrapper, not a fork. - - Accepts what the option can actually arrive as, exactly as - `parse_static_profile()` does: - - * `None` / `""` / `"null"` - unset. - * a JSON string - what config.yaml's flat `str?` option - carries on a Supervisor install. - * a dict - already parsed, from a future nested - schema, from sim/inject.py, or a check. - - Raises `FormulaInvalid`; never returns a half-built spec. - - ⚠️ THE CARD SELF-CHECK IS PART OF PARSING, not a separate step a caller - may forget. Both callers - validate.py's startup gate and the plugin's own - fetch - go through this one function, so there is no route to a price that - has not been graded against the card. - """ + """`price_formula` -> a `FormulaSpec`, or None when unset. `option` selects a row of + `_LEGS`; everything else is shared, so `parse_export_formula()` is a wrapper.""" leg = _LEGS[option] if value is None: return None @@ -534,12 +284,8 @@ def parse_price_formula(value, *, option: str = "price_formula"): f"{option} is {type(value).__name__}, not a JSON object with " f"{', '.join(_TOP_KEYS)}.") - # ⚠️ AN UNKNOWN KEY IS REFUSED, NOT IGNORED, AND THIS IS THE HALF THAT - # KEEPS THE "NEVER A GRAMMAR" PROMISE HONEST. A future shape is a NAMED - # field - clamp_lo, factor_negative, offset_bands - added here and - # refusable on its own. Ignoring unknown keys would mean a user who typed - # "expr" or "formula" gets priced on the fields they also typed, with the - # one they believed in dropped in silence. + # An unknown key is refused, not ignored: a future shape is a NAMED field, + # never a grammar. unknown = sorted(set(value) - set(_TOP_KEYS)) if unknown: raise FormulaInvalid( @@ -559,10 +305,8 @@ def parse_price_formula(value, *, option: str = "price_formula"): f"{option}.resolution is {value.get('resolution')!r}. Choose one " f"of {list(RESOLUTIONS)}, as your tariff card says.") - # ⚠️ `units` IS REQUIRED AND HAS NO DEFAULT. A default here would be the - # factor-1000 class with a friendly face: the add-on would silently assume - # a convention the card never stated, and the card self-check would then - # be graded in units nobody chose. + # `units` is required with no default: a default is the factor-1000 class + # with a friendly face. units = str(value.get("units", "") or "").strip() if units not in UNITS: raise FormulaInvalid( @@ -573,29 +317,22 @@ def parse_price_formula(value, *, option: str = "price_formula"): offset = _finite(value.get("offset"), "offset", option) vat_pct = _finite(value.get("vat_pct"), "vat_pct", option) - # ⚠️ ZERO IS NOT "NO INDEXATION", IT IS A FIXED TARIFF WEARING A DYNAMIC - # ONE'S CLOTHES. Every hour comes out at the same price, so every spread is - # 0, arbitrage can never clear any threshold, and a strategy that never - # fires is indistinguishable from one that is switched off (§14's warning - # box). The add-on already has a first-class way to say "one price": point - # the operator at it rather than let them build it by accident. + # Zero is a fixed tariff in a dynamic one's clothes (arbitrage never + # fires); point the operator at the first-class fixed price. if factor == 0.0: raise FormulaInvalid( f"{option}.factor is 0, so every hour gets the same {leg.what} " f"price{leg.zero_factor}. For one flat price, {leg.flat}.") - # ⚠️ A NEGATIVE FACTOR INVERTS THE MARKET, and the direction is the danger: - # the dearest hour becomes the cheapest one, so the battery is told to - # charge into the evening peak and to discharge at 03:00. That is not a - # mispriced site, it is a site actively trading against itself. + # A negative factor inverts the market: charge into the evening peak, + # discharge at 03:00. if factor < 0.0: raise FormulaInvalid( f"{option}.factor is {factor:g}, which turns the market upside " f"down. {leg.neg_factor}.") - # ⚠️ 0 IS LEGITIMATE AND MUST STAY SO: injection is VAT-exempt in Belgium, - # and some cards are already quoted VAT-inclusive. The bound exists for the - # 106 / 1.06 / 6 confusion, not to make VAT compulsory. + # 0 is legitimate (injection is VAT-exempt; some cards are VAT-inclusive); + # the bound is for the 106 / 1.06 / 6 confusion. if not 0.0 <= vat_pct <= 30.0: raise FormulaInvalid( f"{option}.vat_pct is {vat_pct:g}. Write a percentage between 0 " @@ -607,13 +344,10 @@ def parse_price_formula(value, *, option: str = "price_formula"): card_month=month, card_index_eur_per_mwh=card_index, card_expect_incl_vat=card_expect, option=option) - # ⚠️ THE CARD SELF-CHECK STILL RUNS WHEREVER THERE IS A CARD TO RUN IT - # AGAINST, on either leg, with the same tolerance. What TARIFF-16 adds is - # the honest answer for the leg where the card prints no estimate: NOT a - # weakened comparison and NOT an estimate computed here, but two bounds on - # the COEFFICIENTS that need no printed price - see - # `implausible_injection()` - and a startup line saying the leg is - # ungraded. + # The card self-check still runs wherever there is a card, with the same + # tolerance. Where the card prints no estimate, `implausible_injection()` + # bounds + # the coefficients and a startup line says the leg is ungraded. if spec.card_graded: delta = card_discrepancy(spec) if abs(delta) > CARD_TOLERANCE: @@ -626,60 +360,15 @@ def parse_price_formula(value, *, option: str = "price_formula"): def parse_export_formula(value): - """`price_export_formula` -> the INJECTION leg's `FormulaSpec`, or None. - - TARIFF-16. THE DEFECT THIS EXISTS TO CLOSE: `price.py` modelled injection - as ONE scalar, and the market publishes a line with four properties a - scalar cannot hold. Verbatim off the cards: - - Eneco Zon & Wind Dynamisch 0,1 x BELPEX-H - 1,188 c EUR/kWh - "Het injectietarief is dynamisch en kan per uur wijzigen" - - identical on the January and the September 2026 cards - ENGIE Dynamic (B x eSpot_15) - A c EUR/kWh - Aspiravi (non-dynamic) 0,07 x Belpex - 2 - - 1. ITS OWN FACTOR. Eneco uses 0,102 on consumption and 0,100 on - injection; Aspiravi 0,07 against ~0,10 on its consumption leg. It is - NOT the consumption `B` with a sign flip. - 2. A NEGATIVE OFFSET - `- 1,188`, `- 2`. Note that - `price_markup_eur_kwh: float(0,1)` cannot express a negative markup - at all, so the offset has no home on the consumption side either. - 3. VAT-EXEMPT. "Injectietarieven zijn vrijgesteld van btw", while - consumption is at 6% - a SIX PERCENT ASYMMETRY between the buy leg - and the sell leg, which a single scalar cannot even state. - 4. IT GOES NEGATIVE, WITH NO FLOOR. Eneco's injection price is negative - for any BELPEX-H below 11,88 EUR/MWh, and a PV household exports - precisely in the hours most likely to be negative. You can be charged - to export. - - ⚠️ AND WHAT THIS DOES **NOT** AFFECT TODAY, SAID HERE SO NOBODY READS THE - ABOVE AS A TRADING BUG. `arbitrage.py` does not read - `export_price_eur_kwh` at all - its own comment says so, and gives §9.3's - near-zero Belgian injection price as the reason the economically sound - trade on this market is charge-cheap-to-displace-expensive-IMPORT. Both - hours it compares are all-in CONSUMPTION prices. So no trade the add-on - takes today is decided by this number, and this card is not a correction - of a wrong trade. What it changes is what an operator is told, and what - any future export-aware strategy would rest on: OPT-06 and any export - logic would otherwise inherit a scalar that cannot represent the market, - and the 6% VAT asymmetry would be invisible in the arithmetic. - - ⚠️ THE SCALAR IS NOT REPLACED. `export_price_eur_kwh()` is untouched and - still answers for every site with a flat injection price, which is every - site that exists today; this option ships "". That is `price.py`'s own - prediction coming true rather than a new design - "when it does land, an - injection curve gets a `PriceProvider` of its own and this scalar becomes - its FIXED-product default; the seam is already the right shape for that." - """ + """`price_export_formula` -> the INJECTION leg's `FormulaSpec`, or None (TARIFF-16). + Same parser with `option` moved; the flat `export_price_eur_kwh()` still answers + when unset. Nothing trades on it today. DECISIONS/TARIFF-16-the-injection-leg.md.""" return parse_price_formula(value, option=EXPORT_FORMULA_OPTION) def card_refusal(spec: FormulaSpec, delta: float) -> str: - """The sentence an operator reads when their formula misses its own card. - - It names both numbers and what to check. The usual causes are in DOCS.md, - because each of these errors looks like a working tariff on the status page. - """ + """The sentence an operator reads when their formula misses its own card: both + numbers and what to check (causes are in DOCS.md).""" got = in_card_units(spec, spec.card_index_eur_per_mwh) leg = _LEGS[spec.option] return ( @@ -688,53 +377,14 @@ def card_refusal(spec: FormulaSpec, delta: float) -> str: f"{spec.card_expect_incl_vat:g}. Check {leg.card_causes}.") -# --------------------------------------------------------------------------- # # The two questions the PLUGIN cannot ask, because they need framework numbers -# --------------------------------------------------------------------------- # def implausible_all_in(spec: FormulaSpec, taxes_eur_kwh: float): - """The card's own point, if its LEVEL cannot be a Belgian retail price. - - Returns a refusal sentence or None. ⚠️ CALLED FROM validate.py AND FROM - NOWHERE ELSE, because it reads a FRAMEWORK-owned number: a plugin that - could see `taxes` could add it a second time, which is price.py's trap 1. - - ⚠️ THE LOW END IS GRADED ON THE ENERGY COMPONENT AND THE HIGH END ON THE - ALL-IN FIGURE, AND THAT ASYMMETRY IS THE CHECK RATHER THAN AN OVERSIGHT. - `price_taxes_eur_kwh` ships at 0.11736 (TARIFF-18; it was 0.06 before, and - the argument was already decisive at that figure), which is nearly SIX - times the low bound - so a card read a HUNDRED times too small comes out at - 0.0016 EUR/kWh of energy, is carried over the floor by the flat tax addend - alone, and passes a band applied to the all-in total. The hundredfold error - is precisely what this exists to catch, so the floor is applied where it - can still see it. ⚠️ Raising the default made this trap WIDER, not - narrower: the addend that does the carrying got bigger. The ceiling is - applied to the all-in figure because that is the number the site is really - billed at, and taxes only push it up. - - ⚠️ AND THE LIMIT, WITH THE CLAIM. This grades the CARD'S OWN POINT - one - price, at the monthly index the card assumed - and not every hour of every - day. A live EPEX hour can and should fall outside this band (a negative - hour is the best charging opportunity of the year, and §17 keeps a - regression case for it), so the band is deliberately NOT applied to the - fetched series. What it catches is a formula whose whole LEVEL is wrong, - which is the factor-of-1000 class arriving from the end - `MAX_PLAUSIBLE_EUR_KWH` does not bound. - - ⚠️ AND IT IS THE CONSUMPTION LEG'S BAND ONLY - TARIFF-16 DELIBERATELY DID - NOT POINT IT AT THE INJECTION LEG. Two reasons, and the second is an - acceptance criterion rather than a preference: - - * it adds `taxes_eur_kwh`, and `price_taxes_eur_kwh` is a CONSUMPTION - component. Nothing is added to what a site is PAID for export, so the - `all_in` half would be adding a number the injection leg never carries; - * its floor is `energy >= 0.02 EUR/kWh`, and a REAL Eneco injection price - is negative for any BELPEX-H below 11,88 EUR/MWh - the hours a PV - household actually exports in. Applied to injection this floor would - refuse the market, and refuse it hardest in the hours where knowing the - number matters most. `implausible_injection()` is the injection band - and it is deliberately NOT this function with a lower floor. - """ + """The card's own point, if its LEVEL cannot be a Belgian retail price. Refusal + sentence or None. Called from validate.py only (it reads the framework-owned + taxes). The low end is graded on the energy component and the high end on the + all-in figure; the consumption leg's band only. It grades the card's own point, + not the fetched series. DECISIONS/TARIFF-16-the-injection-leg.md.""" energy = price_eur_kwh(spec, spec.card_index_eur_per_mwh) all_in = energy + float(taxes_eur_kwh) if energy >= PLAUSIBLE_LO_EUR_KWH and all_in <= PLAUSIBLE_HI_EUR_KWH: @@ -747,49 +397,10 @@ def implausible_all_in(spec: FormulaSpec, taxes_eur_kwh: float): def implausible_injection(spec: FormulaSpec): - """The INJECTION leg's plausibility band. Returns a refusal or None. - - ⚠️ A NEGATIVE INJECTION PRICE IS ACCEPTED, NOT REFUSED, AND NEITHER RUNG - BELOW LOOKS AT THE SIGN OR THE LEVEL OF A PRICE. Eneco's published line - `0,1 x BELPEX-H - 1,188` is negative for any BELPEX-H under 11,88 EUR/MWh; - those are the midday hours a PV household exports in, so a band with a - floor above zero - which is what `implausible_all_in()` has, for the - consumption leg's own good reasons - would refuse the real market and - would hide it in exactly the hours it costs money. You CAN be charged to - export. The add-on says so rather than declining to represent it. - - THE PROBLEM THAT LEAVES, STATED BEFORE THE SOLUTION. The consumption - floor was not decoration: it is what catches a whole card read a hundred - times too small, the one unit error the card self-check cannot see because - it cancels out of a comparison between the two sides. A band that spans - zero cannot catch it - 0.00132 EUR/kWh is inside any interval that - contains -0.03. So the bound moves OFF the price and ONTO the - coefficients, where it is index-free and sign-free: - - * PASS-THROUGH. `factor x UNITS[units] / 0.001` is the dimensionless - share of the wholesale price the contract passes through, because one - EUR/MWh IS 0.001 EUR/kWh. Eneco's injection leg is 1.00 and Aspiravi's - is 0.70. A factor typed 100x too small reads 0.01 and 100x too large - reads 100, whatever the index and whatever the sign of the result. - * OFFSET MAGNITUDE. Eneco's is 1,188 c EUR/kWh = 0.01188 EUR/kWh and - Aspiravi's is 2 c = 0.02. A hundredfold error is 1.188 EUR/kWh, which - is not an injection tariff in any direction. The bound is on the - ABSOLUTE value, so a negative offset of a believable size passes and - that is the normal case on this leg. - - ⚠️ TWO RUNGS THAT CAN BOTH ANSWER ONE STIMULUS, SO THEY ARE GRADED ON - THEIR REASONS AND MUTATED TOGETHER (MUTATION.md, TEST-16). A whole card - read 100x too large trips both. The fixtures in test_price.py 17d are - built so that each rung is the ONLY one that fires on its own stimulus. - - ⚠️ AND WHAT THIS CANNOT SEE, WITH THE CLAIM (habit 13). It bounds the - SHAPE of the coefficients, never their truth: a contract really written - against a different index, or last year's coefficients, passes both rungs - and reproduces any card typed beside them. Where a card DOES print an - estimate the self-check grades the pair against it; where it does not - - which is Eneco's injection line - these two bounds are the whole of what - is mechanised, and validate.py says so at startup. - """ + """The INJECTION leg's plausibility band. Refusal or None. A negative price is + accepted; the bounds are on the coefficients (pass-through and offset + magnitude), since a band spanning zero cannot catch a card 100x too small. It + bounds the shape, not the truth. DECISIONS/TARIFF-16-the-injection-leg.md.""" if not PASSTHROUGH_LO <= spec.passthrough <= PASSTHROUGH_HI: return ( f"{spec.option}.factor passes through {spec.passthrough:.4g}x the " @@ -806,10 +417,7 @@ def implausible_injection(spec: FormulaSpec): def stale_card(spec: FormulaSpec, today=None): - """A warning when the card is older than ~6 months, or None. NEVER a - refusal: ENGIE freezes a contract's coefficients for its duration, so an - old card can be exactly right and refusing one would stop the add-on on a - correctly configured site.""" + """A warning when the card is older than ~6 months, or None. Never a refusal (ENGIE freezes coefficients).""" dated = _month_date(spec.card_month) now = today or datetime.now(timezone.utc).date() if dated is None or (now - dated).days <= CARD_STALE_DAYS: @@ -820,36 +428,14 @@ def stale_card(spec: FormulaSpec, today=None): f"contract changed since, copy the new figures from your supplier.") -# --------------------------------------------------------------------------- # # Resolution -# --------------------------------------------------------------------------- # def to_resolution(pairs, resolution: str) -> list: """`[(instant, index EUR/MWh)]` -> the series the CONTRACT is written on. - - `quarter_hourly` hands the feed's own points back, ascending. - `hourly` averages each hour's members into one point AT THE TOP OF THE - HOUR. - - ⚠️ THE BUCKET IS A UTC HOUR AND THAT IS EXACT FOR BELGIUM, not an - approximation: CET and CEST are both WHOLE-hour offsets, so a Brussels hour - and a UTC hour start at the same instant on every day of the year including - both transition days. Keying by a naive local hour is what collides the two - 02:00s in October; keying by the UTC instant cannot. - - ⚠️ AND THIS IS NOT `_Series.at()`'s HOURLY ASSUMPTION COMING BACK - (TARIFF-14). It decides how many points are EMITTED, not how long one - covers - the coverage window is still derived from the neighbours in - `_cover_table()`. A feed that already publishes hourly points averages one - member per hour, i.e. the identity, so this cannot silently assume the - granularity it was handed. - - ⚠️ AN INCOMPLETE HOUR IS AVERAGED OVER WHAT IS THERE rather than dropped or - zero-filled. A horizon that starts at 09:20 has two quarters in its first - hour; their mean is the best statement available about that hour, and - dropping it would leave a hole `at()` answers with `Absence` for an hour - the feed did publish. - """ + `quarter_hourly` returns the feed's points, `hourly` averages each UTC hour + (exact for Belgium: whole-hour offsets) at the top of the hour. An incomplete + hour is averaged over what is there. It sets how many points are emitted, not + how long one covers (TARIFF-14).""" ordered = sorted(pairs, key=lambda r: r[0].astimezone(timezone.utc)) if resolution == "quarter_hourly": return list(ordered) @@ -863,28 +449,16 @@ def to_resolution(pairs, resolution: str) -> list: return [(k, sum(v) / len(v)) for k, v in sorted(buckets.items())] -# --------------------------------------------------------------------------- # # The startup preview -# --------------------------------------------------------------------------- # PREVIEW_HOURS = 24 PREVIEW_MAX_ROWS = 24 def preview_lines(spec: FormulaSpec, rows) -> list: - """`[(instant, index EUR/MWh, all-in EUR/kWh)]` -> lines for the log. - - THE DEFECT THIS EXISTS TO MAKE VISIBLE: every error this module refuses is - one the operator cannot see from the status page, and the two numbers that - would have shown it - the index that went IN and the price that came OUT - - are never printed beside each other anywhere else. A factor of 1000 is - obvious in one line of `61.96 -> 0.13522` and invisible in a plan. - - ⚠️ IT PRINTS A BOUNDED NUMBER OF ROWS AND SAYS SO. At `quarter_hourly` the - next 24 h is 96 points; the cap is a row COUNT rather than "every fourth - point", because sampling every fourth point would assume the granularity - TARIFF-14 spent a card removing the assumption of. - """ + """`[(instant, index EUR/MWh, all-in EUR/kWh)]` -> lines for the log: the index + that went IN beside the price that came OUT. Prints a bounded row COUNT, not + every fourth point (that would assume the granularity).""" if not rows: return [] shown = rows[:PREVIEW_MAX_ROWS]