diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 3f36bac..653ee1b 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -10,15 +10,30 @@ consumes it yet**: this is the seam, landed first, so that the arbitrage work that follows never learns a supplier's name. -**Double-counting the markup is unreachable rather than discouraged.** A plugin -does not return a price; it returns the figure its source published and -declares which of {energy, markup, taxes} that figure contains. The framework -adds the complement, once. A plugin is never handed `price_markup_eur_kwh` at -all - `PluginOptions` raises on it - so the plugin that adds the margin a second -time cannot be written. Engie and TotalEnergies publish a retail figure with -their margin already inside; adding the configured 3 c/kWh on top would have -made every future arbitrage decision wrong by that amount with nothing -reporting it. +**Double-counting the markup is refused rather than discouraged, given a +correct declaration.** A plugin does not return a price; it returns the figure +its source published and declares which of {energy, markup, taxes} that figure +contains. The framework adds the complement, once. Engie and TotalEnergies +publish a retail figure with their margin already inside; adding the configured +3 c/kWh on top would make every future arbitrage decision wrong by that amount +with nothing reporting it. + +Two mechanisms, because one was not enough. `PluginOptions` raises on the +framework-owned keys - and review of PR #16 then wrote a double-counting plugin +three more ways, one of them touching nothing but `PRICE_OPTION_DEFAULTS`, a +public dict in the module every plugin already imports. So `register()` now also +**refuses at import any plugin whose own code mentions a framework key** +(`code_leaks()`, docstrings stripped via `ast` so a plugin's mandated +`SOURCES:` line does not false-positive). Honest wholesale baseline 0.188 +EUR/kWh; every attack landed 0.219. + +**The guarantee's precise shape**, stated because an overclaim is worse than no +claim: the framework adds the complement exactly once *given a correct `sources` +declaration*. A plugin that declares `{energy}` while its feed is really all-in +still comes out one markup high while touching no framework name, and no code +can catch that - the declaration is the plugin's unverifiable claim about +somebody else's tariff. What is mechanised is reachability; what stays human is +truthfulness. **The product type is declared, not inferred from the plugin's name.** `eneco_eddy` is a real-time/imbalance product and gets widened arbitrage @@ -38,6 +53,13 @@ pointless under the first and forbidden under the second - and an implementation that returns `[]` on a failed fetch cannot tell them apart. Every site-level failure returns `no_data`; nothing raises into the control tick. +**What is not verified, and cannot be by an offline suite:** every fixture is +synthetic, so the five feed URLs and JSON shapes are checked against a belief +about what those suppliers publish, not against the feeds. The parse layer is +exercised; the contract with each supplier is not. Expect the first live fetch +of each to need a shape fix, and note that a wrong shape degrades to +no-price-data rather than to a wrong price. + **`manual_csv` is the documented offline fallback** and needs no network, no supplier account and no HTTP stack. See DOCS.md for the format. Its `start` column must carry a UTC offset - a bare local timestamp is refused, because one diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index 5eb466e..4a5842c 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -617,6 +617,14 @@ feed contains and the add-on adds only the rest. Set a markup under a plugin that already carries one and it is **ignored, with a line in the log saying so** — never applied twice. +That holds **given a correct declaration**, and it is worth knowing where the +line is. What the add-on enforces mechanically is that a plugin cannot obtain +the markup or apply it: the numbers are not reachable from plugin code, and a +plugin whose code mentions them fails to load at all. What it cannot check is +whether a plugin described its supplier's feed truthfully — that is a claim +about somebody else's tariff, and no code can verify it. If you write a plugin, +the `sources` declaration is the part a reviewer has to read. + `eneco_eddy` is a real-time/imbalance product rather than a day-ahead one. It is flagged as high-volatility, which widens arbitrage thresholds once the optimiser lands. That comes from the plugin's own declaration, not from its diff --git a/goodwe_controller/app/price.py b/goodwe_controller/app/price.py index df3e3da..7ac7978 100644 --- a/goodwe_controller/app/price.py +++ b/goodwe_controller/app/price.py @@ -26,7 +26,8 @@ be taken on a provisional price). THE THREE THINGS THIS MODULE IS SHAPED BY ========================================= -1. DOUBLE-COUNTING MARKUP IS MADE UNREACHABLE, NOT DISCOURAGED. +1. DOUBLE-COUNTING MARKUP IS REFUSED, NOT DISCOURAGED - GIVEN A CORRECT + DECLARATION. `generic_dayahead` reads a raw EPEX figure and the user's markup and taxes have to be added to it. Engie and TotalEnergies publish a figure that already HAS the markup in it. Add the configured markup to that one too and every @@ -41,11 +42,25 @@ THE THREE THINGS THIS MODULE IS SHAPED BY * `assemble()` - the framework, never a plugin - adds exactly the components the plugin declared it did NOT source, from `price_markup_eur_kwh` / `price_taxes_eur_kwh`. - * ⚠️ A PLUGIN IS NEVER HANDED THOSE TWO NUMBERS. `fetch()` receives a - `PluginOptions`, which is the options dict with the framework-owned keys - REMOVED - reading `price_markup_eur_kwh` from inside a plugin raises - `FrameworkOption` rather than returning a number. That is what turns - "the plugin must not add markup" from a rule into an inability. + * ⚠️ A PLUGIN CANNOT REACH THOSE TWO NUMBERS BY ANY ROUTE, and the word + "any" was earned the hard way. `PluginOptions` raises `FrameworkOption` + on them, which closes the documented path - but review of PR #16 wrote a + double-counting plugin THREE more ways, one of them touching nothing but + `PRICE_OPTION_DEFAULTS`, a public dict in the module every plugin must + already import for `PriceProvider` and `register`. One closed door in a + room with three is not a guarantee. So `register()` now also REFUSES ANY + PLUGIN WHOSE OWN CODE MENTIONS A FRAMEWORK KEY - see `code_leaks()`. + + ⚠️ THE PRECISE SHAPE OF THE GUARANTEE, because an overclaim here is worse + than no claim: **the framework adds the complement exactly once, GIVEN A + CORRECT `sources` DECLARATION.** A plugin that declares `{ENERGY}` while its + feed is really all-in still comes out one markup high, and NO CODE CAN CATCH + THAT - the declaration is the plugin's unverifiable claim about somebody + else's tariff. What is mechanised is that a plugin cannot obtain the numbers + and cannot apply them; what remains human is whether it described its feed + truthfully. The docstring/declaration check in test_price.py verifies that + the plugin's two claims AGREE, not that either is true, and it is worth + knowing which of those two things you are reading. 2. THE PRODUCT TYPE IS DECLARED, NOT INFERRED FROM THE PLUGIN'S NAME. `eneco_eddy` is a real-time / imbalance product (§9.2) and §8.3 says those @@ -104,10 +119,18 @@ TIMEZONE, AND WHY THERE IS NO `zoneinfo` HERE 2026 transitions and counts the points rather than asserting 24. The EU rule is fixed in law and needs no database: CEST from 01:00 UTC on the -last Sunday of March until 01:00 UTC on the last Sunday of October. +last Sunday of March until 01:00 UTC on the last Sunday of October. Review of +PR #16 cross-checked this against `zoneinfo`'s Europe/Brussels over 149,040 +hourly instants, 2020-2036 inclusive, and found ZERO mismatches - so the +residual risk here is not arithmetic, it is political: if the EU ever abolishes +the seasonal change, `brussels_offset()` needs editing. That is the same +exposure as a machine whose `tzdata` has gone stale, and it is one edit in one +function rather than a package upgrade. """ +import ast import csv +import inspect import io import json import logging @@ -116,6 +139,7 @@ from abc import ABC, abstractmethod from dataclasses import dataclass from datetime import datetime, timedelta, timezone from enum import Enum +from textwrap import dedent _LOG = logging.getLogger("goodwe.price") @@ -425,6 +449,20 @@ class _NoData(PriceRead): `for p in read.points`, `read.get("points", [])` and `len(read.points)` all raise `Absence` here rather than quietly reporting that the cheapest hour of the day costs nothing. + + ⚠️ TWO WAYS PAST IT SURVIVE, BOTH FOUND IN REVIEW OF PR #16, AND NEITHER IS + WORTH CODE: + + * `vars(read).get("points", [])` returns `[]`. `vars()` bypasses + `__getattr__` entirely; nothing short of `__slots__` plus a metaclass + would stop it, and anyone writing `vars()` on a variant type has left + the road on purpose. + * `bool(read)` is `True` for BOTH variants, so `if read:` looks like a + check and is not one. Defining `__bool__` to be False here would make + `if read:` "work" - and that is exactly the wrong fix: it would turn a + line that reads like a guard into one that IS a guard by accident, + which is how `.get(k, default)` got into this codebase in the first + place. Use `.match(...)`. There is no falsy variant on purpose. """ kind = "no_data" @@ -492,8 +530,13 @@ class PluginOptions: """The options dict, minus the keys only the framework may read. Reading `price_markup_eur_kwh` here raises `FrameworkOption` rather than - returning 0.03. That is the difference between "the plugin must not add the - markup" being a convention and it being a thing a plugin cannot do. + returning 0.03. + + ⚠️ THIS IS ONE OF THE TWO DOORS, NOT THE ONLY ONE. `_o` is reachable, and + `price_option()` and `PRICE_OPTION_DEFAULTS` are public names in this + module. This class alone is a convention; it is `register()`'s + `code_leaks()` scan that makes it an inability. Do not weaken either half + on the strength of the other. """ def __init__(self, opts: dict): @@ -581,6 +624,62 @@ class PriceProvider(ABC): """ +# ⚠️ THE NAMES A PLUGIN'S OWN CODE MAY NOT MENTION. FRAMEWORK_KEYS is the +# obvious half; the other three are the routes review of PR #16 actually used to +# write a double-counting plugin after `PluginOptions` was in place: +# +# `_o` - PluginOptions' private dict, reachable as +# `opts._o["price_markup_eur_kwh"]`. +# `price_option` - a PUBLIC framework helper that takes the raw +# options dict and answers with the markup. +# `PRICE_OPTION_DEFAULTS` - a PUBLIC module constant. This is the one that +# decided the review: no private access, no options +# dict, no helper - just a dict in the module every +# plugin already imports for `register`. +# +# Any of them turns +0.088 into +0.119 on a wholesale feed with nothing +# anywhere reporting it. +BANNED_IN_PLUGIN_CODE = tuple(sorted(FRAMEWORK_KEYS)) + ( + "PRICE_OPTION_DEFAULTS", "price_option", "_o") + + +def code_leaks(cls) -> list: + """Framework-owned names appearing in a plugin's own CODE. Empty is clean. + + ⚠️ DOCSTRINGS ARE STRIPPED BEFORE THE SEARCH, AND THAT IS NOT COSMETIC. + Every shipped plugin's docstring is REQUIRED to name the components it + sources, so `engie_dynamic` says "markup" in prose and would false-positive + against `price_markup_eur_kwh` on a raw text scan. The check that fires on + the documentation it also mandates is the check that gets deleted. So the + class is parsed, every docstring node is dropped, and the search runs on + `ast.unparse` of what is left - i.e. on code, which is the only thing that + can actually add a number twice. + + ⚠️ FAIL-OPEN ON MISSING SOURCE, DELIBERATELY. `inspect.getsource` needs the + .py file, which the add-on image has (Dockerfile COPYs `app/`) but a frozen + or zipapp build would not. Refusing to start a battery controller because a + STATIC ANALYSIS could not run would trade a real outage for a hypothetical + accounting error, so an unreadable source returns `[]` and logs. CI proves + the scan really runs: test_price.py asserts source was found for all five + shipped plugins, so a build that silently lost it is a red check, not a + quietly disarmed guard. + """ + try: + src = dedent(inspect.getsource(cls)) + tree = ast.parse(src) + except (OSError, TypeError, IndentationError, SyntaxError) as e: + _LOG.warning("could not read %s's source for the framework-key scan " + "(%s); the double-counting guard is NOT enforced for it", + getattr(cls, "id", cls), e) + return [] + for node in ast.walk(tree): + body = getattr(node, "body", None) + if isinstance(body, list) and body and isinstance(body[0], ast.Expr) and isinstance(body[0].value, ast.Constant) and isinstance(body[0].value.value, str): + body.pop(0) + code = ast.unparse(tree) + return sorted({name for name in BANNED_IN_PLUGIN_CODE if name in code}) + + PROVIDERS: dict = {} @@ -595,6 +694,19 @@ def register(cls): if cls.id in PROVIDERS: raise ValueError(f"duplicate provider id {cls.id!r}") cls.capabilities() # fail at import, not at 15:00 on fetch day + leaks = code_leaks(cls) + if leaks: + # ⚠️ AT IMPORT, NOT AT FETCH. A plugin that can reach the markup is a + # plugin that can bill wrong for a month before anyone notices, so it + # does not get to exist - the add-on fails to start and names the line. + raise FrameworkOption( + f"{cls.id!r} mentions {', '.join(leaks)} in its own code. A plugin " + f"returns what its SOURCE published and declares " + f"capabilities().sources; the framework adds the complement, once. " + f"Reaching any of {BANNED_IN_PLUGIN_CODE} lets it be added twice, " + f"and a price that is one markup high is invisible in every log " + f"this add-on writes. Remove the reference; if the plugin genuinely " + f"needs a per-site number, add a NON-framework option key for it.") PROVIDERS[cls.id] = cls return cls @@ -682,12 +794,33 @@ def check_egress(url: str, caps: ProviderCapabilities) -> str: Returns the host, so a caller that forgets to call this is not the same shape as one that did. + + ⚠️ THE SCHEME IS SPLIT OFF BEFORE THE HOST IS PARSED, AND THE ORDER MATTERS + FOR THE TEST MORE THAN FOR THE CODE. The first version stripped a fixed + `len("https://")` from the front, which on an `http://` URL ate two + characters of the hostname instead - `http://api.energy-charts.info/p` + became the host `pi.energy-charts.info`, which then failed the ALLOW-LIST. + So a non-TLS feed was refused, the check named "plain http is refused" + passed, and deleting the TLS rule entirely left it passing. Review of + PR #16 found that by mutation: an assertion that does not test what it + names, inside a security check - this project's own recurring defect. Now + the scheme is parsed as a scheme, the host of an `http://` URL comes out + correct and DECLARED, and the TLS rule is the only thing that can refuse + it - which is what makes the check able to fail. """ if not url: raise SourceUnavailable("no feed URL configured (price_feed_url)") - if not url.startswith("https://"): - raise SourceUnavailable(f"refusing a non-TLS price feed: {url!r}") - host = url[len("https://"):].split("/", 1)[0].split("@")[-1] + scheme, sep, rest = url.partition("://") + if not sep: + raise SourceUnavailable( + f"not an absolute URL, so no host can be checked against the " + f"§16 allow-list: {url!r}") + if scheme.lower() != "https": + raise SourceUnavailable( + f"refusing a non-TLS price feed: {url!r} uses {scheme!r}. §16 " + f"allow-lists outbound destinations and SEC-01 requires TLS on " + f"every one of them.") + host = rest.split("/", 1)[0].split("?", 1)[0].split("@")[-1] host = host.split(":", 1)[0].lower() if host not in {h.lower() for h in caps.egress_hosts}: raise SourceUnavailable( diff --git a/goodwe_controller/app/validate.py b/goodwe_controller/app/validate.py index c256654..c5833a6 100644 --- a/goodwe_controller/app/validate.py +++ b/goodwe_controller/app/validate.py @@ -435,6 +435,15 @@ def _price_problems(opts: dict) -> list[str]: # operator's evidence that they picked a supplier is a status line saying # they picked a supplier. That is (a), and it is the "silently double up" # the PM named on 2026-08-30. + # ⚠️ THIS REFUSAL'S REASON EXPIRES, AND THE MESSAGE BELOW MUST BE EDITED IN + # THE SAME COMMIT THAT LANDS THE FIRST CONSUMER (TARIFF-03 or OPT-04). + # Today the message says the provider "would decide nothing" - true, because + # nothing reads the series. Once something does, both sources really decide + # and this becomes a straightforward two-writer conflict; the refusal stays + # right and its explanation goes wrong, which is worse than either. There is + # a tripwire on exactly this in test_price.py section 12: it greps app/ for + # a `load_prices` consumer and goes RED the moment one appears. Update the + # message, not the check. ha_price = [k for k in ("price_now_entity", "price_avg_entity") if str(opts.get(k, "") or "").strip()] if pid != "off" and ha_price: diff --git a/goodwe_controller/test_price.py b/goodwe_controller/test_price.py index 20da76a..2ed3d51 100644 --- a/goodwe_controller/test_price.py +++ b/goodwe_controller/test_price.py @@ -36,6 +36,7 @@ TWO RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: agree exactly. """ +import ast import asyncio import inspect import io @@ -46,13 +47,16 @@ import sys from datetime import datetime, timedelta, timezone from app.price import ( - ALL_IN, Absence, Component, FrameworkOption, MAX_PLAUSIBLE_EUR_KWH, + ALL_IN, Absence, BANNED_IN_PLUGIN_CODE, Component, FrameworkOption, + MAX_PLAUSIBLE_EUR_KWH, code_leaks, PRICE_OPTION_DEFAULTS, PROVIDERS, PluginOptions, PricePoint, PriceProduct, PriceProvider, PriceRead, ProviderCapabilities, RawPoint, SourceUnavailable, arbitrage_allowed, assemble, brussels, brussels_offset, check_egress, + price_option, export_price_eur_kwh, load_prices, plan_is_provisional, register, volatility_regime, ) +from app import price as _price from app.validate import validate_startup_config fails = [] @@ -68,6 +72,34 @@ def run(coro): return asyncio.run(coro) +def _reason(fn): + """The message fn() raised, so a check can assert WHICH rule refused. + + `raises(...)` only says that something refused. M7 survived because two + different rules could refuse the same URL and the check could not tell them + apart - so where a check is named for a specific rule, it asserts on the + reason instead. + """ + try: + fn() + except Exception as e: # noqa: BLE001 + return str(e) + return "" + + +def raises(fn, exc): + """True if fn() raises exactly `exc`. A DIFFERENT exception is False, not + True - otherwise "it was refused" and "it broke on the way to being + refused" would grade the same, which is how M7 hid.""" + try: + fn() + except exc: + return True + except Exception: + return False + return False + + class _Raised: """A value a check wanted, whose computation raised. @@ -500,6 +532,147 @@ check("...and the wholesale plugin, which really does use them, does not", "generic_dayahead" not in _warned) +# --------------------------------------------------------------------------- # +print() +print("4b. a plugin cannot reach the markup BY ANY ROUTE (review of PR #16)") +# WHY THIS SECTION EXISTS. The first version of this ticket claimed the markup +# was "not in scope" for a plugin because `PluginOptions` refused it. Review +# wrote a double-counting plugin three more ways, one of them touching nothing +# but `PRICE_OPTION_DEFAULTS` - a public dict in the module every plugin must +# already import for `register`. One closed door in a room with three is not a +# guarantee. Honest wholesale baseline 0.188; each attack lands 0.219. +_banned = set(BANNED_IN_PLUGIN_CODE) +check(f"the banned set covers the framework keys AND the three public routes " + f"review actually used {sorted(_banned)}", + _banned == set(_price.FRAMEWORK_KEYS) | {"PRICE_OPTION_DEFAULTS", + "price_option", "_o"}) + +# code_leaks() fails OPEN when inspect.getsource() cannot find the .py - the +# right trade for a battery controller, and a silently disarmed guard if nobody +# checks. THIS is the check that makes CI prove the guard is armed. +_srclen = derived(lambda: {pid: len(inspect.getsource(cls)) + for pid, cls in PROVIDERS.items()}) +check(f"the scan really read every shipped plugin's source, so its fail-open " + f"path is not silently in use {_srclen}", + isinstance(_srclen, dict) and len(_srclen) == 5 + and all(n > 200 for n in _srclen.values())) +_dirty = derived(lambda: {pid: code_leaks(cls) + for pid, cls in PROVIDERS.items() + if code_leaks(cls)}) +check(f"...and all five shipped plugins are clean {_dirty}", _dirty == {}) + +# THE DOCSTRING STRIP IS LOAD-BEARING, AND THIS PROVES IT RATHER THAN ASSERTING +# IT. Every plugin is REQUIRED to name its sourced components in prose, so +# engie_dynamic's docstring contains "markup" - a raw text scan false-positives +# on the documentation the same file mandates, and a check that fires on +# required documentation is a check somebody deletes. +_raw_hits = derived(lambda: sorted( + pid for pid, cls in PROVIDERS.items() + if any(b in inspect.getsource(cls) for b in _banned))) +check(f"a RAW text scan would false-positive on the mandated docstrings " + f"{_raw_hits}, which is why code_leaks() strips them before searching", + isinstance(_raw_hits, list) and _raw_hits != [] and _dirty == {}) + + +class _AttackPrivateDict(PriceProvider): + """A1: reaches into PluginOptions' private dict. + + SOURCES: energy + The framework ADDS: markup, taxes + """ + + id = "_attack_private_dict" + + @classmethod + def capabilities(cls): + return ProviderCapabilities(product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY})) + + async def fetch(self, o, since, until, transport): + return [RawPoint(T0, 0.10 + o._o["price_markup_eur_kwh"], False)] + + +class _AttackHelper(PriceProvider): + """A2: calls a public framework helper with the raw options dict. + + SOURCES: energy + The framework ADDS: markup, taxes + """ + + id = "_attack_helper" + + @classmethod + def capabilities(cls): + return ProviderCapabilities(product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY})) + + async def fetch(self, o, since, until, transport): + return [RawPoint(T0, 0.10 + price_option(o._o, "price_markup_eur_kwh"), + False)] + + +class _AttackPublicConstant(PriceProvider): + """A2b: touches nothing but a public module constant. THE DECIDING ONE. + + SOURCES: energy + The framework ADDS: markup, taxes + """ + + id = "_attack_public_constant" + + @classmethod + def capabilities(cls): + return ProviderCapabilities(product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY})) + + async def fetch(self, o, since, until, transport): + return [RawPoint(T0, + 0.10 + PRICE_OPTION_DEFAULTS["price_markup_eur_kwh"], + False)] + + +_ATTACKS = {"A1 private dict": _AttackPrivateDict, + "A2 public helper": _AttackHelper, + "A2b public constant": _AttackPublicConstant} +for _name, _cls in _ATTACKS.items(): + _found = derived(lambda c=_cls: code_leaks(c)) + check(f"{_name}: the scan names what it reached for {_found}", + isinstance(_found, list) and _found != []) + check(f"...and register() REFUSES it at import, not at fetch time", + raises(lambda c=_cls: register(c), FrameworkOption)) + check(f"...so it never enters the registry", _cls.id not in PROVIDERS) + +# THE COST OF THE DEFECT, MEASURED, so the refusal is not grading itself. +_honest_wholesale = derived(lambda: assemble( + PROVIDERS["generic_dayahead"].capabilities(), + [RawPoint(T0, 0.10, False)], MARKUP, TAXES)[0].eur_per_kwh) +_attacked = derived(lambda: assemble( + _AttackPublicConstant.capabilities(), + [RawPoint(T0, 0.10 + MARKUP, False)], MARKUP, TAXES)[0].eur_per_kwh) +check(f"an unrefused attack would bill {_attacked} against an honest " + f"{_honest_wholesale} - one markup high, every hour, silently", + isinstance(_attacked, float) and isinstance(_honest_wholesale, float) + and abs((_attacked - _honest_wholesale) - MARKUP) < 1e-9) + +# WHAT THIS DOES *NOT* CLOSE, AND CANNOT. A plugin that declares {ENERGY} while +# its feed is really all-in comes out one markup high while touching no +# framework name at all. No code can check that: the declaration is the +# plugin's unverifiable claim about somebody else's tariff. Stated here so the +# guarantee is read at its true size - the complement is added exactly once, +# GIVEN A CORRECT DECLARATION - and so a reviewer who trusts an absolute +# guarantee does not stop checking. Measured, not asserted: +_misdeclared = derived(lambda: assemble( + ProviderCapabilities(product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY})), + [RawPoint(T0, 0.10 + MARKUP + TAXES, False)], MARKUP, TAXES)[0].eur_per_kwh) +check(f"a MIS-DECLARED all-in feed still doubles ({_misdeclared} vs " + f"{_honest_wholesale}) with no framework name touched - the limit of the " + f"guarantee, stated rather than hidden", + isinstance(_misdeclared, float) + and abs((_misdeclared - _honest_wholesale) - (MARKUP + TAXES)) < 1e-9 + and code_leaks(_AttackPrivateDict) != []) + + # --------------------------------------------------------------------------- # print() print("5. product type is declared, not inferred from the name (AC2, trap 2)") @@ -536,8 +709,6 @@ check("volatility_regime refuses a plugin NAME, which is the AC2 violation " # plugin id at all - if it does, the sixth supplier is an edit to price.py's # shared machinery rather than a new class. Derived: build the set of ids that # actually occur, and assert it is empty. -from app import price as _price # noqa: E402 - _framework = [_price.assemble, _price.load_prices, _price.volatility_regime, _price.arbitrage_allowed, _price.plan_is_provisional, _price.check_egress, _price._validate_raw, _price.PriceRead, @@ -606,16 +777,6 @@ check("...and the cheapest hour is computable for the zero series", derived(lambda: min(p.eur_per_kwh for p in _zero.points)) == 0.0) -def raises(fn, exc): - try: - fn() - except exc: - return True - except Exception: - return False - return False - - check("no_data has no .points, so `min(read.points)` cannot invent a free hour", raises(lambda: _none.points, Absence)) check("...no .product to switch a threshold regime on", @@ -630,6 +791,54 @@ check("...and it is not iterable, so `for p in read:` cannot be an empty loop", check("...while its one real field, the reason, is there for an operator", _none.reason == "the feed timed out") +# M3, INHERITED FROM RIG-01 AND CLOSED HERE. `_NoData.__init__(self, +# reason="no price data")` used to go green: a required field with a DEFAULT is +# the exact idiom this whole type exists to forbid, and it survived because +# every check passed a reason explicitly. It is the identical survivor left +# open on sim/riglock.py's `_Unread.cause` - the shape was borrowed including +# its blind spot, which is what borrowing a shape does if nobody re-runs the +# mutations against the copy. +# +# Checked by SIGNATURE rather than by one call, so any default added to any of +# these constructors later is red without somebody remembering to write a new +# case. +def _defaulted(fn): + """Parameters of `fn` that carry a default. `self` is skipped.""" + try: + ps = list(inspect.signature(fn).parameters.values()) + except (TypeError, ValueError) as e: # pragma: no cover + return _Raised(e) + return [q.name for q in ps + if q.name != "self" and q.default is not inspect.Parameter.empty] + + +_ctors = {"PriceRead.no_data": PriceRead.no_data, + "PriceRead.series": PriceRead.series, + "_NoData.__init__": _price._NoData.__init__, + "_Series.__init__": _price._Series.__init__, + "PriceRead.match": PriceRead.match} +_with_defaults = derived(lambda: {n: d for n, d in + ((n, _defaulted(f)) for n, f in _ctors.items()) + if d}) +check(f"no constructor or handler of an outcome type carries a DEFAULT, so a " + f"reading can never be built out of nothing {_with_defaults}", + _with_defaults == {}) +check("...so no_data() with no reason at all is a TypeError at the call site", + raises(lambda: PriceRead.no_data(), TypeError)) +check("...and the variant class itself cannot be built bare either", + raises(lambda: _price._NoData(), TypeError)) +check("...nor can a series be built without its product or provenance", + raises(lambda: _price._Series([PricePoint(brussels(T0), 0.2, False)]), + TypeError)) +# The positive arm, so the four above are not passing because everything raises. +check("...while a fully specified no_data is still perfectly constructible", + PriceRead.no_data("the feed timed out").reason == "the feed timed out") + +# `match` must also refuse POSITIONAL handlers - keyword-only is what makes a +# forgotten arm a TypeError rather than a silently swapped one. +check("match() refuses positional handlers, so the two arms cannot be swapped", + raises(lambda: _none.match(lambda r: 1, lambda r: 2), TypeError)) + check("an EMPTY series cannot be constructed at all, so 'no prices' can never " "wear the shape of an answer", raises(lambda: PriceRead.series([], PriceProduct.DAY_AHEAD, "x", T0), @@ -828,6 +1037,34 @@ check("...and its canonical starts are 25 distinct strings", # --------------------------------------------------------------------------- # print() print("9. the boundary rejects what cannot be a price, and only that") +# M5. Deleting the `Component.ENERGY not in sources` guard used to go green, +# and the consequence is not cosmetic: `adds` becomes {ENERGY, TAXES}, +# `assemble()` never adds an energy component it thinks it is adding, and the +# emitted all-in price silently omits the energy it is supposed to be pricing. +check("capabilities() refuses a feed that declares no ENERGY component", + raises(lambda: ProviderCapabilities( + product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.MARKUP})), ValueError)) +check("...including a completely empty sources set", + raises(lambda: ProviderCapabilities(product=PriceProduct.DAY_AHEAD, + sources=frozenset()), ValueError)) +check("...and an unknown component is refused rather than ignored", + raises(lambda: ProviderCapabilities( + product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY, "grid_fees"})), ValueError)) +check("...and a product given as a string is refused at the declaration too", + raises(lambda: ProviderCapabilities(product="dynamic_dayahead", + sources=frozenset({Component.ENERGY})), + ValueError)) +# THE CONSEQUENCE, derived across every shipped plugin rather than asserted on +# one: the framework must never be in a position to "add" the energy component, +# because there is nothing to add it from. +_adds_energy = derived(lambda: sorted( + pid for pid, cls in PROVIDERS.items() + if Component.ENERGY in cls.capabilities().adds)) +check(f"so no shipped plugin ever asks the framework to invent an energy " + f"price {_adds_energy}", _adds_energy == []) + _caps = PROVIDERS["generic_dayahead"].capabilities() @@ -867,9 +1104,44 @@ check("a declared https host is allowed", check("an undeclared host is refused even over TLS", raises(lambda: check_egress("https://evil.example/p", _g), SourceUnavailable)) -check("plain http is refused", - raises(lambda: check_egress("http://api.energy-charts.info/p", _g), - SourceUnavailable)) +# M7, AND THE ASSERTION IS THE FIX, NOT ONLY THE PARSER. Deleting the TLS rule +# used to leave this check GREEN: the old parser stripped a fixed +# len("https://") from the front, so `http://api.energy-charts.info/p` became +# the host `pi.energy-charts.info`, which failed the ALLOW-LIST instead. The +# feed was refused, the check named "plain http is refused" passed, and the +# rule it was named for could be deleted with no consequence - an assertion +# that does not test what it names, inside a security check. +# +# Three checks now, and the first two are what make the third able to fail: +# prove the host is declared over https, prove it parses identically over +# http, and only then assert that http is refused AND that the reason is TLS. +_tls_url = "http://api.energy-charts.info/p" +check("the same host over https is allowed, so it IS on the allow-list", + derived(lambda: check_egress("https://api.energy-charts.info/p", _g)) + == "api.energy-charts.info") +check("...and the http form parses to that same declared host, so nothing but " + "the TLS rule can refuse it", + derived(lambda: _tls_url.partition("://")[2].split("/", 1)[0]) + == "api.energy-charts.info") +_tls_reason = derived(lambda: _reason(lambda: check_egress(_tls_url, _g))) +check(f"plain http is refused, and refused BY THE TLS RULE - {_tls_reason!r}", + isinstance(_tls_reason, str) and "non-TLS" in _tls_reason) +check("a scheme that is neither is refused the same way, by TLS not by host", + "non-TLS" in str(derived(lambda: _reason( + lambda: check_egress("ftp://api.energy-charts.info/p", _g))))) +# ⚠️ M7b, WHICH I FOUND IN MY OWN FIX FOR M7 AND WHICH IS THE SAME DEFECT ONE +# LAYER DOWN. This check used to read `raises(..., SourceUnavailable)` and +# survived deleting the `if not sep` guard: with no guard, +# "api.energy-charts.info/p".partition("://") leaves scheme = the WHOLE string, +# so the TLS rule refuses it and the check named for the scheme guard passes. +# Refused either way - but the diagnostic an operator gets degrades from "not an +# absolute URL" to "uses 'api.energy-charts.info/p'" as a scheme, which is +# gibberish. Assert the reason, like M7 above. +_noscheme = derived(lambda: _reason( + lambda: check_egress("api.energy-charts.info/p", _g))) +check(f"a URL with no scheme is refused BY THE SCHEME GUARD, with a diagnostic " + f"that names the real problem - {_noscheme!r}", + isinstance(_noscheme, str) and "not an absolute URL" in _noscheme) check("a userinfo prefix cannot smuggle a declared host past the check", raises(lambda: check_egress( "https://api.energy-charts.info@evil.example/p", _g), @@ -953,6 +1225,56 @@ check("the Home Assistant entities alone still start, so nothing is taken away " validate_startup_config(vbase(price_now_entity="sensor.now", price_avg_entity="sensor.avg"), -5000, 5000) == []) +# ⚠️ A TRIPWIRE ON A REASON, NOT ON A BEHAVIOUR, AND IT IS AIMED AT WHOEVER +# LANDS TARIFF-03 OR OPT-04. Rule 7 refuses "provider + HA entities" and its +# stated reason is that the provider series decides NOTHING while the HA pair +# quietly keeps deciding. The refusal stays correct forever; its EXPLANATION +# expires the moment a consumer appears - at that point both really do decide, +# and it becomes a genuine two-writer conflict rather than a silently-ignored +# one. A future reader would then be trusting a reason that is no longer the +# reason, which is the same defect TEST-06's boundary comment names. +# +# So: assert that nothing in app/ consumes the series yet. When you wire +# load_prices() into the optimiser this goes RED, and the fix is to update the +# refusal message in validate.py IN THE SAME COMMIT - not to delete this check. +_APP = os.path.join(os.path.dirname(os.path.abspath(__file__)), "app") +# ⚠️ AST, NOT A SUBSTRING, AND THE FIRST DRAFT PROVED WHY. A text scan for +# "load_prices" matched the COMMENT in validate.py that explains this very +# tripwire, so the check went red against its own documentation - the same +# false-positive class as the plugin docstrings in section 4b, hit twice in one +# ticket. Comments and docstrings are not in an AST, so a name/import walk sees +# code and only code. +def _imports_or_calls(path, name): + try: + tree = ast.parse(open(path, encoding="utf-8").read()) + except (OSError, SyntaxError): + return False + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and any(a.name == name + for a in node.names): + return True + if isinstance(node, ast.Name) and node.id == name: + return True + if isinstance(node, ast.Attribute) and node.attr == name: + return True + return False + + +_consumers = derived(lambda: sorted( + f for f in os.listdir(_APP) + if f.endswith(".py") and f != "price.py" + and _imports_or_calls(os.path.join(_APP, f), "load_prices"))) +check(f"no module consumes load_prices() yet, so rule 7's stated reason - 'the " + f"provider decides nothing' - is still TRUE {_consumers}. When this goes " + f"red, update that message in validate.py in the same commit", + _consumers == []) +# ...and the other half: the registry IS on the startup path, which is what +# makes an unwired seam anchored rather than dead code. +check("...while validate.py does import the registry, so the seam is exercised " + "on every boot", + derived(lambda: "PROVIDERS" in open( + os.path.join(_APP, "validate.py"), encoding="utf-8").read()) is True) + check("manual_csv with no path is refused - the offline fallback with no file " "is §11.4 reached by accident", why(validate_startup_config(vbase(price_provider="manual_csv",