TARIFF-02 review: close the three other doors to the markup, and stop overclaiming
checks / unit (pull_request) Successful in 2m34s

Review of PR #16 wrote a double-counting plugin four ways. PluginOptions
blocked one. The one that decided it touched nothing but
PRICE_OPTION_DEFAULTS - a public dict in the module every plugin must
already import for register() - so "the number is not in scope" was
simply false, and an absolute guarantee a reviewer trusts is worse than
a stated limit, because they stop checking.

register() now refuses, AT IMPORT, any plugin whose own code mentions a
framework-owned name: FRAMEWORK_KEYS plus PRICE_OPTION_DEFAULTS,
price_option and _o. code_leaks() parses the class and drops every
docstring node before searching, because every plugin is REQUIRED to
name its sourced components in prose - engie_dynamic says "markup" in
its docstring and a raw text scan false-positives on the documentation
the same file mandates. A check that fires on required documentation is
a check somebody deletes. test_price.py asserts that a raw scan really
would have false-positived, so the strip is proven load-bearing rather
than asserted.

It fails OPEN if inspect.getsource() cannot find the .py, which the
add-on image always can. Refusing to start a battery controller because
a static analysis could not run would trade a real outage for a
hypothetical accounting error - so instead the checks assert source was
found for all five shipped plugins, and a build that lost it is a red
check rather than a quietly disarmed guard.

All three reachable attacks are written out as plugins in the suite and
each must be refused at registration. The cost is measured, not
asserted: honest wholesale 0.188, every attack 0.219 - one markup high,
every hour, invisible in every log this add-on writes.

The guarantee is now stated at its true size everywhere it appears: the
framework adds the complement exactly once, GIVEN A CORRECT sources
DECLARATION. A plugin that declares {ENERGY} while its feed is all-in
still doubles while touching no framework name, and no code can catch
that - the declaration is an unverifiable claim about somebody else's
tariff. Measured in the suite (0.276) so the limit is a number rather
than a caveat. The docstring/declaration check verifies that a plugin's
two claims AGREE, not that either is true; that now says so.

The three surviving mutations:

M3 - _NoData's reason taking a default. This is the identical survivor
left open on RIG-01's _Unread.cause: the shape was borrowed from
sim/riglock.py including its blind spot, which is what borrowing a shape
does if nobody re-runs the mutations against the copy. Closed by
SIGNATURE - no constructor or handler of an outcome type may carry a
default - so the next one is red without anyone writing a new case.

M5 - the ENERGY-required guard was untested. Its consequence is that
adds becomes {ENERGY, TAXES} and the emitted price silently omits the
energy it is pricing.

M7 - deleting the TLS refusal went green AND the check named for it
still passed, because the old parser stripped a fixed len("https://")
and mangled http://api.energy-charts.info/p into the host
pi.energy-charts.info, which then failed the ALLOW-LIST. An assertion
that does not test what it names, inside a security check. Fixed on both
sides: the scheme is parsed as a scheme, so an http URL now resolves to
its real and declared host and only the TLS rule can refuse it; and the
check asserts the REASON. Fixing it surfaced M7b - the no-scheme guard
was survivable the same way - found by mutating my own fix, and closed
the same way.

Also carried forward: rule 7's refusal is correct forever but its stated
reason ("the provider decides nothing") expires the moment TARIFF-03 or
OPT-04 lands. test_price.py section 12 now greps app/ by AST for a
load_prices consumer and goes red when one appears, telling whoever
lands it to update the message in the same commit. The first draft of
that tripwire used a substring and matched the comment in validate.py
explaining the tripwire - the same false-positive class as the plugin
docstrings, hit twice in one ticket, and fixed the same way.

Documented rather than coded around: vars(read).get("points", []) and
bool(read) both still get past the absence type, and neither is worth
code - defining __bool__ would turn a line that reads like a guard into
one that IS a guard by accident, which is how .get(k, default) got here
in the first place.

Stated plainly in DOCS/CHANGELOG/PR: every fixture is synthetic, so the
five feed URLs and JSON shapes are verified against a belief about what
those suppliers publish rather than against the feeds. A wrong shape
degrades to no-price-data, not to a wrong price.

Review's own DST verification recorded: 149,040 hourly instants,
2020-2036, zero mismatches against zoneinfo. The residual is political,
not arithmetic, and the docstring now says so.

24 mutations driven through the suite, 0 survivors. Whole add-on suite
green. No deleted lines outside this ticket's own files; main.py still
untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC
This commit is contained in:
2026-08-30 19:35:36 +02:00
co-authored by Claude Opus 5
parent 852fc7ced6
commit e102cde841
5 changed files with 531 additions and 37 deletions
+31 -9
View File
@@ -10,15 +10,30 @@
consumes it yet**: this is the seam, landed first, so that the arbitrage work
that follows never learns a supplier's name.
**Double-counting the markup is unreachable rather than discouraged.** A plugin
does not return a price; it returns the figure its source published and
declares which of {energy, markup, taxes} that figure contains. The framework
adds the complement, once. A plugin is never handed `price_markup_eur_kwh` at
all - `PluginOptions` raises on it - so the plugin that adds the margin a second
time cannot be written. Engie and TotalEnergies publish a retail figure with
their margin already inside; adding the configured 3 c/kWh on top would have
made every future arbitrage decision wrong by that amount with nothing
reporting it.
**Double-counting the markup is refused rather than discouraged, given a
correct declaration.** A plugin does not return a price; it returns the figure
its source published and declares which of {energy, markup, taxes} that figure
contains. The framework adds the complement, once. Engie and TotalEnergies
publish a retail figure with their margin already inside; adding the configured
3 c/kWh on top would make every future arbitrage decision wrong by that amount
with nothing reporting it.
Two mechanisms, because one was not enough. `PluginOptions` raises on the
framework-owned keys - and review of PR #16 then wrote a double-counting plugin
three more ways, one of them touching nothing but `PRICE_OPTION_DEFAULTS`, a
public dict in the module every plugin already imports. So `register()` now also
**refuses at import any plugin whose own code mentions a framework key**
(`code_leaks()`, docstrings stripped via `ast` so a plugin's mandated
`SOURCES:` line does not false-positive). Honest wholesale baseline 0.188
EUR/kWh; every attack landed 0.219.
**The guarantee's precise shape**, stated because an overclaim is worse than no
claim: the framework adds the complement exactly once *given a correct `sources`
declaration*. A plugin that declares `{energy}` while its feed is really all-in
still comes out one markup high while touching no framework name, and no code
can catch that - the declaration is the plugin's unverifiable claim about
somebody else's tariff. What is mechanised is reachability; what stays human is
truthfulness.
**The product type is declared, not inferred from the plugin's name.**
`eneco_eddy` is a real-time/imbalance product and gets widened arbitrage
@@ -38,6 +53,13 @@ pointless under the first and forbidden under the second - and an implementation
that returns `[]` on a failed fetch cannot tell them apart. Every site-level
failure returns `no_data`; nothing raises into the control tick.
**What is not verified, and cannot be by an offline suite:** every fixture is
synthetic, so the five feed URLs and JSON shapes are checked against a belief
about what those suppliers publish, not against the feeds. The parse layer is
exercised; the contract with each supplier is not. Expect the first live fetch
of each to need a shape fix, and note that a wrong shape degrades to
no-price-data rather than to a wrong price.
**`manual_csv` is the documented offline fallback** and needs no network, no
supplier account and no HTTP stack. See DOCS.md for the format. Its `start`
column must carry a UTC offset - a bare local timestamp is refused, because one
+8
View File
@@ -617,6 +617,14 @@ feed contains and the add-on adds only the rest. Set a markup under a plugin
that already carries one and it is **ignored, with a line in the log saying so**
— never applied twice.
That holds **given a correct declaration**, and it is worth knowing where the
line is. What the add-on enforces mechanically is that a plugin cannot obtain
the markup or apply it: the numbers are not reachable from plugin code, and a
plugin whose code mentions them fails to load at all. What it cannot check is
whether a plugin described its supplier's feed truthfully — that is a claim
about somebody else's tariff, and no code can verify it. If you write a plugin,
the `sources` declaration is the part a reviewer has to read.
`eneco_eddy` is a real-time/imbalance product rather than a day-ahead one. It is
flagged as high-volatility, which widens arbitrage thresholds once the
optimiser lands. That comes from the plugin's own declaration, not from its
+145 -12
View File
@@ -26,7 +26,8 @@ be taken on a provisional price).
THE THREE THINGS THIS MODULE IS SHAPED BY
=========================================
1. DOUBLE-COUNTING MARKUP IS MADE UNREACHABLE, NOT DISCOURAGED.
1. DOUBLE-COUNTING MARKUP IS REFUSED, NOT DISCOURAGED - GIVEN A CORRECT
DECLARATION.
`generic_dayahead` reads a raw EPEX figure and the user's markup and taxes
have to be added to it. Engie and TotalEnergies publish a figure that already
HAS the markup in it. Add the configured markup to that one too and every
@@ -41,11 +42,25 @@ THE THREE THINGS THIS MODULE IS SHAPED BY
* `assemble()` - the framework, never a plugin - adds exactly the
components the plugin declared it did NOT source, from
`price_markup_eur_kwh` / `price_taxes_eur_kwh`.
* ⚠️ A PLUGIN IS NEVER HANDED THOSE TWO NUMBERS. `fetch()` receives a
`PluginOptions`, which is the options dict with the framework-owned keys
REMOVED - reading `price_markup_eur_kwh` from inside a plugin raises
`FrameworkOption` rather than returning a number. That is what turns
"the plugin must not add markup" from a rule into an inability.
* ⚠️ A PLUGIN CANNOT REACH THOSE TWO NUMBERS BY ANY ROUTE, and the word
"any" was earned the hard way. `PluginOptions` raises `FrameworkOption`
on them, which closes the documented path - but review of PR #16 wrote a
double-counting plugin THREE more ways, one of them touching nothing but
`PRICE_OPTION_DEFAULTS`, a public dict in the module every plugin must
already import for `PriceProvider` and `register`. One closed door in a
room with three is not a guarantee. So `register()` now also REFUSES ANY
PLUGIN WHOSE OWN CODE MENTIONS A FRAMEWORK KEY - see `code_leaks()`.
⚠️ THE PRECISE SHAPE OF THE GUARANTEE, because an overclaim here is worse
than no claim: **the framework adds the complement exactly once, GIVEN A
CORRECT `sources` DECLARATION.** A plugin that declares `{ENERGY}` while its
feed is really all-in still comes out one markup high, and NO CODE CAN CATCH
THAT - the declaration is the plugin's unverifiable claim about somebody
else's tariff. What is mechanised is that a plugin cannot obtain the numbers
and cannot apply them; what remains human is whether it described its feed
truthfully. The docstring/declaration check in test_price.py verifies that
the plugin's two claims AGREE, not that either is true, and it is worth
knowing which of those two things you are reading.
2. THE PRODUCT TYPE IS DECLARED, NOT INFERRED FROM THE PLUGIN'S NAME.
`eneco_eddy` is a real-time / imbalance product (§9.2) and §8.3 says those
@@ -104,10 +119,18 @@ TIMEZONE, AND WHY THERE IS NO `zoneinfo` HERE
2026 transitions and counts the points rather than asserting 24.
The EU rule is fixed in law and needs no database: CEST from 01:00 UTC on the
last Sunday of March until 01:00 UTC on the last Sunday of October.
last Sunday of March until 01:00 UTC on the last Sunday of October. Review of
PR #16 cross-checked this against `zoneinfo`'s Europe/Brussels over 149,040
hourly instants, 2020-2036 inclusive, and found ZERO mismatches - so the
residual risk here is not arithmetic, it is political: if the EU ever abolishes
the seasonal change, `brussels_offset()` needs editing. That is the same
exposure as a machine whose `tzdata` has gone stale, and it is one edit in one
function rather than a package upgrade.
"""
import ast
import csv
import inspect
import io
import json
import logging
@@ -116,6 +139,7 @@ from abc import ABC, abstractmethod
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from textwrap import dedent
_LOG = logging.getLogger("goodwe.price")
@@ -425,6 +449,20 @@ class _NoData(PriceRead):
`for p in read.points`, `read.get("points", [])` and `len(read.points)` all
raise `Absence` here rather than quietly reporting that the cheapest hour of
the day costs nothing.
⚠️ TWO WAYS PAST IT SURVIVE, BOTH FOUND IN REVIEW OF PR #16, AND NEITHER IS
WORTH CODE:
* `vars(read).get("points", [])` returns `[]`. `vars()` bypasses
`__getattr__` entirely; nothing short of `__slots__` plus a metaclass
would stop it, and anyone writing `vars()` on a variant type has left
the road on purpose.
* `bool(read)` is `True` for BOTH variants, so `if read:` looks like a
check and is not one. Defining `__bool__` to be False here would make
`if read:` "work" - and that is exactly the wrong fix: it would turn a
line that reads like a guard into one that IS a guard by accident,
which is how `.get(k, default)` got into this codebase in the first
place. Use `.match(...)`. There is no falsy variant on purpose.
"""
kind = "no_data"
@@ -492,8 +530,13 @@ class PluginOptions:
"""The options dict, minus the keys only the framework may read.
Reading `price_markup_eur_kwh` here raises `FrameworkOption` rather than
returning 0.03. That is the difference between "the plugin must not add the
markup" being a convention and it being a thing a plugin cannot do.
returning 0.03.
⚠️ THIS IS ONE OF THE TWO DOORS, NOT THE ONLY ONE. `_o` is reachable, and
`price_option()` and `PRICE_OPTION_DEFAULTS` are public names in this
module. This class alone is a convention; it is `register()`'s
`code_leaks()` scan that makes it an inability. Do not weaken either half
on the strength of the other.
"""
def __init__(self, opts: dict):
@@ -581,6 +624,62 @@ class PriceProvider(ABC):
"""
# ⚠️ THE NAMES A PLUGIN'S OWN CODE MAY NOT MENTION. FRAMEWORK_KEYS is the
# obvious half; the other three are the routes review of PR #16 actually used to
# write a double-counting plugin after `PluginOptions` was in place:
#
# `_o` - PluginOptions' private dict, reachable as
# `opts._o["price_markup_eur_kwh"]`.
# `price_option` - a PUBLIC framework helper that takes the raw
# options dict and answers with the markup.
# `PRICE_OPTION_DEFAULTS` - a PUBLIC module constant. This is the one that
# decided the review: no private access, no options
# dict, no helper - just a dict in the module every
# plugin already imports for `register`.
#
# Any of them turns +0.088 into +0.119 on a wholesale feed with nothing
# anywhere reporting it.
BANNED_IN_PLUGIN_CODE = tuple(sorted(FRAMEWORK_KEYS)) + (
"PRICE_OPTION_DEFAULTS", "price_option", "_o")
def code_leaks(cls) -> list:
"""Framework-owned names appearing in a plugin's own CODE. Empty is clean.
⚠️ DOCSTRINGS ARE STRIPPED BEFORE THE SEARCH, AND THAT IS NOT COSMETIC.
Every shipped plugin's docstring is REQUIRED to name the components it
sources, so `engie_dynamic` says "markup" in prose and would false-positive
against `price_markup_eur_kwh` on a raw text scan. The check that fires on
the documentation it also mandates is the check that gets deleted. So the
class is parsed, every docstring node is dropped, and the search runs on
`ast.unparse` of what is left - i.e. on code, which is the only thing that
can actually add a number twice.
⚠️ FAIL-OPEN ON MISSING SOURCE, DELIBERATELY. `inspect.getsource` needs the
.py file, which the add-on image has (Dockerfile COPYs `app/`) but a frozen
or zipapp build would not. Refusing to start a battery controller because a
STATIC ANALYSIS could not run would trade a real outage for a hypothetical
accounting error, so an unreadable source returns `[]` and logs. CI proves
the scan really runs: test_price.py asserts source was found for all five
shipped plugins, so a build that silently lost it is a red check, not a
quietly disarmed guard.
"""
try:
src = dedent(inspect.getsource(cls))
tree = ast.parse(src)
except (OSError, TypeError, IndentationError, SyntaxError) as e:
_LOG.warning("could not read %s's source for the framework-key scan "
"(%s); the double-counting guard is NOT enforced for it",
getattr(cls, "id", cls), e)
return []
for node in ast.walk(tree):
body = getattr(node, "body", None)
if isinstance(body, list) and body and isinstance(body[0], ast.Expr) and isinstance(body[0].value, ast.Constant) and isinstance(body[0].value.value, str):
body.pop(0)
code = ast.unparse(tree)
return sorted({name for name in BANNED_IN_PLUGIN_CODE if name in code})
PROVIDERS: dict = {}
@@ -595,6 +694,19 @@ def register(cls):
if cls.id in PROVIDERS:
raise ValueError(f"duplicate provider id {cls.id!r}")
cls.capabilities() # fail at import, not at 15:00 on fetch day
leaks = code_leaks(cls)
if leaks:
# ⚠️ AT IMPORT, NOT AT FETCH. A plugin that can reach the markup is a
# plugin that can bill wrong for a month before anyone notices, so it
# does not get to exist - the add-on fails to start and names the line.
raise FrameworkOption(
f"{cls.id!r} mentions {', '.join(leaks)} in its own code. A plugin "
f"returns what its SOURCE published and declares "
f"capabilities().sources; the framework adds the complement, once. "
f"Reaching any of {BANNED_IN_PLUGIN_CODE} lets it be added twice, "
f"and a price that is one markup high is invisible in every log "
f"this add-on writes. Remove the reference; if the plugin genuinely "
f"needs a per-site number, add a NON-framework option key for it.")
PROVIDERS[cls.id] = cls
return cls
@@ -682,12 +794,33 @@ def check_egress(url: str, caps: ProviderCapabilities) -> str:
Returns the host, so a caller that forgets to call this is not the same
shape as one that did.
⚠️ THE SCHEME IS SPLIT OFF BEFORE THE HOST IS PARSED, AND THE ORDER MATTERS
FOR THE TEST MORE THAN FOR THE CODE. The first version stripped a fixed
`len("https://")` from the front, which on an `http://` URL ate two
characters of the hostname instead - `http://api.energy-charts.info/p`
became the host `pi.energy-charts.info`, which then failed the ALLOW-LIST.
So a non-TLS feed was refused, the check named "plain http is refused"
passed, and deleting the TLS rule entirely left it passing. Review of
PR #16 found that by mutation: an assertion that does not test what it
names, inside a security check - this project's own recurring defect. Now
the scheme is parsed as a scheme, the host of an `http://` URL comes out
correct and DECLARED, and the TLS rule is the only thing that can refuse
it - which is what makes the check able to fail.
"""
if not url:
raise SourceUnavailable("no feed URL configured (price_feed_url)")
if not url.startswith("https://"):
raise SourceUnavailable(f"refusing a non-TLS price feed: {url!r}")
host = url[len("https://"):].split("/", 1)[0].split("@")[-1]
scheme, sep, rest = url.partition("://")
if not sep:
raise SourceUnavailable(
f"not an absolute URL, so no host can be checked against the "
f"§16 allow-list: {url!r}")
if scheme.lower() != "https":
raise SourceUnavailable(
f"refusing a non-TLS price feed: {url!r} uses {scheme!r}. §16 "
f"allow-lists outbound destinations and SEC-01 requires TLS on "
f"every one of them.")
host = rest.split("/", 1)[0].split("?", 1)[0].split("@")[-1]
host = host.split(":", 1)[0].lower()
if host not in {h.lower() for h in caps.egress_hosts}:
raise SourceUnavailable(
+9
View File
@@ -435,6 +435,15 @@ def _price_problems(opts: dict) -> list[str]:
# operator's evidence that they picked a supplier is a status line saying
# they picked a supplier. That is (a), and it is the "silently double up"
# the PM named on 2026-08-30.
# ⚠️ THIS REFUSAL'S REASON EXPIRES, AND THE MESSAGE BELOW MUST BE EDITED IN
# THE SAME COMMIT THAT LANDS THE FIRST CONSUMER (TARIFF-03 or OPT-04).
# Today the message says the provider "would decide nothing" - true, because
# nothing reads the series. Once something does, both sources really decide
# and this becomes a straightforward two-writer conflict; the refusal stays
# right and its explanation goes wrong, which is worse than either. There is
# a tripwire on exactly this in test_price.py section 12: it greps app/ for
# a `load_prices` consumer and goes RED the moment one appears. Update the
# message, not the check.
ha_price = [k for k in ("price_now_entity", "price_avg_entity")
if str(opts.get(k, "") or "").strip()]
if pid != "off" and ha_price:
+338 -16
View File
@@ -36,6 +36,7 @@ TWO RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
agree exactly.
"""
import ast
import asyncio
import inspect
import io
@@ -46,13 +47,16 @@ import sys
from datetime import datetime, timedelta, timezone
from app.price import (
ALL_IN, Absence, Component, FrameworkOption, MAX_PLAUSIBLE_EUR_KWH,
ALL_IN, Absence, BANNED_IN_PLUGIN_CODE, Component, FrameworkOption,
MAX_PLAUSIBLE_EUR_KWH, code_leaks,
PRICE_OPTION_DEFAULTS, PROVIDERS, PluginOptions, PricePoint, PriceProduct,
PriceProvider, PriceRead, ProviderCapabilities, RawPoint, SourceUnavailable,
arbitrage_allowed, assemble, brussels, brussels_offset, check_egress,
price_option,
export_price_eur_kwh, load_prices, plan_is_provisional, register,
volatility_regime,
)
from app import price as _price
from app.validate import validate_startup_config
fails = []
@@ -68,6 +72,34 @@ def run(coro):
return asyncio.run(coro)
def _reason(fn):
"""The message fn() raised, so a check can assert WHICH rule refused.
`raises(...)` only says that something refused. M7 survived because two
different rules could refuse the same URL and the check could not tell them
apart - so where a check is named for a specific rule, it asserts on the
reason instead.
"""
try:
fn()
except Exception as e: # noqa: BLE001
return str(e)
return ""
def raises(fn, exc):
"""True if fn() raises exactly `exc`. A DIFFERENT exception is False, not
True - otherwise "it was refused" and "it broke on the way to being
refused" would grade the same, which is how M7 hid."""
try:
fn()
except exc:
return True
except Exception:
return False
return False
class _Raised:
"""A value a check wanted, whose computation raised.
@@ -500,6 +532,147 @@ check("...and the wholesale plugin, which really does use them, does not",
"generic_dayahead" not in _warned)
# --------------------------------------------------------------------------- #
print()
print("4b. a plugin cannot reach the markup BY ANY ROUTE (review of PR #16)")
# WHY THIS SECTION EXISTS. The first version of this ticket claimed the markup
# was "not in scope" for a plugin because `PluginOptions` refused it. Review
# wrote a double-counting plugin three more ways, one of them touching nothing
# but `PRICE_OPTION_DEFAULTS` - a public dict in the module every plugin must
# already import for `register`. One closed door in a room with three is not a
# guarantee. Honest wholesale baseline 0.188; each attack lands 0.219.
_banned = set(BANNED_IN_PLUGIN_CODE)
check(f"the banned set covers the framework keys AND the three public routes "
f"review actually used {sorted(_banned)}",
_banned == set(_price.FRAMEWORK_KEYS) | {"PRICE_OPTION_DEFAULTS",
"price_option", "_o"})
# code_leaks() fails OPEN when inspect.getsource() cannot find the .py - the
# right trade for a battery controller, and a silently disarmed guard if nobody
# checks. THIS is the check that makes CI prove the guard is armed.
_srclen = derived(lambda: {pid: len(inspect.getsource(cls))
for pid, cls in PROVIDERS.items()})
check(f"the scan really read every shipped plugin's source, so its fail-open "
f"path is not silently in use {_srclen}",
isinstance(_srclen, dict) and len(_srclen) == 5
and all(n > 200 for n in _srclen.values()))
_dirty = derived(lambda: {pid: code_leaks(cls)
for pid, cls in PROVIDERS.items()
if code_leaks(cls)})
check(f"...and all five shipped plugins are clean {_dirty}", _dirty == {})
# THE DOCSTRING STRIP IS LOAD-BEARING, AND THIS PROVES IT RATHER THAN ASSERTING
# IT. Every plugin is REQUIRED to name its sourced components in prose, so
# engie_dynamic's docstring contains "markup" - a raw text scan false-positives
# on the documentation the same file mandates, and a check that fires on
# required documentation is a check somebody deletes.
_raw_hits = derived(lambda: sorted(
pid for pid, cls in PROVIDERS.items()
if any(b in inspect.getsource(cls) for b in _banned)))
check(f"a RAW text scan would false-positive on the mandated docstrings "
f"{_raw_hits}, which is why code_leaks() strips them before searching",
isinstance(_raw_hits, list) and _raw_hits != [] and _dirty == {})
class _AttackPrivateDict(PriceProvider):
"""A1: reaches into PluginOptions' private dict.
SOURCES: energy
The framework ADDS: markup, taxes
"""
id = "_attack_private_dict"
@classmethod
def capabilities(cls):
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.ENERGY}))
async def fetch(self, o, since, until, transport):
return [RawPoint(T0, 0.10 + o._o["price_markup_eur_kwh"], False)]
class _AttackHelper(PriceProvider):
"""A2: calls a public framework helper with the raw options dict.
SOURCES: energy
The framework ADDS: markup, taxes
"""
id = "_attack_helper"
@classmethod
def capabilities(cls):
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.ENERGY}))
async def fetch(self, o, since, until, transport):
return [RawPoint(T0, 0.10 + price_option(o._o, "price_markup_eur_kwh"),
False)]
class _AttackPublicConstant(PriceProvider):
"""A2b: touches nothing but a public module constant. THE DECIDING ONE.
SOURCES: energy
The framework ADDS: markup, taxes
"""
id = "_attack_public_constant"
@classmethod
def capabilities(cls):
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.ENERGY}))
async def fetch(self, o, since, until, transport):
return [RawPoint(T0,
0.10 + PRICE_OPTION_DEFAULTS["price_markup_eur_kwh"],
False)]
_ATTACKS = {"A1 private dict": _AttackPrivateDict,
"A2 public helper": _AttackHelper,
"A2b public constant": _AttackPublicConstant}
for _name, _cls in _ATTACKS.items():
_found = derived(lambda c=_cls: code_leaks(c))
check(f"{_name}: the scan names what it reached for {_found}",
isinstance(_found, list) and _found != [])
check(f"...and register() REFUSES it at import, not at fetch time",
raises(lambda c=_cls: register(c), FrameworkOption))
check(f"...so it never enters the registry", _cls.id not in PROVIDERS)
# THE COST OF THE DEFECT, MEASURED, so the refusal is not grading itself.
_honest_wholesale = derived(lambda: assemble(
PROVIDERS["generic_dayahead"].capabilities(),
[RawPoint(T0, 0.10, False)], MARKUP, TAXES)[0].eur_per_kwh)
_attacked = derived(lambda: assemble(
_AttackPublicConstant.capabilities(),
[RawPoint(T0, 0.10 + MARKUP, False)], MARKUP, TAXES)[0].eur_per_kwh)
check(f"an unrefused attack would bill {_attacked} against an honest "
f"{_honest_wholesale} - one markup high, every hour, silently",
isinstance(_attacked, float) and isinstance(_honest_wholesale, float)
and abs((_attacked - _honest_wholesale) - MARKUP) < 1e-9)
# WHAT THIS DOES *NOT* CLOSE, AND CANNOT. A plugin that declares {ENERGY} while
# its feed is really all-in comes out one markup high while touching no
# framework name at all. No code can check that: the declaration is the
# plugin's unverifiable claim about somebody else's tariff. Stated here so the
# guarantee is read at its true size - the complement is added exactly once,
# GIVEN A CORRECT DECLARATION - and so a reviewer who trusts an absolute
# guarantee does not stop checking. Measured, not asserted:
_misdeclared = derived(lambda: assemble(
ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.ENERGY})),
[RawPoint(T0, 0.10 + MARKUP + TAXES, False)], MARKUP, TAXES)[0].eur_per_kwh)
check(f"a MIS-DECLARED all-in feed still doubles ({_misdeclared} vs "
f"{_honest_wholesale}) with no framework name touched - the limit of the "
f"guarantee, stated rather than hidden",
isinstance(_misdeclared, float)
and abs((_misdeclared - _honest_wholesale) - (MARKUP + TAXES)) < 1e-9
and code_leaks(_AttackPrivateDict) != [])
# --------------------------------------------------------------------------- #
print()
print("5. product type is declared, not inferred from the name (AC2, trap 2)")
@@ -536,8 +709,6 @@ check("volatility_regime refuses a plugin NAME, which is the AC2 violation "
# plugin id at all - if it does, the sixth supplier is an edit to price.py's
# shared machinery rather than a new class. Derived: build the set of ids that
# actually occur, and assert it is empty.
from app import price as _price # noqa: E402
_framework = [_price.assemble, _price.load_prices, _price.volatility_regime,
_price.arbitrage_allowed, _price.plan_is_provisional,
_price.check_egress, _price._validate_raw, _price.PriceRead,
@@ -606,16 +777,6 @@ check("...and the cheapest hour is computable for the zero series",
derived(lambda: min(p.eur_per_kwh for p in _zero.points)) == 0.0)
def raises(fn, exc):
try:
fn()
except exc:
return True
except Exception:
return False
return False
check("no_data has no .points, so `min(read.points)` cannot invent a free hour",
raises(lambda: _none.points, Absence))
check("...no .product to switch a threshold regime on",
@@ -630,6 +791,54 @@ check("...and it is not iterable, so `for p in read:` cannot be an empty loop",
check("...while its one real field, the reason, is there for an operator",
_none.reason == "the feed timed out")
# M3, INHERITED FROM RIG-01 AND CLOSED HERE. `_NoData.__init__(self,
# reason="no price data")` used to go green: a required field with a DEFAULT is
# the exact idiom this whole type exists to forbid, and it survived because
# every check passed a reason explicitly. It is the identical survivor left
# open on sim/riglock.py's `_Unread.cause` - the shape was borrowed including
# its blind spot, which is what borrowing a shape does if nobody re-runs the
# mutations against the copy.
#
# Checked by SIGNATURE rather than by one call, so any default added to any of
# these constructors later is red without somebody remembering to write a new
# case.
def _defaulted(fn):
"""Parameters of `fn` that carry a default. `self` is skipped."""
try:
ps = list(inspect.signature(fn).parameters.values())
except (TypeError, ValueError) as e: # pragma: no cover
return _Raised(e)
return [q.name for q in ps
if q.name != "self" and q.default is not inspect.Parameter.empty]
_ctors = {"PriceRead.no_data": PriceRead.no_data,
"PriceRead.series": PriceRead.series,
"_NoData.__init__": _price._NoData.__init__,
"_Series.__init__": _price._Series.__init__,
"PriceRead.match": PriceRead.match}
_with_defaults = derived(lambda: {n: d for n, d in
((n, _defaulted(f)) for n, f in _ctors.items())
if d})
check(f"no constructor or handler of an outcome type carries a DEFAULT, so a "
f"reading can never be built out of nothing {_with_defaults}",
_with_defaults == {})
check("...so no_data() with no reason at all is a TypeError at the call site",
raises(lambda: PriceRead.no_data(), TypeError))
check("...and the variant class itself cannot be built bare either",
raises(lambda: _price._NoData(), TypeError))
check("...nor can a series be built without its product or provenance",
raises(lambda: _price._Series([PricePoint(brussels(T0), 0.2, False)]),
TypeError))
# The positive arm, so the four above are not passing because everything raises.
check("...while a fully specified no_data is still perfectly constructible",
PriceRead.no_data("the feed timed out").reason == "the feed timed out")
# `match` must also refuse POSITIONAL handlers - keyword-only is what makes a
# forgotten arm a TypeError rather than a silently swapped one.
check("match() refuses positional handlers, so the two arms cannot be swapped",
raises(lambda: _none.match(lambda r: 1, lambda r: 2), TypeError))
check("an EMPTY series cannot be constructed at all, so 'no prices' can never "
"wear the shape of an answer",
raises(lambda: PriceRead.series([], PriceProduct.DAY_AHEAD, "x", T0),
@@ -828,6 +1037,34 @@ check("...and its canonical starts are 25 distinct strings",
# --------------------------------------------------------------------------- #
print()
print("9. the boundary rejects what cannot be a price, and only that")
# M5. Deleting the `Component.ENERGY not in sources` guard used to go green,
# and the consequence is not cosmetic: `adds` becomes {ENERGY, TAXES},
# `assemble()` never adds an energy component it thinks it is adding, and the
# emitted all-in price silently omits the energy it is supposed to be pricing.
check("capabilities() refuses a feed that declares no ENERGY component",
raises(lambda: ProviderCapabilities(
product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.MARKUP})), ValueError))
check("...including a completely empty sources set",
raises(lambda: ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
sources=frozenset()), ValueError))
check("...and an unknown component is refused rather than ignored",
raises(lambda: ProviderCapabilities(
product=PriceProduct.DAY_AHEAD,
sources=frozenset({Component.ENERGY, "grid_fees"})), ValueError))
check("...and a product given as a string is refused at the declaration too",
raises(lambda: ProviderCapabilities(product="dynamic_dayahead",
sources=frozenset({Component.ENERGY})),
ValueError))
# THE CONSEQUENCE, derived across every shipped plugin rather than asserted on
# one: the framework must never be in a position to "add" the energy component,
# because there is nothing to add it from.
_adds_energy = derived(lambda: sorted(
pid for pid, cls in PROVIDERS.items()
if Component.ENERGY in cls.capabilities().adds))
check(f"so no shipped plugin ever asks the framework to invent an energy "
f"price {_adds_energy}", _adds_energy == [])
_caps = PROVIDERS["generic_dayahead"].capabilities()
@@ -867,9 +1104,44 @@ check("a declared https host is allowed",
check("an undeclared host is refused even over TLS",
raises(lambda: check_egress("https://evil.example/p", _g),
SourceUnavailable))
check("plain http is refused",
raises(lambda: check_egress("http://api.energy-charts.info/p", _g),
SourceUnavailable))
# M7, AND THE ASSERTION IS THE FIX, NOT ONLY THE PARSER. Deleting the TLS rule
# used to leave this check GREEN: the old parser stripped a fixed
# len("https://") from the front, so `http://api.energy-charts.info/p` became
# the host `pi.energy-charts.info`, which failed the ALLOW-LIST instead. The
# feed was refused, the check named "plain http is refused" passed, and the
# rule it was named for could be deleted with no consequence - an assertion
# that does not test what it names, inside a security check.
#
# Three checks now, and the first two are what make the third able to fail:
# prove the host is declared over https, prove it parses identically over
# http, and only then assert that http is refused AND that the reason is TLS.
_tls_url = "http://api.energy-charts.info/p"
check("the same host over https is allowed, so it IS on the allow-list",
derived(lambda: check_egress("https://api.energy-charts.info/p", _g))
== "api.energy-charts.info")
check("...and the http form parses to that same declared host, so nothing but "
"the TLS rule can refuse it",
derived(lambda: _tls_url.partition("://")[2].split("/", 1)[0])
== "api.energy-charts.info")
_tls_reason = derived(lambda: _reason(lambda: check_egress(_tls_url, _g)))
check(f"plain http is refused, and refused BY THE TLS RULE - {_tls_reason!r}",
isinstance(_tls_reason, str) and "non-TLS" in _tls_reason)
check("a scheme that is neither is refused the same way, by TLS not by host",
"non-TLS" in str(derived(lambda: _reason(
lambda: check_egress("ftp://api.energy-charts.info/p", _g)))))
# ⚠️ M7b, WHICH I FOUND IN MY OWN FIX FOR M7 AND WHICH IS THE SAME DEFECT ONE
# LAYER DOWN. This check used to read `raises(..., SourceUnavailable)` and
# survived deleting the `if not sep` guard: with no guard,
# "api.energy-charts.info/p".partition("://") leaves scheme = the WHOLE string,
# so the TLS rule refuses it and the check named for the scheme guard passes.
# Refused either way - but the diagnostic an operator gets degrades from "not an
# absolute URL" to "uses 'api.energy-charts.info/p'" as a scheme, which is
# gibberish. Assert the reason, like M7 above.
_noscheme = derived(lambda: _reason(
lambda: check_egress("api.energy-charts.info/p", _g)))
check(f"a URL with no scheme is refused BY THE SCHEME GUARD, with a diagnostic "
f"that names the real problem - {_noscheme!r}",
isinstance(_noscheme, str) and "not an absolute URL" in _noscheme)
check("a userinfo prefix cannot smuggle a declared host past the check",
raises(lambda: check_egress(
"https://api.energy-charts.info@evil.example/p", _g),
@@ -953,6 +1225,56 @@ check("the Home Assistant entities alone still start, so nothing is taken away "
validate_startup_config(vbase(price_now_entity="sensor.now",
price_avg_entity="sensor.avg"),
-5000, 5000) == [])
# ⚠️ A TRIPWIRE ON A REASON, NOT ON A BEHAVIOUR, AND IT IS AIMED AT WHOEVER
# LANDS TARIFF-03 OR OPT-04. Rule 7 refuses "provider + HA entities" and its
# stated reason is that the provider series decides NOTHING while the HA pair
# quietly keeps deciding. The refusal stays correct forever; its EXPLANATION
# expires the moment a consumer appears - at that point both really do decide,
# and it becomes a genuine two-writer conflict rather than a silently-ignored
# one. A future reader would then be trusting a reason that is no longer the
# reason, which is the same defect TEST-06's boundary comment names.
#
# So: assert that nothing in app/ consumes the series yet. When you wire
# load_prices() into the optimiser this goes RED, and the fix is to update the
# refusal message in validate.py IN THE SAME COMMIT - not to delete this check.
_APP = os.path.join(os.path.dirname(os.path.abspath(__file__)), "app")
# ⚠️ AST, NOT A SUBSTRING, AND THE FIRST DRAFT PROVED WHY. A text scan for
# "load_prices" matched the COMMENT in validate.py that explains this very
# tripwire, so the check went red against its own documentation - the same
# false-positive class as the plugin docstrings in section 4b, hit twice in one
# ticket. Comments and docstrings are not in an AST, so a name/import walk sees
# code and only code.
def _imports_or_calls(path, name):
try:
tree = ast.parse(open(path, encoding="utf-8").read())
except (OSError, SyntaxError):
return False
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and any(a.name == name
for a in node.names):
return True
if isinstance(node, ast.Name) and node.id == name:
return True
if isinstance(node, ast.Attribute) and node.attr == name:
return True
return False
_consumers = derived(lambda: sorted(
f for f in os.listdir(_APP)
if f.endswith(".py") and f != "price.py"
and _imports_or_calls(os.path.join(_APP, f), "load_prices")))
check(f"no module consumes load_prices() yet, so rule 7's stated reason - 'the "
f"provider decides nothing' - is still TRUE {_consumers}. When this goes "
f"red, update that message in validate.py in the same commit",
_consumers == [])
# ...and the other half: the registry IS on the startup path, which is what
# makes an unwired seam anchored rather than dead code.
check("...while validate.py does import the registry, so the seam is exercised "
"on every boot",
derived(lambda: "PROVIDERS" in open(
os.path.join(_APP, "validate.py"), encoding="utf-8").read()) is True)
check("manual_csv with no path is refused - the offline fallback with no file "
"is §11.4 reached by accident",
why(validate_startup_config(vbase(price_provider="manual_csv",