Claude Opus 5
17b7b447c4
checks / unit (pull_request) Successful in 7m10s
`generic_dayahead.fetch()` accepted `since` and `until` and requested the
bare `?bzn=BE`, which answers ONE Brussels day. Everything past today was
then filled from yesterday's profile by FCST-04, flagged provisional, and
§8.3 drops provisional hours from the trade set -- so the optimiser never
saw tomorrow as firm, at any hour of any day, and nothing reported it.
Measured on 4257e38 at 2026-09-10T09:00+02:00, 48 h horizon, live feed,
yesterday seeded through the shipped read path:
60 firm points (15 whole hours) + 33 carried hours of 48 requested
carried: 24 h on 2026-09-11, 9 h on 2026-09-12
arbitrage(0.10): buy 11:45 and sell 19:45 -- BOTH on 2026-09-10
firm local days = [2026-09-10]
The ceiling was structural: with a one-day window, at hour H a 48 h horizon
is at most 24-H firm and 24+H carried. At 23:00 that is 1 and 47, and the
remaining firm spread (0.0464 EUR/kWh) no longer clears the default 0.10
threshold, so arbitrage is refused outright. FCST-04's fill covers the
window until the ~13:00-14:00 CET auction; it was never meant to cover
every hour of every day.
The data exists and the endpoint serves it: verified 2026-09-10T07:45:49Z,
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns 192 quarter-hours across
two local days. Committed unmodified beside TARIFF-08's captures, with the
URL, retrieval time and sha256 recorded and RE-DERIVED by the replay.
THE RANGE IS DERIVED ON THE BRUSSELS CALENDAR, not the UTC one. 22:30 UTC
in summer is already tomorrow in Belgium, so a UTC-derived start asks for
the day the controller has just left and drops the day it is standing in,
for the last one or two hours of every day. That is `_bdate()`'s lesson one
layer up with a worse consequence: `_bdate()` misfiles an hour, this never
fetches it. `until` is exclusive, so local midnight belongs to the day
before it; an inverted window collapses to one day rather than a backwards
range. Both ends are graded and both are driven red by mutation.
After, same code path and window, a feed serving both days: 39 firm hours
and 9 carried (was 15 and 33), firm across TWO Brussels days, and the
permitted cycle buys on one day and sells on the next.
`price_feed_url` IS THE EXACT URL AND THE PLUGIN APPENDS NOTHING. `start`/
`end` are this feed's spelling and the provider declares a second host
whose parameters are different, so merging would produce a request nobody
typed and the short horizon anyway. The cost is real, so it is not silent:
a WARNING on the shipped logger at fetch time names the consequence and the
start..end that would otherwise have been asked for. `dated_feed_url()`
refuses to append to a base that already carries a query string, so the
blind-append shape cannot come back by edit.
DECISIONS/TARIFF-09-the-range-and-the-override.md argues the alternative.
Before publication nothing regresses and it is graded, not assumed: at
09:46 CEST the two-day request came back byte-identical to the one-day
body, and that stays a `series` of what exists with FCST-04 filling the
rest. A `no_data` there would switch a working site's arbitrage off every
morning -- the mutation that adds that rule is red by name.
The plugin contract is still NOT enforced: three shipped plugins point at
endpoints nobody has verified, and a rule obliging them to pass a range
would be satisfied by an invented parameter name -- TARIFF-08's defect
moved from the response to the request. Which plugins consume the range is
pinned as a literal instead.
11 mutations, all red, no survivors; each applied to a fresh copy of the
tree rather than reverted, through tools/mutation_run.sh. One is +0 B (a
single price digit inside the committed capture) and the re-derived sha256
catches it. MUTATION.md habit 20 records why a fixture could never have
reached this: a captured body is the answer to a URL, and no property of a
body says which URL was asked for. All 29 suites green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
1953 lines
96 KiB
Python
1953 lines
96 KiB
Python
"""Runnable check for the price provider framework. `python3 test_price.py`
|
|
|
|
WHAT THIS FILE IS DEFENDING, in the order the ticket names it:
|
|
|
|
1. Every shipped plugin emits the EXACT canonical schema (§9.3), all-in.
|
|
2. Adding a plugin needs no optimiser change - so the framework must contain
|
|
no plugin name, and a plugin invented HERE, in this file, must reach the
|
|
right volatility regime without price.py being touched.
|
|
3. `manual_csv` works with no network at all.
|
|
4. The injection price is a separate scalar (§9.3).
|
|
5. "No price data" is a DIFFERENT OUTCOME from "prices, all of them zero".
|
|
|
|
TWO RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
|
|
|
|
* EVERY VERDICT IS DERIVED. There is no `check("the schema is right", True)`
|
|
anywhere below. Where a check needs an expected number, the number is
|
|
computed here from the same inputs by different arithmetic than price.py
|
|
uses, and where it needs a set, the set is built from the objects rather
|
|
than typed out.
|
|
|
|
* NO STIMULUS IS EVALUATED WHERE THE QUANTITY IT GRADES IS DEGENERATE. That
|
|
is not decoration here, it is the main hazard in a markup test:
|
|
|
|
- markup and taxes must DIFFER and both be NON-ZERO, or "added markup
|
|
twice" and "added markup and taxes once each" produce the same number
|
|
and the double-counting check grades nothing. The shipped defaults are
|
|
0.03 and 0.06 and 0.06 IS EXACTLY TWICE 0.03, so this file deliberately
|
|
uses 0.031 / 0.057 for the arithmetic and asserts the non-degeneracy
|
|
conditions as checks in their own right rather than assuming them.
|
|
- the raw price must be non-zero, or an all-in figure equals markup+taxes
|
|
and a plugin that dropped the energy component entirely still passes.
|
|
- a fixture must carry BOTH a provisional and a non-provisional point, or
|
|
`plan_is_provisional` returns the same answer under any bug.
|
|
- the DST checks run on the 23-hour and 25-hour days, not on a normal
|
|
24-hour day where a fixed-24-slot implementation and a correct one
|
|
agree exactly.
|
|
|
|
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
|
|
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
|
|
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
|
|
edit reverted inside one second is invisible to CPython's staleness test. That
|
|
is TEST-15, and it has already turned one live mutation into a false survivor.
|
|
Read MUTATION.md before you report a survivor.
|
|
"""
|
|
|
|
import ast
|
|
import asyncio
|
|
import hashlib
|
|
import inspect
|
|
import io
|
|
import json
|
|
import logging
|
|
import os
|
|
import sys
|
|
from datetime import date, datetime, timedelta, timezone
|
|
from textwrap import dedent
|
|
|
|
from app.price import (
|
|
ALL_IN, Absence, BANNED_IN_PLUGIN_CODE, Component, FrameworkOption,
|
|
MAX_PLAUSIBLE_EUR_KWH, code_leaks,
|
|
PRICE_OPTION_DEFAULTS, PROVIDERS, PluginOptions, PricePoint, PriceProduct,
|
|
PriceProvider, PriceRead, ProviderCapabilities, RawPoint, SourceUnavailable,
|
|
arbitrage_allowed, assemble, brussels, brussels_date_range,
|
|
brussels_offset, check_egress, dated_feed_url,
|
|
price_option,
|
|
export_price_eur_kwh, load_prices, plan_is_provisional, register,
|
|
volatility_regime,
|
|
)
|
|
from app import price as _price
|
|
from app.validate import validate_startup_config
|
|
|
|
fails = []
|
|
|
|
|
|
def check(name, cond):
|
|
print(f" {'ok ' if cond else 'FAIL'} {name}")
|
|
if not cond:
|
|
fails.append(name)
|
|
|
|
|
|
def run(coro):
|
|
return asyncio.run(coro)
|
|
|
|
|
|
def _reason(fn):
|
|
"""The message fn() raised, so a check can assert WHICH rule refused.
|
|
|
|
`raises(...)` only says that something refused. M7 survived because two
|
|
different rules could refuse the same URL and the check could not tell them
|
|
apart - so where a check is named for a specific rule, it asserts on the
|
|
reason instead.
|
|
"""
|
|
try:
|
|
fn()
|
|
except Exception as e: # noqa: BLE001
|
|
return str(e)
|
|
return ""
|
|
|
|
|
|
def raises(fn, exc):
|
|
"""True if fn() raises exactly `exc`. A DIFFERENT exception is False, not
|
|
True - otherwise "it was refused" and "it broke on the way to being
|
|
refused" would grade the same, which is how M7 hid."""
|
|
try:
|
|
fn()
|
|
except exc:
|
|
return True
|
|
except Exception:
|
|
return False
|
|
return False
|
|
|
|
|
|
class _Raised:
|
|
"""A value a check wanted, whose computation raised.
|
|
|
|
Compares unequal to everything, is falsy, and prints what went wrong.
|
|
"""
|
|
|
|
def __init__(self, e):
|
|
self.e = e
|
|
|
|
def __eq__(self, other):
|
|
return False
|
|
|
|
def __hash__(self):
|
|
return id(self)
|
|
|
|
def __bool__(self):
|
|
return False
|
|
|
|
def __repr__(self):
|
|
return f"<raised {type(self.e).__name__}: {self.e}>"
|
|
|
|
|
|
def derived(fn):
|
|
"""Compute a value a check grades. An exception becomes `_Raised`.
|
|
|
|
⚠️ THIS IS NOT A DEFAULT, and the distinction is the one this whole ticket
|
|
is about. `_Raised` is not `[]`, not `None`, not 0; it equals nothing, it is
|
|
falsy, and it renders as the exception that produced it. Its only job is
|
|
that a mutation which makes a derivation IMPOSSIBLE - the reviewer removing
|
|
`_NoData`'s missing `.points`, say - reports as a failed check naming the
|
|
exception, instead of aborting this file before the sections below it run.
|
|
A silent `except: return []` here would be the defect under test, in the
|
|
harness that tests it.
|
|
"""
|
|
try:
|
|
return fn()
|
|
except Exception as e: # noqa: BLE001
|
|
return _Raised(e)
|
|
|
|
|
|
UTC = timezone.utc
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
|
|
# ⚠️ NOT the shipped 0.03 / 0.06. See this module's docstring: 0.06 is exactly
|
|
# 2 x 0.03, so under the shipped pair "the framework added the markup twice and
|
|
# no taxes" and "the framework added one markup and one taxes" differ by
|
|
# 0.03 - but several OTHER confusions collapse. These two are co-prime-ish in
|
|
# the only sense that matters here, and the three degeneracy conditions are
|
|
# asserted below rather than assumed.
|
|
MARKUP = 0.031
|
|
TAXES = 0.057
|
|
|
|
|
|
def opts(**over):
|
|
"""A price configuration. Everything below is this, spoilt."""
|
|
o = {"price_provider": "off",
|
|
"price_markup_eur_kwh": MARKUP,
|
|
"price_taxes_eur_kwh": TAXES,
|
|
"price_timeout_s": 5}
|
|
o.update(over)
|
|
return o
|
|
|
|
|
|
def hours(n, start, step_h=1):
|
|
return [start + timedelta(hours=i * step_h) for i in range(n)]
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print("0. the arithmetic this file grades with must not be degenerate")
|
|
# ⚠️ These four ARE the guard against a check that passes for the wrong reason.
|
|
# If any of them goes red, every markup assertion below is measuring nothing.
|
|
check("markup and taxes are both non-zero", MARKUP != 0 and TAXES != 0)
|
|
check("...and differ, so 'added the wrong one' is visible",
|
|
MARKUP != TAXES)
|
|
check("...and markup twice != markup plus taxes, so double-counting the markup "
|
|
"is distinguishable from correctly adding both",
|
|
abs(2 * MARKUP - (MARKUP + TAXES)) > 1e-9)
|
|
check("...and the shipped defaults are NOT what this file grades with, because "
|
|
f"config.yaml's {PRICE_OPTION_DEFAULTS['price_taxes_eur_kwh']} is exactly "
|
|
f"twice its {PRICE_OPTION_DEFAULTS['price_markup_eur_kwh']}",
|
|
abs(PRICE_OPTION_DEFAULTS["price_taxes_eur_kwh"]
|
|
- 2 * PRICE_OPTION_DEFAULTS["price_markup_eur_kwh"]) < 1e-9
|
|
and (MARKUP, TAXES) != (PRICE_OPTION_DEFAULTS["price_markup_eur_kwh"],
|
|
PRICE_OPTION_DEFAULTS["price_taxes_eur_kwh"]))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("1. the shipped defaults live in config.yaml, and here only as a mirror")
|
|
# ⚠️ CFG-03. The add-on already carries option defaults in three places. This
|
|
# ticket adds ONE in-code mirror (PRICE_OPTION_DEFAULTS) and pins it to
|
|
# config.yaml by reading the file rather than by transcribing it - the same
|
|
# move test_validate.py makes for max_w and meter_freeze_s. If they drift, this
|
|
# goes red and somebody has to decide which is right.
|
|
_CFG = os.path.join(os.path.dirname(os.path.abspath(__file__)), "config.yaml")
|
|
with open(_CFG, "r", encoding="utf-8") as fh:
|
|
_cfg_lines = fh.read().splitlines()
|
|
|
|
|
|
def shipped(key):
|
|
"""The value config.yaml ships, from the `options:` block, parsed by hand.
|
|
|
|
No yaml module in the shipped image, and this is two lines of parsing
|
|
against a file whose shape is known - the alternative is a transcription,
|
|
which is the thing being guarded against.
|
|
"""
|
|
in_schema = False
|
|
for line in _cfg_lines:
|
|
if line.startswith("schema:"):
|
|
in_schema = True
|
|
if in_schema or not line.startswith(f" {key}:"):
|
|
continue
|
|
raw = line.split(":", 1)[1].split("#", 1)[0].strip().strip('"').strip("'")
|
|
# OPT-04 added the first BOOLEAN price option, and without this the
|
|
# mirror check would compare the string "false" against `False` and go
|
|
# red for a file that is perfectly in step. Parsing a bool as a bool is
|
|
# what the yaml module this image does not carry would have done; it
|
|
# widens nothing, because a key whose shipped value disagrees with the
|
|
# mirror still drifts.
|
|
if raw in ("true", "false"):
|
|
return raw == "true"
|
|
try:
|
|
return int(raw)
|
|
except ValueError:
|
|
pass
|
|
try:
|
|
return float(raw)
|
|
except ValueError:
|
|
return raw
|
|
return KeyError
|
|
|
|
|
|
_found = {k: shipped(k) for k in PRICE_OPTION_DEFAULTS}
|
|
_missing = sorted(k for k, v in _found.items() if v is KeyError)
|
|
check(f"config.yaml ships every price option ({len(PRICE_OPTION_DEFAULTS)} keys)",
|
|
_missing == [])
|
|
_drift = sorted(k for k, v in _found.items()
|
|
if v is not KeyError and v != PRICE_OPTION_DEFAULTS[k])
|
|
check(f"...and each shipped default equals the in-code mirror {_found}",
|
|
_drift == [])
|
|
# The schema block must list them too, or the Supervisor drops the option and
|
|
# the add-on silently runs on the mirror's value instead of the operator's.
|
|
_schema_block = _cfg_lines[_cfg_lines.index("schema:"):]
|
|
_unschemad = sorted(k for k in PRICE_OPTION_DEFAULTS
|
|
if not any(ln.startswith(f" {k}:") for ln in _schema_block))
|
|
check("...and every one of them is in the schema block too", _unschemad == [])
|
|
# ⚠️ Derived from the registry, not typed: the schema's provider list must offer
|
|
# exactly the plugins that exist, or the UI and the code disagree about what is
|
|
# installable.
|
|
_line = next(ln for ln in _schema_block if ln.startswith(" price_provider:"))
|
|
_schema_ids = set(_line.split("list(", 1)[1].rstrip(")").split("|"))
|
|
check(f"the schema offers exactly the registered plugins plus 'off' "
|
|
f"{sorted(_schema_ids)}",
|
|
_schema_ids == set(PROVIDERS) | {"off"})
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("2. every shipped plugin emits the exact canonical schema (AC1)")
|
|
|
|
# One recorded fixture per plugin, none of them a real fetch. Prices are
|
|
# deliberately distinct, non-zero, and one of them NEGATIVE - §17 keeps a
|
|
# negative-price regression case, and a windy Sunday is the best charging hour
|
|
# of the year, so a framework that clamped it away would cost real money.
|
|
T0 = datetime(2026, 1, 15, 0, 0, tzinfo=UTC)
|
|
# ⚠️ THE NEGATIVE HOUR IS -95 EUR/MWh, NOT -12.5, AND THE MAGNITUDE IS THE
|
|
# POINT. All-in adds 0.088 EUR/kWh here, so a -12.5 EUR/MWh hour comes out at
|
|
# +0.0755 and the "a negative price survives" check below would be grading a
|
|
# positive number - passing whether or not the framework clamped anything.
|
|
# -95 EUR/MWh is both realistic for a windy Belgian Sunday and still negative
|
|
# after markup and taxes, which is the only version of the stimulus that grades
|
|
# what it claims to.
|
|
EPEX_EUR_MWH = [61.4, -95.0, 183.0] # EUR/MWh, one deeply negative
|
|
RETAIL_EUR_KWH = [0.1341, 0.0872, 0.2610] # EUR/kWh, markup already inside
|
|
EDDY_EUR_MWH = [141.0, 96.2, 268.4] # EUR/MWh, all-in
|
|
|
|
|
|
def synthetic_row_feed(key, values, provisional_at=1):
|
|
"""An INVENTED row-per-point body. `provisional_at` makes ONE point
|
|
provisional and the rest not, so `plan_is_provisional` is graded against a
|
|
mixed series - an all-true or all-false fixture would return the same
|
|
answer under any bug.
|
|
|
|
⚠️ THE NAME SAYS `synthetic` BECAUSE TARIFF-08 IS WHAT HAPPENS WHEN IT DOES
|
|
NOT. This shape - `{"data": [{"start": ..., "price_eur_mwh": ...}]}` - was
|
|
invented by whoever wrote `_points_from_json()`, and until 2026-09-10 it
|
|
was the ONLY shape any price check in this repository had ever fed a
|
|
parser, `generic_dayahead` included. The public EPEX feed publishes
|
|
parallel arrays and no `data` key at all, so every live fetch that plugin
|
|
ever made raised `SourceUnavailable` while the suite stayed green: the
|
|
checks were grading the parser against the parser's own assumption. §11.4
|
|
then turned the failure into a quiet fall back to the static profile, so a
|
|
site would have sat on it forever.
|
|
|
|
The three plugins still fed from here - `engie_dynamic`,
|
|
`totalenergies_dynamic`, `eneco_eddy` - have NO captured body, because
|
|
their endpoints are unverified (TARIFF-07 is the open decision on whether
|
|
they are withdrawn). This fixture therefore grades that their DECLARATIONS
|
|
and the framework arithmetic are right; it grades NOTHING about what those
|
|
feeds publish, and it must not be read as evidence that they parse.
|
|
`generic_dayahead` is replayed from real bytes instead - section 2b.
|
|
"""
|
|
return json.dumps({"data": [
|
|
{"start": (T0 + timedelta(hours=i)).isoformat(),
|
|
key: v,
|
|
"provisional": (i == provisional_at)}
|
|
for i, v in enumerate(values)]}).encode()
|
|
|
|
|
|
def columnar_feed(values, provisional_at=None, unit="EUR / MWh",
|
|
step_min=60, start=None, prices=None):
|
|
"""A body in the shape `api.energy-charts.info` really publishes.
|
|
|
|
Parallel arrays and no per-point object; `unix_seconds` are UTC epoch
|
|
seconds. `prices` overrides the price column outright, which is how the
|
|
unequal-length case below is built without hand-writing JSON.
|
|
|
|
⚠️ THERE IS NO PER-POINT `provisional` IN THIS SHAPE, so `provisional_at`
|
|
exists only to be REFUSED - the columnar document cannot carry the flag and
|
|
the plugin supplies it. It is a keyword the callers do not pass; the
|
|
parameter is here so that a future caller who reaches for it gets a
|
|
TypeError rather than a silently ignored argument.
|
|
"""
|
|
if provisional_at is not None:
|
|
raise TypeError(
|
|
"the columnar document has no per-point provisional flag; "
|
|
"`generic_dayahead` declares it in fetch(). See "
|
|
"_points_from_columnar_json().")
|
|
t0 = start if start is not None else T0
|
|
secs = [int((t0 + timedelta(minutes=step_min * i)).timestamp())
|
|
for i in range(len(values))]
|
|
doc = {"unix_seconds": secs,
|
|
"price": list(values) if prices is None else list(prices),
|
|
"unit": unit,
|
|
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
|
|
"from Bundesnetzagentur | SMARD.de",
|
|
"deprecated": False}
|
|
return json.dumps(doc).encode()
|
|
|
|
|
|
CSV_BODY = ("start,eur_per_kwh,provisional\n"
|
|
"2026-01-15T00:00:00+01:00,0.1834,false\n"
|
|
"2026-01-15T01:00:00+01:00,-0.0210,true\n"
|
|
"2026-01-15T02:00:00+01:00,0.3115,false\n")
|
|
|
|
# The fixture, the transport that serves it, and - crucially - the ENERGY
|
|
# component of each point, computed here so the all-in expectation below is
|
|
# arithmetic this file did itself.
|
|
# ⚠️ `generic_dayahead` IS FED THE COLUMNAR SHAPE AND THE OTHER THREE ARE NOT,
|
|
# and the asymmetry is the honest one rather than an oversight: the columnar
|
|
# document is what the one endpoint anybody has ever reached actually returns.
|
|
FIXTURES = {
|
|
"generic_dayahead": (columnar_feed(EPEX_EUR_MWH),
|
|
[v / 1000 for v in EPEX_EUR_MWH]),
|
|
"engie_dynamic": (synthetic_row_feed("price_eur_kwh", RETAIL_EUR_KWH),
|
|
list(RETAIL_EUR_KWH)),
|
|
"totalenergies_dynamic": (synthetic_row_feed("price_eur_kwh",
|
|
RETAIL_EUR_KWH),
|
|
list(RETAIL_EUR_KWH)),
|
|
"eneco_eddy": (synthetic_row_feed("price_eur_mwh", EDDY_EUR_MWH),
|
|
[v / 1000 for v in EDDY_EUR_MWH]),
|
|
"manual_csv": (None, [0.1834, -0.0210, 0.3115]),
|
|
}
|
|
|
|
check(f"all five plugins the ticket lists are registered {sorted(PROVIDERS)}",
|
|
set(PROVIDERS) == {"generic_dayahead", "engie_dynamic",
|
|
"totalenergies_dynamic", "eneco_eddy", "manual_csv"})
|
|
check("...and the fixture set covers every one of them",
|
|
set(FIXTURES) == set(PROVIDERS))
|
|
|
|
|
|
def canned(body):
|
|
"""A transport that serves recorded bytes and records that it was used."""
|
|
calls = []
|
|
|
|
async def t(url, timeout_s):
|
|
calls.append((url, timeout_s))
|
|
return body
|
|
t.calls = calls
|
|
return t
|
|
|
|
|
|
def load(pid, extra=None, transport=None):
|
|
o = opts(price_provider=pid, **(extra or {}))
|
|
return run(load_prices(o, T0, T0 + timedelta(hours=48),
|
|
transport=transport))
|
|
|
|
|
|
import tempfile # noqa: E402
|
|
|
|
_csv_dir = tempfile.mkdtemp(prefix="tariff02-")
|
|
CSV_PATH = os.path.join(_csv_dir, "prices.csv")
|
|
with open(CSV_PATH, "w", encoding="utf-8") as fh:
|
|
fh.write(CSV_BODY)
|
|
|
|
|
|
def load_any(pid):
|
|
body, _ = FIXTURES[pid]
|
|
if pid == "manual_csv":
|
|
return load(pid, {"price_csv_path": CSV_PATH}, transport=refuse_transport)
|
|
return load(pid, transport=canned(body))
|
|
|
|
|
|
async def _refuse(url, timeout_s):
|
|
raise AssertionError(f"the offline plugin reached the network: {url}")
|
|
|
|
refuse_transport = _refuse
|
|
|
|
READS = {pid: load_any(pid) for pid in PROVIDERS}
|
|
|
|
_wrong_kind = sorted(p for p, r in READS.items() if r.kind != "series")
|
|
check(f"every plugin returned a series from its fixture", _wrong_kind == [])
|
|
|
|
# THE SCHEMA. Exactly three keys, in every point of every plugin, and the check
|
|
# derives the key set from the objects rather than asserting a typed literal.
|
|
_keys = derived(lambda: {frozenset(p.keys())
|
|
for r in READS.values()
|
|
for p in r.canonical()["prices"]})
|
|
check(f"every canonical point has exactly the three §9.3 keys, and only those "
|
|
f"-> {_keys}",
|
|
_keys == {frozenset({"start", "eur_per_kwh", "provisional"})})
|
|
check("the envelope is exactly {'prices': [...]}",
|
|
derived(lambda: all(set(r.canonical()) == {"prices"}
|
|
for r in READS.values())) is True)
|
|
_badtype = derived(lambda: sorted(
|
|
p for p, r in READS.items()
|
|
if not all(isinstance(pt["eur_per_kwh"], float)
|
|
and isinstance(pt["provisional"], bool)
|
|
and isinstance(pt["start"], str)
|
|
for pt in r.canonical()["prices"])))
|
|
check(f"...with a float price, a bool flag and an ISO string start {_badtype}",
|
|
_badtype == [])
|
|
_naive = derived(lambda: sorted(p for p, r in READS.items()
|
|
if any(pt.start.utcoffset() is None
|
|
for pt in r.points)))
|
|
check(f"every start is timezone-aware (§9.3 shows +01:00) {_naive}",
|
|
_naive == [])
|
|
_offsets = derived(lambda: {pt.start.utcoffset()
|
|
for r in READS.values() for pt in r.points})
|
|
check(f"...and at a Belgian winter offset for a January fixture {_offsets}",
|
|
_offsets == {timedelta(hours=1)})
|
|
check("the whole envelope round-trips through json.dumps",
|
|
derived(lambda: all(json.loads(json.dumps(r.canonical())) == r.canonical()
|
|
for r in READS.values())) is True)
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("2b. TARIFF-08 - the SHIPPED parser, replayed against REAL captured "
|
|
"bodies")
|
|
#
|
|
# ⚠️ WHY THIS SECTION EXISTS, WHICH IS THE MORE IMPORTANT HALF OF THE TICKET.
|
|
# Until 2026-09-10 every price fixture in this repository - here, in
|
|
# test_price_history.py and in test_static_tariff.py - was a
|
|
# `{"data": [{"start": ..., "price_eur_mwh": ...}]}` document invented by
|
|
# whoever wrote `_points_from_json()`. `api.energy-charts.info` has never
|
|
# published that shape. So the suite validated the parser against the parser's
|
|
# own assumption: a closed loop that cannot notice the assumption is wrong, and
|
|
# it stayed green while `generic_dayahead` raised `SourceUnavailable` on every
|
|
# live fetch it ever made. §11.4 then made that silent - a parse error becomes
|
|
# `no_data` becomes the static profile with arbitrage off, which is the CORRECT
|
|
# degradation and says nothing at all about the cause.
|
|
#
|
|
# The only thing that breaks the loop is bytes nobody here wrote. Both fixtures
|
|
# below are unmodified response bodies, captured by hand with curl on
|
|
# 2026-09-10; `scenarios/energy-charts-be-price-captures.json` records the URL,
|
|
# the retrieval time, the licence as returned and a sha256 of each, and the
|
|
# checksums are RE-DERIVED here so a fixture edited to make a check pass stops
|
|
# being a capture and starts being a failure.
|
|
#
|
|
# ⚠️ THE OTHER THREE NETWORKED PLUGINS HAVE NO CAPTURE AND THIS FILE DOES NOT
|
|
# PRETEND OTHERWISE. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy`
|
|
# point at endpoints nobody has verified; TARIFF-07 is the open decision on
|
|
# whether they are withdrawn. Their fixtures are named `synthetic_row_feed` so
|
|
# no reader can mistake them for evidence that those feeds parse, and the check
|
|
# below states, derived from the shipped `PROVIDERS` registry, exactly which
|
|
# plugins are replayed from real bytes and which are not.
|
|
|
|
_CAPTURES = json.load(open(
|
|
os.path.join(HERE, "scenarios", "energy-charts-be-price-captures.json"),
|
|
encoding="utf-8"))
|
|
|
|
|
|
def _capture(name):
|
|
with open(os.path.join(HERE, "scenarios", name), "rb") as fh:
|
|
return fh.read()
|
|
|
|
|
|
_REAL = {n: _capture(n) for n in _CAPTURES["files"]}
|
|
_bad_sha = sorted(n for n, b in _REAL.items()
|
|
if hashlib.sha256(b).hexdigest()
|
|
!= _CAPTURES["files"][n]["sha256"])
|
|
check(f"all three captured bodies are the bytes the provenance file "
|
|
f"recorded {sorted(_REAL)}",
|
|
len(_REAL) == 3 and _bad_sha == [])
|
|
|
|
# The document really is columnar, derived from the bytes rather than asserted:
|
|
# if it ever grows a `data` key this check says so before any parser does.
|
|
_shapes = {n: sorted(json.loads(b.decode("utf-8"))) for n, b in _REAL.items()}
|
|
check(f"...and every one is COLUMNAR - parallel arrays, no 'data' key "
|
|
f"{sorted({tuple(v) for v in _shapes.values()})}",
|
|
all(v == ["deprecated", "license_info", "price", "unit", "unix_seconds"]
|
|
for v in _shapes.values()))
|
|
check("...which is precisely the shape the shipped parser used to demand a "
|
|
"'data' list from",
|
|
all("data" not in v for v in _shapes.values()))
|
|
|
|
# ⚠️ CC BY 4.0 IS WHY THESE BYTES MAY BE IN THE REPOSITORY AT ALL. DATA-02
|
|
# established that the BE zone is redistributable with attribution; the feed
|
|
# restates the licence per response, so a capture whose own licence line has
|
|
# changed is a capture that must not stay committed.
|
|
_lic = {n: json.loads(b.decode("utf-8"))["license_info"] for n, b in _REAL.items()}
|
|
check(f"...and each carries the CC BY 4.0 line the provenance block cites",
|
|
all("CC BY 4.0" in v for v in _lic.values())
|
|
and all(_lic[n] == _CAPTURES["files"][n]["licence_as_returned"]
|
|
for n in _REAL))
|
|
|
|
_QH = "energy-charts-be-price-2026-09-10-quarterhourly.json"
|
|
_H1 = "energy-charts-be-price-2025-09-10-hourly.json"
|
|
|
|
# ⚠️ THE TWO GRANULARITIES ARE NOT DECORATION. Belgium moved to a 15-minute MTU
|
|
# on 2025-10-01 and the same endpoint answers 96 points at 900 s for a day after
|
|
# it and 24 at 3600 s for a day before it. A fixture at one spacing cannot grade
|
|
# whether anything assumes the other, so both are replayed and the spacings are
|
|
# DERIVED from the bodies rather than typed.
|
|
_spacing = {}
|
|
for _n, _b in _REAL.items():
|
|
_u = json.loads(_b.decode("utf-8"))["unix_seconds"]
|
|
_spacing[_n] = sorted({_u[i + 1] - _u[i] for i in range(len(_u) - 1)})
|
|
check(f"the two captures really differ in granularity - {_spacing[_QH]} s over "
|
|
f"{len(json.loads(_REAL[_QH])['price'])} points and {_spacing[_H1]} s "
|
|
f"over {len(json.loads(_REAL[_H1])['price'])} points - or replaying both "
|
|
f"grades nothing more than replaying one",
|
|
_spacing[_QH] == [900] and _spacing[_H1] == [3600])
|
|
|
|
_REPLAY = {n: load("generic_dayahead", transport=canned(b))
|
|
for n, b in _REAL.items()}
|
|
_not_series = sorted(n for n, r in _REPLAY.items() if r.kind != "series")
|
|
check(f"the SHIPPED generic_dayahead parses both real bodies "
|
|
f"{[r.words for r in _REPLAY.values()]}",
|
|
_not_series == [])
|
|
|
|
# The point count is the source's own interval count, taken from the document.
|
|
_counts = derived(lambda: {n: len(r.points) for n, r in _REPLAY.items()})
|
|
_want_counts = {n: len(json.loads(b.decode("utf-8"))["price"])
|
|
for n, b in _REAL.items()}
|
|
check(f"...emitting one point per published interval, no decimation "
|
|
f"{_counts}", _counts == _want_counts)
|
|
check(f"...which for the quarter-hourly body is 96 points and not 24 - the "
|
|
f"15-minute MTU is the shape of the real feed, not a hypothetical",
|
|
_want_counts[_QH] == 96 and _want_counts[_H1] == 24)
|
|
|
|
# §9.3: unix_seconds are UTC instants, RENDERED at the Brussels offset. Both
|
|
# captured days are inside CEST, so the correct first start is local midnight at
|
|
# +02:00. ⚠️ A NAIVE READING - `datetime.fromtimestamp(t)` with no tzinfo - is
|
|
# what this grades: it would produce the host's local wall clock, and on a UTC
|
|
# host that is 22:00 of the previous day. The expectation is computed here from
|
|
# the raw epoch second by different arithmetic than price.py uses.
|
|
_first_raw = {n: json.loads(b.decode("utf-8"))["unix_seconds"][0]
|
|
for n, b in _REAL.items()}
|
|
_first_utc = {n: datetime(1970, 1, 1, tzinfo=UTC) + timedelta(seconds=t)
|
|
for n, t in _first_raw.items()}
|
|
_first_got = derived(lambda: {n: r.points[0].start
|
|
for n, r in _REPLAY.items()})
|
|
check(f"every start is the UTC instant the feed published "
|
|
f"{ {n: v.isoformat() for n, v in _first_utc.items()} }",
|
|
isinstance(_first_got, dict)
|
|
and all(_first_got[n] == _first_utc[n] for n in _REAL))
|
|
check(f"...rendered at the Brussels offset, so a September day starts at "
|
|
f"00:00+02:00 and NOT at the 22:00 a naive local reading gives "
|
|
f"{ {n: v.isoformat() for n, v in (_first_got if isinstance(_first_got, dict) else {}).items()} }",
|
|
isinstance(_first_got, dict)
|
|
and all(v.utcoffset() == timedelta(hours=2) for v in _first_got.values())
|
|
and all((v.hour, v.minute) == (0, 0) for v in _first_got.values()))
|
|
|
|
# EUR/MWh -> all-in EUR/kWh, recomputed here from the raw column.
|
|
_allin_want = {n: [round(v / 1000 + MARKUP + TAXES, 6)
|
|
for v in json.loads(b.decode("utf-8"))["price"]]
|
|
for n, b in _REAL.items()}
|
|
_allin_got = derived(lambda: {n: [p.eur_per_kwh for p in r.points]
|
|
for n, r in _REPLAY.items()})
|
|
check(f"...and every price is the published EUR/MWh figure over 1000 plus "
|
|
f"markup and taxes (first: {[v[0] for v in _allin_want.values()]})",
|
|
_allin_got == _allin_want)
|
|
# ⚠️ NON-DEGENERACY. If the captured prices happened to be all equal, the
|
|
# arithmetic check above would pass under a parser that emitted the same number
|
|
# for every interval.
|
|
_raw_spread = {n: round(max(v) - min(v), 6)
|
|
for n, v in ((k, json.loads(b.decode("utf-8"))["price"])
|
|
for k, b in _REAL.items())}
|
|
check(f"...on bodies whose prices actually vary, or that check grades a "
|
|
f"constant {_raw_spread}",
|
|
all(v > 1.0 for v in _raw_spread.values()))
|
|
|
|
|
|
# --- the refusals, driven from the real body by breaking one thing ---------- #
|
|
|
|
def _edited(name, **fields):
|
|
"""The captured body with one field replaced. Everything else is real."""
|
|
doc = json.loads(_REAL[name].decode("utf-8"))
|
|
doc.update(fields)
|
|
return json.dumps(doc).encode()
|
|
|
|
|
|
_full = json.loads(_REAL[_QH].decode("utf-8"))
|
|
_short = load("generic_dayahead",
|
|
transport=canned(_edited(_QH, price=_full["price"][:-4])))
|
|
check(f"a `price` array 4 entries short of `unix_seconds` is no_data, NOT a "
|
|
f"series truncated to {len(_full['price']) - 4} points - the missing "
|
|
f"tail of a Belgian day is the evening peak, and a short series is "
|
|
f"traded on where no data at all switches arbitrage off (§11.4)",
|
|
_short.kind == "no_data")
|
|
check("...and the reason names the two lengths, so an operator can see it was "
|
|
"a length mismatch and not an outage",
|
|
_short.kind == "no_data"
|
|
and str(len(_full["unix_seconds"])) in _short.reason
|
|
and str(len(_full["price"]) - 4) in _short.reason)
|
|
# The other direction, so the check is not a one-sided `len(price) < len(time)`.
|
|
_long = load("generic_dayahead",
|
|
transport=canned(_edited(_QH, price=_full["price"] + [1.0])))
|
|
check("...and a `price` array one entry LONGER is refused too",
|
|
_long.kind == "no_data")
|
|
# ⚠️ THE CONTROL. Without it, "refuses everything" passes both checks above.
|
|
check("...while the untouched body is still a series, so the refusal is the "
|
|
"length and not the editing",
|
|
load("generic_dayahead",
|
|
transport=canned(_edited(_QH))).kind == "series")
|
|
|
|
_unit = load("generic_dayahead", transport=canned(_edited(_QH,
|
|
unit="EUR / kWh")))
|
|
check("a feed that changes its unit to EUR/kWh is no_data, not every price "
|
|
"silently divided by 1000 - MAX_PLAUSIBLE_EUR_KWH only bounds prices "
|
|
"that are too LARGE, so nothing downstream could catch it",
|
|
_unit.kind == "no_data")
|
|
|
|
_nulls = _full["price"][:]
|
|
for _i in (3, 17):
|
|
_nulls[_i] = None
|
|
_nulled = load("generic_dayahead", transport=canned(_edited(_QH,
|
|
price=_nulls)))
|
|
check(f"a published null is DROPPED, leaving a hole rather than a zero - a "
|
|
f"missing quarter is not a free quarter",
|
|
_nulled.kind == "series"
|
|
and derived(lambda: len(_nulled.points)) == len(_full["price"]) - 2)
|
|
check("...and the dropped instants are the ones the feed left null, not the "
|
|
"first two",
|
|
derived(lambda: sorted(
|
|
int(p.start.timestamp()) for p in _nulled.points))
|
|
== sorted(t for i, t in enumerate(_full["unix_seconds"])
|
|
if i not in (3, 17)))
|
|
_allnull = load("generic_dayahead",
|
|
transport=canned(_edited(_QH,
|
|
price=[None] * len(_full["price"]))))
|
|
check("...but a body with a price for nothing at all is no_data, not an empty "
|
|
"series", _allnull.kind == "no_data")
|
|
# ⚠️ WHICH RULE REFUSED IT, not merely that something did. `_validate_raw()`
|
|
# would also refuse an empty list, one layer further out and with a sentence
|
|
# that says nothing about the feed - so asserting only `no_data` here would
|
|
# grade the same under a parser that had lost this refusal entirely.
|
|
check(f"...refused by the COLUMNAR parser, naming what arrived, rather than "
|
|
f"reaching the boundary validator as a generic empty list - "
|
|
f"{_allnull.reason!r}",
|
|
_allnull.kind == "no_data"
|
|
and f"{len(_full['price'])} instants" in _allnull.reason
|
|
and "a price for none of them" in _allnull.reason)
|
|
|
|
# ⚠️ THE TWO WARNINGS ARE GRADED, NOT ASSUMED. §11.4 is exactly what makes a
|
|
# price defect quiet - a parse failure becomes no_data becomes the static
|
|
# profile, correctly and silently - so an unverified diagnostic is the first
|
|
# thing here to rot. Both are captured off the shipped logger.
|
|
_wbuf = io.StringIO()
|
|
_wh = logging.StreamHandler(_wbuf)
|
|
logging.getLogger("goodwe.price").addHandler(_wh)
|
|
load("generic_dayahead", transport=canned(_edited(_QH, price=_nulls)))
|
|
_dep = load("generic_dayahead", transport=canned(_edited(_QH,
|
|
deprecated=True)))
|
|
logging.getLogger("goodwe.price").removeHandler(_wh)
|
|
_warn = _wbuf.getvalue()
|
|
_n_null = len([v for v in _nulls if v is None])
|
|
check(f"the dropped nulls are reported with a count ({_n_null} of "
|
|
f"{len(_nulls)}), so a series with holes in it is visible to an operator "
|
|
f"rather than merely short",
|
|
f"{_n_null} of {len(_nulls)}" in _warn)
|
|
check("...and a feed announcing deprecated=true says so WHILE IT STILL WORKS - "
|
|
"the day it stops, §11.4 makes its death look exactly like a quiet one",
|
|
"deprecated" in _warn and _dep.kind == "series")
|
|
|
|
# Which plugins are replayed from real bytes, derived from the registry so a
|
|
# sixth supplier cannot be added without this line noticing.
|
|
_networked = sorted(pid for pid, cls in PROVIDERS.items()
|
|
if cls.capabilities().egress_hosts)
|
|
_replayed = ["generic_dayahead"]
|
|
check(f"the networked plugins are {_networked}; {_replayed} is replayed from a "
|
|
f"captured real body and {sorted(set(_networked) - set(_replayed))} have "
|
|
f"no capture because their endpoints are unverified (TARIFF-07)",
|
|
_networked == sorted(["generic_dayahead", "engie_dynamic",
|
|
"totalenergies_dynamic", "eneco_eddy"]))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("2c. TARIFF-09 - the RANGE the caller asked for, on the Brussels "
|
|
"calendar")
|
|
#
|
|
# ⚠️ WHAT WAS WRONG, AND WHY NO CHECK IN THIS FILE COULD SEE IT.
|
|
# `generic_dayahead.fetch()` accepted `since` and `until` and requested the
|
|
# bare `https://api.energy-charts.info/price?bzn=BE`, which answers ONE
|
|
# Brussels day. Everything past today was then filled from yesterday's profile
|
|
# by FCST-04, flagged provisional, and §8.3 dropped all of it from the trade
|
|
# set - so the optimiser never saw tomorrow as firm, at any hour of any day.
|
|
# Measured on the merged tree at 2026-09-10T09:00+02:00 over a 48 h horizon:
|
|
# 15 firm whole hours and 33 carried, both ends of the only permitted cycle
|
|
# inside ONE calendar day; and at 23:00, 1 firm hour and 47 carried, where the
|
|
# firm spread no longer clears the round-trip cost at all.
|
|
#
|
|
# The reason section 2b could not catch it is that section 2b fed the parser
|
|
# BYTES and never looked at the REQUEST. `canned()` has always recorded the URL
|
|
# it was asked for and nothing asserted on it - so the transport knew the
|
|
# plugin was ignoring its arguments and no check ever asked. The checks below
|
|
# assert on the URL, and the two-day body they replay is a capture of what that
|
|
# URL really answers.
|
|
|
|
_TD = "energy-charts-be-price-2026-09-09_2026-09-10-twoday.json"
|
|
|
|
# ⚠️ NON-DEGENERACY FIRST (habit 1): the two-day capture must really cover TWO
|
|
# Brussels days and the one-day capture ONE, or "the range reached the feed"
|
|
# is graded against two fixtures that are the same shape.
|
|
_bdays = {n: sorted({brussels(datetime.fromtimestamp(t, UTC)).date().isoformat()
|
|
for t in json.loads(b.decode("utf-8"))["unix_seconds"]})
|
|
for n, b in _REAL.items()}
|
|
check(f"the captures span the local-day counts this section needs - "
|
|
f"{ {n: len(v) for n, v in _bdays.items()} } - or the range check has "
|
|
f"nothing to distinguish {_bdays[_TD]} from {_bdays[_QH]}",
|
|
len(_bdays[_TD]) == 2 and len(_bdays[_QH]) == 1
|
|
and len(_bdays[_H1]) == 1)
|
|
_npts = {n: len(json.loads(b.decode("utf-8"))["price"])
|
|
for n, b in _REAL.items()}
|
|
check(f"...and the two-day body carries 192 quarter-hours where the one-day "
|
|
f"body carries 96, which is the counterfactual this card is built on "
|
|
f"{_npts}",
|
|
len(json.loads(_REAL[_TD].decode("utf-8"))["price"]) == 192
|
|
and len(json.loads(_REAL[_QH].decode("utf-8"))["price"]) == 96)
|
|
|
|
# --- the URL the SHIPPED plugin asks for ----------------------------------- #
|
|
|
|
|
|
def asked_url(since, until, extra=None):
|
|
"""The URL `generic_dayahead` really requests for a window. Behaviour,
|
|
not source text: it comes back off the injected transport."""
|
|
t = canned(_REAL[_TD])
|
|
o = opts(price_provider="generic_dayahead", **(extra or {}))
|
|
run(load_prices(o, since, until, transport=t))
|
|
return t.calls[0][0]
|
|
|
|
|
|
# The expectation is built HERE, from the window, by different arithmetic than
|
|
# price.py uses: a literal string with the two dates spelled out.
|
|
_w0 = datetime.fromisoformat("2026-09-09T09:17:43+02:00")
|
|
check(f"a 48 h horizon from 09:17 local asks the feed for the three Brussels "
|
|
f"days it touches -> {asked_url(_w0, _w0 + timedelta(hours=48))!r}",
|
|
asked_url(_w0, _w0 + timedelta(hours=48))
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-09&end=2026-09-11")
|
|
check("...and the shipped default no longer asks the bare URL that answers a "
|
|
"single day",
|
|
asked_url(_w0, _w0 + timedelta(hours=48))
|
|
!= "https://api.energy-charts.info/price?bzn=BE")
|
|
|
|
# ⚠️ THE BRUSSELS/UTC BOUNDARY, WHICH IS THE TRAP. `since`/`until` are
|
|
# INSTANTS and the API takes DATES. At 22:30 UTC in September Belgium is
|
|
# already on the NEXT calendar day, so a range derived from `since.date()` -
|
|
# or from `since.astimezone(utc).date()`, which is the same thing - asks for
|
|
# the day the controller has just left and drops the day it is standing in.
|
|
# This is `_bdate()`'s lesson one layer up: price_history.py MISFILES an hour
|
|
# when it gets this wrong; here the hours are never fetched at all.
|
|
_edge = datetime.fromisoformat("2026-09-10T22:30:00+00:00")
|
|
check(f"the precondition: {_edge.isoformat()} really is a different day in "
|
|
f"Brussels ({brussels(_edge).date().isoformat()}) than in UTC "
|
|
f"({_edge.date().isoformat()}), or the boundary check grades nothing",
|
|
brussels(_edge).date().isoformat() != _edge.date().isoformat())
|
|
check(f"...so the range starts on the BRUSSELS day, 2026-09-11, and not on "
|
|
f"the UTC day 2026-09-10 -> "
|
|
f"{asked_url(_edge, _edge + timedelta(hours=12))!r}",
|
|
asked_url(_edge, _edge + timedelta(hours=12))
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-11&end=2026-09-11")
|
|
|
|
# ...and the SAME trap on the other end. `until` at 22:30 UTC is already
|
|
# 00:30 of the next Brussels day, so a UTC-derived END stops one day short and
|
|
# the last hours of the horizon are never fetched. Both ends of the range
|
|
# derivation are graded, or a mutation only has to get one of them right.
|
|
_edge2_a = datetime.fromisoformat("2026-09-09T12:00:00+00:00")
|
|
_edge2_b = datetime.fromisoformat("2026-09-10T22:30:00+00:00")
|
|
check(f"the precondition: {_edge2_b.isoformat()} is "
|
|
f"{brussels(_edge2_b).date().isoformat()} in Brussels and "
|
|
f"{_edge2_b.date().isoformat()} in UTC, so the two answers differ",
|
|
brussels(_edge2_b).date().isoformat() != _edge2_b.date().isoformat())
|
|
check(f"...so the range ENDS on the Brussels day 2026-09-11, not on the UTC "
|
|
f"day 2026-09-10 -> {asked_url(_edge2_a, _edge2_b)!r}",
|
|
asked_url(_edge2_a, _edge2_b)
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-09&end=2026-09-11")
|
|
|
|
# ⚠️ `until` IS EXCLUSIVE, so local midnight belongs to the day BEFORE it. A
|
|
# 24 h horizon from local midnight is ONE day, not two.
|
|
_mid = datetime.fromisoformat("2026-09-09T00:00:00+02:00")
|
|
check(f"a horizon ending exactly at local midnight asks for one day, because "
|
|
f"`until` is exclusive -> {asked_url(_mid, _mid + timedelta(hours=24))!r}",
|
|
asked_url(_mid, _mid + timedelta(hours=24))
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-09&end=2026-09-09")
|
|
check(f"...while one second past midnight is two days, so the rule is the "
|
|
f"exclusive END and not a fixed subtraction -> "
|
|
f"{asked_url(_mid, _mid + timedelta(hours=24, seconds=1))!r}",
|
|
asked_url(_mid, _mid + timedelta(hours=24, seconds=1))
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-09&end=2026-09-10")
|
|
check(f"an empty or inverted window collapses to a single day rather than "
|
|
f"asking for a backwards range no feed has a defined answer for -> "
|
|
f"{asked_url(_mid, _mid - timedelta(hours=5))!r}",
|
|
asked_url(_mid, _mid - timedelta(hours=5))
|
|
== "https://api.energy-charts.info/price?bzn=BE"
|
|
"&start=2026-09-09&end=2026-09-09")
|
|
|
|
# The helper on its own, over the DST transitions, where a fixed-offset
|
|
# assumption and a correct one disagree.
|
|
_mar = datetime.fromisoformat("2026-03-29T00:30:00+00:00") # 02:30 CEST
|
|
_bdr = derived(lambda: brussels_date_range(_mar, _mar + timedelta(hours=1)))
|
|
check(f"brussels_date_range() is right across the spring transition, where "
|
|
f"00:30 UTC is already 02:30 local -> {_bdr}",
|
|
_bdr == (date(2026, 3, 29), date(2026, 3, 29)))
|
|
|
|
# --- the two-day body, end to end through the SHIPPED parser --------------- #
|
|
|
|
_two = load("generic_dayahead", transport=canned(_REAL[_TD]))
|
|
check(f"the shipped generic_dayahead parses the real two-day body "
|
|
f"{_two.words}", _two.kind == "series")
|
|
def _series_days(read):
|
|
return derived(lambda: sorted({brussels(p.start).date().isoformat()
|
|
for p in read.points}))
|
|
|
|
|
|
_two_days = _series_days(_two)
|
|
check(f"...and the series it produces spans TWO Brussels local days "
|
|
f"{_two_days}, where the one-day body's spans one "
|
|
f"{_series_days(_REPLAY[_QH])}",
|
|
_two_days == ["2026-09-09", "2026-09-10"]
|
|
and len(_two_days) == 2)
|
|
check(f"...one point per published interval, 192 of them, no decimation and "
|
|
f"no day dropped ({derived(lambda: len(_two.points))})",
|
|
derived(lambda: len(_two.points)) == 192)
|
|
|
|
# --- the SHORT ANSWER, which is normal and must stay a series -------------- #
|
|
#
|
|
# ⚠️ ASKING FOR TOMORROW BEFORE THE ~13:00-14:00 CET AUCTION RETURNS FEWER DAYS
|
|
# THAN REQUESTED, AND THAT IS NOT AN ERROR. Measured: at 2026-09-10T07:46:55Z
|
|
# the request `?bzn=BE&start=2026-09-10&end=2026-09-11` came back BYTE-IDENTICAL
|
|
# to the bare-URL capture - one day, 96 quarter-hours. A `no_data` there would
|
|
# switch a working site's arbitrage off every morning, so the requirement is
|
|
# that the short body stays a `series` of what EXISTS and FCST-04's fill covers
|
|
# the rest with flagged, provisional hours. The identity is re-derived from the
|
|
# bytes rather than restated.
|
|
_prepub = _CAPTURES["files"][_QH]["also_returned_verbatim_by"]
|
|
check(f"the pre-publication range request {_prepub['url']!r} returned the "
|
|
f"SAME BYTES as the bare default URL - one day where two were asked for",
|
|
hashlib.sha256(_REAL[_QH]).hexdigest()
|
|
== _CAPTURES["files"][_QH]["sha256"]
|
|
and "start=2026-09-10&end=2026-09-11" in _prepub["url"])
|
|
check(f"...and the shipped parser answers that short body with a SERIES of "
|
|
f"what exists, never no_data - {_REPLAY[_QH].words}",
|
|
_REPLAY[_QH].kind == "series"
|
|
and derived(lambda: len(_REPLAY[_QH].points)) == 96)
|
|
# ⚠️ THE COUNTS ARE TAKEN THROUGH `derived()` AND COMPARED ONLY WHEN BOTH ARE
|
|
# NUMBERS. A mutation that turns the short answer into `no_data` makes
|
|
# `.points` raise `Absence` by design, and `_Raised < _Raised` is a TypeError -
|
|
# which takes the file down with a traceback instead of reporting a FAIL. A
|
|
# check whose failure mode is a crash has no name attached (habit 5's shape,
|
|
# with an exception in place of a hang).
|
|
_n_short = derived(lambda: len(_REPLAY[_QH].points))
|
|
_n_two = derived(lambda: len(_two.points))
|
|
check(f"...which is strictly fewer points than the two-day body "
|
|
f"({_n_short} < {_n_two}), so 'short' is a real property of this "
|
|
f"stimulus and not a name for it",
|
|
isinstance(_n_short, int) and isinstance(_n_two, int)
|
|
and _n_short < _n_two)
|
|
|
|
# --- price_feed_url: the exact URL, and the cost said out loud ------------- #
|
|
#
|
|
# ⚠️ THE DECISION. An override is used VERBATIM and this plugin appends
|
|
# nothing to it. Merging our `start`/`end` into an operator's query string
|
|
# would mean re-serialising a URL they wrote, and `start`/`end` are
|
|
# energy-charts' spelling - and this provider also DECLARES a SECOND host,
|
|
# `web-api.tp.entsoe.eu`, whose date parameters are not those and are not
|
|
# verified here, so appending our vocabulary there yields two ignored
|
|
# parameters and the short horizon anyway. The cost is
|
|
# that an overriding site silently gets the horizon this card exists to fix,
|
|
# so it is NOT silent: a warning names the consequence at fetch time.
|
|
_ovr = "https://api.energy-charts.info/price?bzn=BE&somethingelse=1"
|
|
check(f"an override is requested EXACTLY as given, with nothing appended -> "
|
|
f"{asked_url(_w0, _w0 + timedelta(hours=48), {'price_feed_url': _ovr})!r}",
|
|
asked_url(_w0, _w0 + timedelta(hours=48),
|
|
{"price_feed_url": _ovr}) == _ovr)
|
|
|
|
_obuf = io.StringIO()
|
|
_oh = logging.StreamHandler(_obuf)
|
|
logging.getLogger("goodwe.price").addHandler(_oh)
|
|
asked_url(_w0, _w0 + timedelta(hours=48), {"price_feed_url": _ovr})
|
|
_ovr_log = _obuf.getvalue()
|
|
_obuf2 = io.StringIO()
|
|
_oh2 = logging.StreamHandler(_obuf2)
|
|
logging.getLogger("goodwe.price").removeHandler(_oh)
|
|
logging.getLogger("goodwe.price").addHandler(_oh2)
|
|
asked_url(_w0, _w0 + timedelta(hours=48))
|
|
_dflt_log = _obuf2.getvalue()
|
|
logging.getLogger("goodwe.price").removeHandler(_oh2)
|
|
check(f"...and the operator is TOLD what it costs them, at fetch time, naming "
|
|
f"the horizon consequence rather than only the substitution",
|
|
"price_feed_url" in _ovr_log and _ovr in _ovr_log
|
|
and "provisional" in _ovr_log and "2026-09-11" in _ovr_log)
|
|
# ⚠️ THE CONTROL. Without it, "warns about everything" passes the line above.
|
|
check("...while the DEFAULT path says nothing, so the warning is about the "
|
|
"override and not about fetching",
|
|
"price_feed_url is set" not in _dflt_log)
|
|
|
|
# The builder refuses to append to a base that already has a query string,
|
|
# which is the "URL nobody intended" this decision is meant to make
|
|
# unreachable. Driven, not read off the source.
|
|
check(f"dated_feed_url() refuses a base that already carries a query string, "
|
|
f"rather than producing a second '?' - "
|
|
f"{_reason(lambda: dated_feed_url('https://h/p?a=1', b='2'))!r}",
|
|
raises(lambda: dated_feed_url("https://h/p?a=1", b="2"),
|
|
SourceUnavailable))
|
|
check("...and a clean base still builds, so the refusal is the existing query "
|
|
"string and not the builder",
|
|
dated_feed_url("https://h/p", b="2") == "https://h/p?b=2")
|
|
|
|
# --- the plugin contract: who consumes the range, pinned ------------------- #
|
|
#
|
|
# ⚠️ THIS IS A SOURCE-SHAPE PIN AND IS NAMED AS ONE (habit 4). The claim is
|
|
# about the CODE - which plugins reference the range arguments they are handed
|
|
# - which is the same kind of claim `code_leaks()` makes, and it is not
|
|
# evidence that any feed parses. `generic_dayahead`'s BEHAVIOUR is graded above
|
|
# by the URL it really requests.
|
|
#
|
|
# It exists because `fetch(self, opts, since, until, transport)` promises every
|
|
# plugin the range and the promise is NOT enforced. Enforcing it would oblige
|
|
# the three plugins whose endpoints nobody has verified (TARIFF-07) to invent a
|
|
# date-parameter vocabulary - TARIFF-08's defect moved from the response to the
|
|
# request. So the split is pinned instead, and a sixth supplier has to change
|
|
# this line rather than inherit the silence.
|
|
def _uses_range(cls):
|
|
src = dedent(inspect.getsource(cls.fetch))
|
|
tree = ast.parse(src)
|
|
names = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}
|
|
return bool(names & {"since", "until"})
|
|
|
|
|
|
_uses = sorted(pid for pid, cls in PROVIDERS.items() if _uses_range(cls))
|
|
check(f"the plugins that CONSUME their `since`/`until` are {_uses}; the rest "
|
|
f"take the range and ignore it, which is what TARIFF-09 fixed for "
|
|
f"generic_dayahead and what TARIFF-07 has to decide for the others",
|
|
_uses == ["generic_dayahead"])
|
|
check("...and `manual_csv` is honestly among the ones that do not: it reads a "
|
|
"whole file and the fill windows it downstream, so a range parameter "
|
|
"would have nowhere to go",
|
|
"manual_csv" in sorted(set(PROVIDERS) - set(_uses)))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("3. all-in: energy + markup + taxes, added exactly once (AC1, trap 1)")
|
|
# The expected figure is computed HERE, from the fixture's energy component and
|
|
# the plugin's own declaration, by different arithmetic than assemble() uses.
|
|
for pid, read in sorted(READS.items()):
|
|
caps = PROVIDERS[pid].capabilities()
|
|
_, energy = FIXTURES[pid]
|
|
add = (MARKUP if Component.MARKUP in caps.adds else 0.0) + \
|
|
(TAXES if Component.TAXES in caps.adds else 0.0)
|
|
want = [round(e + add, 6) for e in energy]
|
|
got = derived(lambda r=read: [p.eur_per_kwh for p in r.points])
|
|
check(f"{pid}: all-in == published {sorted(c.value for c in caps.sources)} "
|
|
f"+ framework {sorted(c.value for c in caps.adds) or 'nothing'} "
|
|
f"-> {got}",
|
|
got == want)
|
|
|
|
# ⚠️ THE ONE THAT MATTERS. generic_dayahead is the only plugin whose feed is
|
|
# wholesale, so it is the only one that gets both components - and the three
|
|
# retail plugins must each get strictly less. Derived by comparing the two
|
|
# plugins' output on the SAME energy figure rather than by restating numbers.
|
|
def added(pid):
|
|
return derived(lambda: round(READS[pid].points[0].eur_per_kwh
|
|
- FIXTURES[pid][1][0], 6))
|
|
|
|
|
|
_generic_add, _engie_add, _eddy_add = (added("generic_dayahead"),
|
|
added("engie_dynamic"),
|
|
added("eneco_eddy"))
|
|
check(f"the wholesale feed gets markup+taxes ({_generic_add})",
|
|
isinstance(_generic_add, float)
|
|
and abs(_generic_add - (MARKUP + TAXES)) < 1e-9)
|
|
check(f"a retail feed that already carries the markup gets taxes only "
|
|
f"({_engie_add}), one markup less than the wholesale one",
|
|
isinstance(_engie_add, float) and isinstance(_generic_add, float)
|
|
and abs(_engie_add - TAXES) < 1e-9
|
|
and abs((_generic_add - _engie_add) - MARKUP) < 1e-9)
|
|
check(f"an all-in feed gets nothing added ({_eddy_add})",
|
|
isinstance(_eddy_add, float) and abs(_eddy_add) < 1e-9)
|
|
|
|
# Negative prices survive. A framework that clamped them would turn the best
|
|
# charging hour of the year into an average one, silently.
|
|
_neg = derived(lambda: [p.eur_per_kwh for p in READS["generic_dayahead"].points
|
|
if p.eur_per_kwh < 0])
|
|
check(f"a negative day-ahead price passes through, all-in {_neg}",
|
|
_neg == [round(EPEX_EUR_MWH[1] / 1000 + MARKUP + TAXES, 6)])
|
|
|
|
# provisional is per point and comes from the feed, not from the series.
|
|
#
|
|
# ⚠️ `generic_dayahead` IS EXCLUDED FROM THE "MIXED" REQUIREMENT, AND THE
|
|
# EXCLUSION IS THE POINT RATHER THAN A GAP. The columnar document the real feed
|
|
# publishes has no per-point provisional field - it is a settled auction result
|
|
# - so the plugin declares `False` for all of them. Demanding a mixed series
|
|
# from it would mean feeding it an invented body carrying an invented flag,
|
|
# which is exactly the closed loop TARIFF-08 exists to remove. It is graded on
|
|
# what its real body can grade instead, immediately below.
|
|
for pid in sorted(set(READS) - {"generic_dayahead"}):
|
|
read = READS[pid]
|
|
flags = derived(lambda r=read: [p.provisional for p in r.points])
|
|
check(f"{pid}: provisional is per point and mixed {flags}",
|
|
isinstance(flags, list) and any(flags) and not all(flags))
|
|
check(f"...so plan_is_provisional is derived from those points",
|
|
derived(lambda r=read: plan_is_provisional(r)) is True)
|
|
_gen_flags = derived(lambda: [p.provisional
|
|
for p in READS["generic_dayahead"].points])
|
|
check(f"generic_dayahead's settled auction comes out settled at every point "
|
|
f"{_gen_flags}",
|
|
isinstance(_gen_flags, list) and len(_gen_flags) == len(EPEX_EUR_MWH)
|
|
and not any(_gen_flags))
|
|
check("...so plan_is_provisional is False for it, derived from the same points",
|
|
derived(lambda: plan_is_provisional(READS["generic_dayahead"])) is False)
|
|
# ...and the same function says False when nothing is provisional. Without this
|
|
# arm, "always True" would pass every check above.
|
|
_settled = PriceRead.series(
|
|
[PricePoint(brussels(T0), 0.2, False)], PriceProduct.DAY_AHEAD, "x", T0)
|
|
check("a fully settled series is NOT provisional",
|
|
plan_is_provisional(_settled) is False)
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("4. double-counting the markup is detectable, and unreachable (trap 1)")
|
|
|
|
|
|
class _Rogue(PriceProvider):
|
|
"""A plugin that adds the markup itself while declaring it did not.
|
|
|
|
NOT registered - it exists only here, as the mutation the design is meant to
|
|
survive. It is what a plugin author writes when they copy the wholesale
|
|
plugin, hardcode their supplier's margin, and forget to update `sources`.
|
|
"""
|
|
|
|
id = "_rogue_not_shipped"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY}))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
# The margin the author hardcoded, because they could not read the
|
|
# configured one - see the FrameworkOption check below.
|
|
return [RawPoint(T0, RETAIL_EUR_KWH[0], False)]
|
|
|
|
|
|
_honest_caps = PROVIDERS["engie_dynamic"].capabilities()
|
|
_honest = assemble(_honest_caps, [RawPoint(T0, RETAIL_EUR_KWH[0], False)],
|
|
MARKUP, TAXES)[0].eur_per_kwh
|
|
_rogue_raw = run(_Rogue().fetch(PluginOptions({}), T0, T0, None))
|
|
_rogue = assemble(_Rogue.capabilities(), _rogue_raw, MARKUP, TAXES)[0].eur_per_kwh
|
|
_delta = derived(lambda: round(_rogue - _honest, 6))
|
|
check(f"a plugin that pre-adds the markup and declares it did not comes out "
|
|
f"{_delta} high - exactly one markup, so the defect is DETECTABLE and "
|
|
f"attributable rather than merely wrong",
|
|
isinstance(_delta, float) and abs(_delta - MARKUP) < 1e-9)
|
|
check("...and it is not confusable with 'the taxes were added twice'",
|
|
isinstance(_delta, float) and abs(_delta - TAXES) > 1e-9)
|
|
|
|
# ...and now the structural half: the number is not reachable from a plugin, so
|
|
# the honest author cannot write that line even by accident.
|
|
_po = PluginOptions({"price_markup_eur_kwh": MARKUP,
|
|
"price_taxes_eur_kwh": TAXES,
|
|
"export_price_eur_kwh": 0.04,
|
|
"price_csv_path": "/x"})
|
|
_refused = []
|
|
for k in ("price_markup_eur_kwh", "price_taxes_eur_kwh", "export_price_eur_kwh"):
|
|
try:
|
|
_po.get(k)
|
|
except FrameworkOption:
|
|
_refused.append(k)
|
|
check(f"a plugin cannot read the framework's own options {sorted(_refused)}",
|
|
sorted(_refused) == sorted(["price_markup_eur_kwh", "price_taxes_eur_kwh",
|
|
"export_price_eur_kwh"]))
|
|
check("...and `k in opts` is false for them too, so a probe cannot find them",
|
|
not any(k in _po for k in ("price_markup_eur_kwh", "price_taxes_eur_kwh")))
|
|
check("...while the options a plugin legitimately needs still come through",
|
|
_po.get("price_csv_path") == "/x"
|
|
and _po.get("price_timeout_s") == PRICE_OPTION_DEFAULTS["price_timeout_s"])
|
|
|
|
# The prose and the declaration must agree. A docstring that says "the framework
|
|
# adds taxes" over a class declaring it sources them is a 3 c/kWh error with a
|
|
# reassuring comment on top.
|
|
_doc_drift = []
|
|
for pid, cls in sorted(PROVIDERS.items()):
|
|
line = next((ln for ln in (cls.__doc__ or "").splitlines()
|
|
if ln.strip().startswith("SOURCES:")), "")
|
|
named = {w.strip() for w in line.split(":", 1)[-1].split(",") if w.strip()}
|
|
declared = {c.value for c in cls.capabilities().sources}
|
|
if named != declared:
|
|
_doc_drift.append((pid, sorted(named), sorted(declared)))
|
|
check(f"every plugin's docstring names exactly the components it declares "
|
|
f"({len(PROVIDERS)} checked)", _doc_drift == [])
|
|
|
|
# The ignored-markup warning: a plugin whose feed is already all-in must SAY it
|
|
# is ignoring the operator's numbers rather than dropping them silently.
|
|
_buf = io.StringIO()
|
|
_h = logging.StreamHandler(_buf)
|
|
logging.getLogger("goodwe.price").addHandler(_h)
|
|
load_any("eneco_eddy")
|
|
load_any("generic_dayahead")
|
|
logging.getLogger("goodwe.price").removeHandler(_h)
|
|
_warned = _buf.getvalue()
|
|
check("an all-in plugin logs that the configured markup/taxes are ignored",
|
|
"double-count" in _warned and "eneco_eddy" in _warned)
|
|
check("...and the wholesale plugin, which really does use them, does not",
|
|
"generic_dayahead" not in _warned)
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("4b. a plugin cannot reach the markup BY ANY ROUTE (review of PR #16)")
|
|
# WHY THIS SECTION EXISTS. The first version of this ticket claimed the markup
|
|
# was "not in scope" for a plugin because `PluginOptions` refused it. Review
|
|
# wrote a double-counting plugin three more ways, one of them touching nothing
|
|
# but `PRICE_OPTION_DEFAULTS` - a public dict in the module every plugin must
|
|
# already import for `register`. One closed door in a room with three is not a
|
|
# guarantee. Honest wholesale baseline 0.188; each attack lands 0.219.
|
|
_banned = set(BANNED_IN_PLUGIN_CODE)
|
|
check(f"the banned set covers the framework keys AND the three public routes "
|
|
f"review actually used {sorted(_banned)}",
|
|
_banned == set(_price.FRAMEWORK_KEYS) | {"PRICE_OPTION_DEFAULTS",
|
|
"price_option", "_o"})
|
|
|
|
# code_leaks() fails OPEN when inspect.getsource() cannot find the .py - the
|
|
# right trade for a battery controller, and a silently disarmed guard if nobody
|
|
# checks. THIS is the check that makes CI prove the guard is armed.
|
|
_srclen = derived(lambda: {pid: len(inspect.getsource(cls))
|
|
for pid, cls in PROVIDERS.items()})
|
|
check(f"the scan really read every shipped plugin's source, so its fail-open "
|
|
f"path is not silently in use {_srclen}",
|
|
isinstance(_srclen, dict) and len(_srclen) == 5
|
|
and all(n > 200 for n in _srclen.values()))
|
|
_dirty = derived(lambda: {pid: code_leaks(cls)
|
|
for pid, cls in PROVIDERS.items()
|
|
if code_leaks(cls)})
|
|
check(f"...and all five shipped plugins are clean {_dirty}", _dirty == {})
|
|
|
|
# THE DOCSTRING STRIP IS LOAD-BEARING, AND THIS PROVES IT RATHER THAN ASSERTING
|
|
# IT. Every plugin is REQUIRED to name its sourced components in prose, so
|
|
# engie_dynamic's docstring contains "markup" - a raw text scan false-positives
|
|
# on the documentation the same file mandates, and a check that fires on
|
|
# required documentation is a check somebody deletes.
|
|
_raw_hits = derived(lambda: sorted(
|
|
pid for pid, cls in PROVIDERS.items()
|
|
if any(b in inspect.getsource(cls) for b in _banned)))
|
|
check(f"a RAW text scan would false-positive on the mandated docstrings "
|
|
f"{_raw_hits}, which is why code_leaks() strips them before searching",
|
|
isinstance(_raw_hits, list) and _raw_hits != [] and _dirty == {})
|
|
|
|
|
|
class _AttackPrivateDict(PriceProvider):
|
|
"""A1: reaches into PluginOptions' private dict.
|
|
|
|
SOURCES: energy
|
|
The framework ADDS: markup, taxes
|
|
"""
|
|
|
|
id = "_attack_private_dict"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY}))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
return [RawPoint(T0, 0.10 + o._o["price_markup_eur_kwh"], False)]
|
|
|
|
|
|
class _AttackHelper(PriceProvider):
|
|
"""A2: calls a public framework helper with the raw options dict.
|
|
|
|
SOURCES: energy
|
|
The framework ADDS: markup, taxes
|
|
"""
|
|
|
|
id = "_attack_helper"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY}))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
return [RawPoint(T0, 0.10 + price_option(o._o, "price_markup_eur_kwh"),
|
|
False)]
|
|
|
|
|
|
class _AttackPublicConstant(PriceProvider):
|
|
"""A2b: touches nothing but a public module constant. THE DECIDING ONE.
|
|
|
|
SOURCES: energy
|
|
The framework ADDS: markup, taxes
|
|
"""
|
|
|
|
id = "_attack_public_constant"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY}))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
return [RawPoint(T0,
|
|
0.10 + PRICE_OPTION_DEFAULTS["price_markup_eur_kwh"],
|
|
False)]
|
|
|
|
|
|
_ATTACKS = {"A1 private dict": _AttackPrivateDict,
|
|
"A2 public helper": _AttackHelper,
|
|
"A2b public constant": _AttackPublicConstant}
|
|
for _name, _cls in _ATTACKS.items():
|
|
_found = derived(lambda c=_cls: code_leaks(c))
|
|
check(f"{_name}: the scan names what it reached for {_found}",
|
|
isinstance(_found, list) and _found != [])
|
|
check(f"...and register() REFUSES it at import, not at fetch time",
|
|
raises(lambda c=_cls: register(c), FrameworkOption))
|
|
check(f"...so it never enters the registry", _cls.id not in PROVIDERS)
|
|
|
|
# THE COST OF THE DEFECT, MEASURED, so the refusal is not grading itself.
|
|
_honest_wholesale = derived(lambda: assemble(
|
|
PROVIDERS["generic_dayahead"].capabilities(),
|
|
[RawPoint(T0, 0.10, False)], MARKUP, TAXES)[0].eur_per_kwh)
|
|
_attacked = derived(lambda: assemble(
|
|
_AttackPublicConstant.capabilities(),
|
|
[RawPoint(T0, 0.10 + MARKUP, False)], MARKUP, TAXES)[0].eur_per_kwh)
|
|
check(f"an unrefused attack would bill {_attacked} against an honest "
|
|
f"{_honest_wholesale} - one markup high, every hour, silently",
|
|
isinstance(_attacked, float) and isinstance(_honest_wholesale, float)
|
|
and abs((_attacked - _honest_wholesale) - MARKUP) < 1e-9)
|
|
|
|
# WHAT THIS DOES *NOT* CLOSE, AND CANNOT. A plugin that declares {ENERGY} while
|
|
# its feed is really all-in comes out one markup high while touching no
|
|
# framework name at all. No code can check that: the declaration is the
|
|
# plugin's unverifiable claim about somebody else's tariff. Stated here so the
|
|
# guarantee is read at its true size - the complement is added exactly once,
|
|
# GIVEN A CORRECT DECLARATION - and so a reviewer who trusts an absolute
|
|
# guarantee does not stop checking. Measured, not asserted:
|
|
_misdeclared = derived(lambda: assemble(
|
|
ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY})),
|
|
[RawPoint(T0, 0.10 + MARKUP + TAXES, False)], MARKUP, TAXES)[0].eur_per_kwh)
|
|
check(f"a MIS-DECLARED all-in feed still doubles ({_misdeclared} vs "
|
|
f"{_honest_wholesale}) with no framework name touched - the limit of the "
|
|
f"guarantee, stated rather than hidden",
|
|
isinstance(_misdeclared, float)
|
|
and abs((_misdeclared - _honest_wholesale) - (MARKUP + TAXES)) < 1e-9
|
|
and code_leaks(_AttackPrivateDict) != [])
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("5. product type is declared, not inferred from the name (AC2, trap 2)")
|
|
_products = {pid: cls.capabilities().product for pid, cls in PROVIDERS.items()}
|
|
check(f"eneco_eddy declares itself real-time/imbalance per §9.2 "
|
|
f"({_products['eneco_eddy'].value})",
|
|
_products["eneco_eddy"] is PriceProduct.REALTIME)
|
|
check("...and it is the only shipped one that is",
|
|
[p for p, v in _products.items() if v is PriceProduct.REALTIME]
|
|
== ["eneco_eddy"])
|
|
_regimes = {pid: volatility_regime(v) for pid, v in _products.items()}
|
|
check(f"...so §8.3's widened thresholds reach it through the ENUM {_regimes}",
|
|
_regimes["eneco_eddy"] == "wide"
|
|
and set(v for p, v in _regimes.items() if p != "eneco_eddy") == {"normal"})
|
|
check("the regime travels on the series a consumer is handed",
|
|
derived(lambda: (READS["eneco_eddy"].regime,
|
|
READS["generic_dayahead"].regime)) == ("wide", "normal"))
|
|
def _accepts_name():
|
|
"""Would volatility_regime() take a plugin id? If it would, somebody can
|
|
start passing one and AC2 rots without a single check going red."""
|
|
try:
|
|
volatility_regime("eneco_eddy")
|
|
return True
|
|
except TypeError:
|
|
return False
|
|
|
|
|
|
check("volatility_regime refuses a plugin NAME, which is the AC2 violation "
|
|
"written out",
|
|
not _accepts_name())
|
|
|
|
|
|
# ⚠️ THE MECHANICAL FORM OF AC2. The framework's own code must contain no
|
|
# plugin id at all - if it does, the sixth supplier is an edit to price.py's
|
|
# shared machinery rather than a new class. Derived: build the set of ids that
|
|
# actually occur, and assert it is empty.
|
|
_framework = [_price.assemble, _price.load_prices, _price.volatility_regime,
|
|
_price.arbitrage_allowed, _price.plan_is_provisional,
|
|
_price.check_egress, _price._validate_raw, _price.PriceRead,
|
|
_price._Series, _price._NoData, _price.PluginOptions,
|
|
_price.register, _price.price_option]
|
|
_src = "\n".join(inspect.getsource(o) for o in _framework)
|
|
_leaked = sorted(pid for pid in PROVIDERS if pid in _src)
|
|
check(f"no plugin id appears anywhere in the framework's code {_leaked or '[]'}",
|
|
_leaked == [])
|
|
|
|
# ...and the positive half: a plugin defined HERE, in a file price.py has never
|
|
# heard of, gets the right regime with no edit anywhere.
|
|
@register
|
|
class _NewSupplier(PriceProvider):
|
|
"""A sixth supplier, invented in the test file. AC2 in one class.
|
|
|
|
SOURCES: energy
|
|
The framework ADDS: markup, taxes
|
|
"""
|
|
|
|
id = "_test_sixth_supplier"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.REALTIME,
|
|
sources=frozenset({Component.ENERGY}),
|
|
egress_hosts=("api.example.invalid",))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
body = await transport("https://api.example.invalid/p",
|
|
float(o.get("price_timeout_s")))
|
|
return _price._points_from_json(body, "price_eur_mwh", 0.001, False)
|
|
|
|
|
|
_sixth = load("_test_sixth_supplier",
|
|
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH)))
|
|
check("a plugin invented outside price.py resolves through the registry",
|
|
_sixth.kind == "series" and _sixth.provider == "_test_sixth_supplier")
|
|
_sixth_regime = derived(lambda: _sixth.regime)
|
|
check(f"...and lands in the wide-threshold regime purely from its declaration "
|
|
f"({_sixth_regime})", _sixth_regime == "wide")
|
|
check("...and the framework applied its markup and taxes correctly too",
|
|
derived(lambda: [p.eur_per_kwh for p in _sixth.points])
|
|
== [round(v / 1000 + MARKUP + TAXES, 6) for v in EPEX_EUR_MWH])
|
|
del PROVIDERS["_test_sixth_supplier"] # do not leak into later sections
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("6. no price data is its own outcome, not an empty list (trap 3, §11.4)")
|
|
|
|
_none = PriceRead.no_data("the feed timed out")
|
|
# ⚠️ THE DISTINCTION THE WHOLE TYPE EXISTS FOR. A day of free electricity and a
|
|
# day with no price data are different instructions: arbitrage is POINTLESS
|
|
# under the first and FORBIDDEN under the second.
|
|
_zero = PriceRead.series([PricePoint(brussels(T0 + timedelta(hours=i)), 0.0,
|
|
False) for i in range(3)],
|
|
PriceProduct.DAY_AHEAD, "x", T0)
|
|
check("a zero-price series and no-price-data are different kinds",
|
|
_zero.kind != _none.kind)
|
|
check("...arbitrage is allowed under zero prices and forbidden under no data",
|
|
arbitrage_allowed(_zero) is True and arbitrage_allowed(_none) is False)
|
|
check("...the plan is provisional under no data and settled under zero prices",
|
|
plan_is_provisional(_none) is True and plan_is_provisional(_zero) is False)
|
|
check("...and the cheapest hour is computable for the zero series",
|
|
derived(lambda: min(p.eur_per_kwh for p in _zero.points)) == 0.0)
|
|
|
|
|
|
check("no_data has no .points, so `min(read.points)` cannot invent a free hour",
|
|
raises(lambda: _none.points, Absence))
|
|
check("...no .product to switch a threshold regime on",
|
|
raises(lambda: _none.product, Absence))
|
|
check("...no .at(), so an hourly lookup cannot silently succeed",
|
|
raises(lambda: _none.at(T0), Absence))
|
|
check("...and .get('points', []) - the idiom the whole rework exists to kill - "
|
|
"raises rather than returning the default",
|
|
raises(lambda: _none.get("points", []), Absence))
|
|
check("...and it is not iterable, so `for p in read:` cannot be an empty loop",
|
|
raises(lambda: list(_none), Absence))
|
|
check("...while its one real field, the reason, is there for an operator",
|
|
_none.reason == "the feed timed out")
|
|
|
|
# M3, INHERITED FROM RIG-01 AND CLOSED HERE. `_NoData.__init__(self,
|
|
# reason="no price data")` used to go green: a required field with a DEFAULT is
|
|
# the exact idiom this whole type exists to forbid, and it survived because
|
|
# every check passed a reason explicitly. It is the identical survivor left
|
|
# open on sim/riglock.py's `_Unread.cause` - the shape was borrowed including
|
|
# its blind spot, which is what borrowing a shape does if nobody re-runs the
|
|
# mutations against the copy.
|
|
#
|
|
# Checked by SIGNATURE rather than by one call, so any default added to any of
|
|
# these constructors later is red without somebody remembering to write a new
|
|
# case.
|
|
def _defaulted(fn):
|
|
"""Parameters of `fn` that carry a default. `self` is skipped."""
|
|
try:
|
|
ps = list(inspect.signature(fn).parameters.values())
|
|
except (TypeError, ValueError) as e: # pragma: no cover
|
|
return _Raised(e)
|
|
return [q.name for q in ps
|
|
if q.name != "self" and q.default is not inspect.Parameter.empty]
|
|
|
|
|
|
_ctors = {"PriceRead.no_data": PriceRead.no_data,
|
|
"PriceRead.series": PriceRead.series,
|
|
"_NoData.__init__": _price._NoData.__init__,
|
|
"_Series.__init__": _price._Series.__init__,
|
|
"PriceRead.match": PriceRead.match}
|
|
_with_defaults = derived(lambda: {n: d for n, d in
|
|
((n, _defaulted(f)) for n, f in _ctors.items())
|
|
if d})
|
|
check(f"no constructor or handler of an outcome type carries a DEFAULT, so a "
|
|
f"reading can never be built out of nothing {_with_defaults}",
|
|
_with_defaults == {})
|
|
check("...so no_data() with no reason at all is a TypeError at the call site",
|
|
raises(lambda: PriceRead.no_data(), TypeError))
|
|
check("...and the variant class itself cannot be built bare either",
|
|
raises(lambda: _price._NoData(), TypeError))
|
|
check("...nor can a series be built without its product or provenance",
|
|
raises(lambda: _price._Series([PricePoint(brussels(T0), 0.2, False)]),
|
|
TypeError))
|
|
# The positive arm, so the four above are not passing because everything raises.
|
|
check("...while a fully specified no_data is still perfectly constructible",
|
|
PriceRead.no_data("the feed timed out").reason == "the feed timed out")
|
|
|
|
# `match` must also refuse POSITIONAL handlers - keyword-only is what makes a
|
|
# forgotten arm a TypeError rather than a silently swapped one.
|
|
check("match() refuses positional handlers, so the two arms cannot be swapped",
|
|
raises(lambda: _none.match(lambda r: 1, lambda r: 2), TypeError))
|
|
|
|
check("an EMPTY series cannot be constructed at all, so 'no prices' can never "
|
|
"wear the shape of an answer",
|
|
raises(lambda: PriceRead.series([], PriceProduct.DAY_AHEAD, "x", T0),
|
|
ValueError))
|
|
check("a series built with a plugin NAME instead of a product enum is refused",
|
|
raises(lambda: PriceRead.series([PricePoint(brussels(T0), 0.2, False)],
|
|
"eneco_eddy", "x", T0), ValueError))
|
|
check("no_data with a blank reason is refused - the reason is its only field",
|
|
raises(lambda: PriceRead.no_data(""), ValueError))
|
|
|
|
|
|
def _forgot():
|
|
return _none.match(series=lambda r: r.points) # no_data omitted
|
|
|
|
|
|
check("a caller who forgets the no_data arm gets a TypeError at the call site, "
|
|
"not a fall-through", raises(_forgot, TypeError))
|
|
|
|
# .at() outside the series raises rather than returning a neighbour or None.
|
|
_s = READS["generic_dayahead"]
|
|
check("series.at() inside the series returns the covering point",
|
|
derived(lambda: _s.at(_s.points[0].start + timedelta(minutes=59))
|
|
is _s.points[0]) is True)
|
|
check("...and raises outside it rather than returning None to be `or`-ed away",
|
|
derived(lambda: (
|
|
raises(lambda: _s.at(_s.points[-1].start + timedelta(days=2)),
|
|
Absence)
|
|
and _s.covers(_s.points[0].start)
|
|
and not _s.covers(_s.points[-1].start + timedelta(days=2)))) is True)
|
|
|
|
# Every failure a site can have must come back as no_data, never as a traceback
|
|
# into the tick. Derived: drive each failure and collect the kinds.
|
|
async def _boom(url, timeout_s):
|
|
raise OSError("connection reset by peer")
|
|
|
|
|
|
_failures = {
|
|
"the source is unreachable": load("generic_dayahead", transport=_boom),
|
|
"the body is not JSON": load("generic_dayahead", transport=canned(b"<html>")),
|
|
"the feed is empty": load("generic_dayahead",
|
|
transport=canned(columnar_feed([]))),
|
|
# ⚠️ THE ROW SHAPE IS A FAILURE FOR THIS PLUGIN, NOT A SECOND DIALECT. If
|
|
# `_points_from_columnar_json()` ever grows a sniff for `data`, this case
|
|
# goes from no_data to a series and TARIFF-08's scope rule is broken by a
|
|
# check rather than by a review.
|
|
"the body is the OTHER shape": load(
|
|
"generic_dayahead",
|
|
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))),
|
|
"the provider id is unknown": load("no_such_supplier"),
|
|
"no provider is configured": load("off"),
|
|
"the csv is missing": load("manual_csv",
|
|
{"price_csv_path": "/no/such/file.csv"},
|
|
transport=refuse_transport),
|
|
}
|
|
_not_nodata = sorted(k for k, r in _failures.items() if r.kind != "no_data")
|
|
check(f"every site-level failure returns no_data instead of raising "
|
|
f"({len(_failures)} cases)", _not_nodata == [])
|
|
_no_reason = derived(lambda: sorted(k for k, r in _failures.items()
|
|
if len(r.reason) < 10))
|
|
check(f"...each with a reason an operator can act on {_no_reason}",
|
|
_no_reason == [])
|
|
check("...and none of them is mistakable for a series",
|
|
derived(lambda: all(not arbitrage_allowed(r)
|
|
for r in _failures.values())) is True)
|
|
|
|
# ⚠️ The one thing that must NOT degrade to no_data: a plugin reaching for the
|
|
# markup. That is a code defect, and swallowing it would hide exactly what
|
|
# trap 1 exists to catch.
|
|
class _Thief(PriceProvider):
|
|
id = "_thief"
|
|
|
|
@classmethod
|
|
def capabilities(cls):
|
|
return ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY}))
|
|
|
|
async def fetch(self, o, since, until, transport):
|
|
return [RawPoint(T0, 0.1 + o.get("price_markup_eur_kwh"), False)]
|
|
|
|
|
|
PROVIDERS["_thief"] = _Thief
|
|
check("a plugin that reaches for the markup raises rather than degrading to "
|
|
"no_data - that is a bug, not a site condition",
|
|
raises(lambda: load("_thief"), FrameworkOption))
|
|
del PROVIDERS["_thief"]
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("7. manual_csv works fully offline (AC3)")
|
|
_had_aiohttp = "aiohttp" in sys.modules
|
|
_csv_read = load("manual_csv", {"price_csv_path": CSV_PATH},
|
|
transport=refuse_transport)
|
|
_csv_prices = derived(lambda: [p.eur_per_kwh for p in _csv_read.points])
|
|
check("the offline plugin produced a series without touching the transport - "
|
|
"the transport raises if called at all",
|
|
_csv_read.kind == "series" and _csv_prices == [0.1834, -0.021, 0.3115])
|
|
check(f"...it is all-in already, so the framework added nothing {_csv_prices}",
|
|
_csv_prices == [round(v, 6) for v in
|
|
[float(ln.split(",")[1]) for ln in
|
|
CSV_BODY.strip().splitlines()[1:]]])
|
|
check("...including the negative hour",
|
|
isinstance(_csv_prices, list) and any(v < 0 for v in _csv_prices))
|
|
check("...and the per-point provisional flag came from the file's third column",
|
|
derived(lambda: [p.provisional for p in _csv_read.points])
|
|
== [ln.split(",")[2] == "true"
|
|
for ln in CSV_BODY.strip().splitlines()[1:]])
|
|
check("...and no HTTP stack was imported to do it"
|
|
if not _had_aiohttp else
|
|
"...(aiohttp was already imported before this file ran; not a finding)",
|
|
_had_aiohttp or "aiohttp" not in sys.modules)
|
|
check("manual_csv declares no egress hosts at all",
|
|
PROVIDERS["manual_csv"].capabilities().egress_hosts == ())
|
|
|
|
_ManualCsv = PROVIDERS["manual_csv"]
|
|
check("a CSV with no header is refused rather than parsed as data",
|
|
raises(lambda: _ManualCsv.parse("2026-01-15T00:00:00+01:00,0.18\n"),
|
|
SourceUnavailable))
|
|
check("a CSV start with no offset is refused, naming the fix",
|
|
raises(lambda: _ManualCsv.parse("start,eur_per_kwh\n2026-01-15 00:00,0.18\n"),
|
|
SourceUnavailable))
|
|
check("...and the two-column form (no provisional) is accepted, defaulting to "
|
|
"settled",
|
|
[p.provisional for p in
|
|
_ManualCsv.parse("start,eur_per_kwh\n2026-01-15T00:00:00+01:00,0.18\n")]
|
|
== [False])
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("8. the two 2026 DST transitions, where a 24-slot array is wrong")
|
|
# ⚠️ RUN ON THE 23- AND 25-HOUR DAYS ON PURPOSE. On any normal day a fixed
|
|
# 24-slot implementation, a naive-local-hour key and a correct one all agree
|
|
# exactly, so a check on 15 January grades nothing about DST.
|
|
check("the 2026 transitions are where this repo expects them (last Sundays)",
|
|
_price._last_sunday(2026, 3) == 29 and _price._last_sunday(2026, 10) == 25)
|
|
|
|
|
|
def local_day(day_start_utc, n):
|
|
return [brussels(day_start_utc + timedelta(hours=i)) for i in range(n)]
|
|
|
|
|
|
# Spring forward: local day 2026-03-29 runs 2026-03-28T23:00Z .. 2026-03-29T22:00Z
|
|
_spring = local_day(datetime(2026, 3, 28, 23, tzinfo=UTC), 23)
|
|
_spring_end = brussels(datetime(2026, 3, 29, 22, tzinfo=UTC))
|
|
check(f"the spring day has 23 hours, ending exactly at the next midnight "
|
|
f"({_spring[0].isoformat()} .. {_spring_end.isoformat()})",
|
|
_spring[0].hour == 0 and _spring[0].day == 29
|
|
and _spring_end.hour == 0 and _spring_end.day == 30)
|
|
check(f"...crossing exactly one offset change "
|
|
f"{sorted(str(o) for o in {p.utcoffset() for p in _spring})}",
|
|
{p.utcoffset() for p in _spring} == {timedelta(hours=1),
|
|
timedelta(hours=2)})
|
|
check("...and 02:00 local never happens that day, which a 24-slot array would "
|
|
"have invented", 2 not in {p.hour for p in _spring})
|
|
|
|
# Fall back: local day 2026-10-25 runs 2026-10-24T22:00Z .. 2026-10-25T22:00Z
|
|
_fall = local_day(datetime(2026, 10, 24, 22, tzinfo=UTC), 25)
|
|
_fall_end = brussels(datetime(2026, 10, 25, 23, tzinfo=UTC))
|
|
check(f"the autumn day has 25 hours ({_fall[0].isoformat()} .. "
|
|
f"{_fall_end.isoformat()})",
|
|
_fall[0].hour == 0 and _fall[0].day == 25
|
|
and _fall_end.hour == 0 and _fall_end.day == 26)
|
|
_wall = [p.replace(tzinfo=None) for p in _fall]
|
|
_dups = len(_wall) - len(set(_wall))
|
|
check(f"...and exactly one local wall-clock hour occurs TWICE ({_dups}), which "
|
|
f"a naive-local-hour key would collide into one mispriced hour",
|
|
_dups == 1)
|
|
check("...distinguished by their offsets, which is why the canonical format "
|
|
"carries one",
|
|
len({p.isoformat() for p in _fall}) == 25)
|
|
# A normal day, stated as the control: without it, "23" and "25" prove nothing
|
|
# about whether the code counts at all.
|
|
_normal = local_day(datetime(2026, 1, 14, 23, tzinfo=UTC), 24)
|
|
check("...while an ordinary January day has 24 hours and one offset",
|
|
len({p.replace(tzinfo=None) for p in _normal}) == 24
|
|
and len({p.utcoffset() for p in _normal}) == 1)
|
|
check("brussels() refuses a naive datetime rather than assuming an offset",
|
|
raises(lambda: brussels(datetime(2026, 1, 15, 0, 0)), ValueError))
|
|
check("the offset boundary is 01:00 UTC, not local midnight",
|
|
brussels_offset(datetime(2026, 3, 29, 0, 59, tzinfo=UTC)) ==
|
|
timezone(timedelta(hours=1))
|
|
and brussels_offset(datetime(2026, 3, 29, 1, 0, tzinfo=UTC)) ==
|
|
timezone(timedelta(hours=2)))
|
|
|
|
# A feed that publishes the duplicated hour must survive it. The 25-hour day is
|
|
# 25 strictly increasing instants; a plugin keyed on local hours would emit 24
|
|
# and be rejected by the boundary validator instead of silently mispricing.
|
|
_dst_feed = columnar_feed([50.0 + i for i in range(25)],
|
|
start=datetime(2026, 10, 24, 22, tzinfo=UTC))
|
|
_dst_read = load("generic_dayahead", transport=canned(_dst_feed))
|
|
_dst_n = derived(lambda: len(_dst_read.points))
|
|
check(f"a 25-point autumn feed is accepted whole ({_dst_n} points)",
|
|
_dst_read.kind == "series" and _dst_n == 25)
|
|
check("...and its canonical starts are 25 distinct strings",
|
|
derived(lambda: len({p["start"]
|
|
for p in _dst_read.canonical()["prices"]})) == 25)
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("9. the boundary rejects what cannot be a price, and only that")
|
|
# M5. Deleting the `Component.ENERGY not in sources` guard used to go green,
|
|
# and the consequence is not cosmetic: `adds` becomes {ENERGY, TAXES},
|
|
# `assemble()` never adds an energy component it thinks it is adding, and the
|
|
# emitted all-in price silently omits the energy it is supposed to be pricing.
|
|
check("capabilities() refuses a feed that declares no ENERGY component",
|
|
raises(lambda: ProviderCapabilities(
|
|
product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.MARKUP})), ValueError))
|
|
check("...including a completely empty sources set",
|
|
raises(lambda: ProviderCapabilities(product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset()), ValueError))
|
|
check("...and an unknown component is refused rather than ignored",
|
|
raises(lambda: ProviderCapabilities(
|
|
product=PriceProduct.DAY_AHEAD,
|
|
sources=frozenset({Component.ENERGY, "grid_fees"})), ValueError))
|
|
check("...and a product given as a string is refused at the declaration too",
|
|
raises(lambda: ProviderCapabilities(product="dynamic_dayahead",
|
|
sources=frozenset({Component.ENERGY})),
|
|
ValueError))
|
|
# THE CONSEQUENCE, derived across every shipped plugin rather than asserted on
|
|
# one: the framework must never be in a position to "add" the energy component,
|
|
# because there is nothing to add it from.
|
|
_adds_energy = derived(lambda: sorted(
|
|
pid for pid, cls in PROVIDERS.items()
|
|
if Component.ENERGY in cls.capabilities().adds))
|
|
check(f"so no shipped plugin ever asks the framework to invent an energy "
|
|
f"price {_adds_energy}", _adds_energy == [])
|
|
|
|
_caps = PROVIDERS["generic_dayahead"].capabilities()
|
|
|
|
|
|
def boundary(points):
|
|
return raises(lambda: _price._validate_raw(points, _caps), SourceUnavailable)
|
|
|
|
|
|
check("NaN is refused", boundary([RawPoint(T0, float("nan"), False)]))
|
|
check("infinity is refused", boundary([RawPoint(T0, float("inf"), False)]))
|
|
check(f"a EUR/MWh figure read as EUR/kWh is refused (>{MAX_PLAUSIBLE_EUR_KWH})",
|
|
boundary([RawPoint(T0, 61.4, False)]))
|
|
check("a naive start is refused",
|
|
boundary([RawPoint(datetime(2026, 1, 15), 0.1, False)]))
|
|
check("duplicate starts are refused - the DST collision, caught",
|
|
boundary([RawPoint(T0, 0.1, False), RawPoint(T0, 0.2, False)]))
|
|
check("out-of-order starts are refused",
|
|
boundary([RawPoint(T0 + timedelta(hours=1), 0.1, False),
|
|
RawPoint(T0, 0.2, False)]))
|
|
check("a plugin returning something that is not a RawPoint is refused",
|
|
boundary([PricePoint(brussels(T0), 0.1, False)]))
|
|
check("an empty list is refused", boundary([]))
|
|
# ...and the negative-price control, so the rules above are not just "reject
|
|
# everything". A validator that refused this would cost money on a windy Sunday.
|
|
check("a NEGATIVE price at a plausible magnitude is accepted",
|
|
not boundary([RawPoint(T0, -0.35, False)]))
|
|
check("...and so is a large-but-plausible one at the limit",
|
|
not boundary([RawPoint(T0, MAX_PLAUSIBLE_EUR_KWH - 0.01, False)]))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("10. egress is declared by the plugin and enforced (§16)")
|
|
_g = PROVIDERS["generic_dayahead"].capabilities()
|
|
check("a declared https host is allowed",
|
|
check_egress("https://api.energy-charts.info/price?bzn=BE", _g)
|
|
== "api.energy-charts.info")
|
|
check("an undeclared host is refused even over TLS",
|
|
raises(lambda: check_egress("https://evil.example/p", _g),
|
|
SourceUnavailable))
|
|
# M7, AND THE ASSERTION IS THE FIX, NOT ONLY THE PARSER. Deleting the TLS rule
|
|
# used to leave this check GREEN: the old parser stripped a fixed
|
|
# len("https://") from the front, so `http://api.energy-charts.info/p` became
|
|
# the host `pi.energy-charts.info`, which failed the ALLOW-LIST instead. The
|
|
# feed was refused, the check named "plain http is refused" passed, and the
|
|
# rule it was named for could be deleted with no consequence - an assertion
|
|
# that does not test what it names, inside a security check.
|
|
#
|
|
# Three checks now, and the first two are what make the third able to fail:
|
|
# prove the host is declared over https, prove it parses identically over
|
|
# http, and only then assert that http is refused AND that the reason is TLS.
|
|
_tls_url = "http://api.energy-charts.info/p"
|
|
check("the same host over https is allowed, so it IS on the allow-list",
|
|
derived(lambda: check_egress("https://api.energy-charts.info/p", _g))
|
|
== "api.energy-charts.info")
|
|
check("...and the http form parses to that same declared host, so nothing but "
|
|
"the TLS rule can refuse it",
|
|
derived(lambda: _tls_url.partition("://")[2].split("/", 1)[0])
|
|
== "api.energy-charts.info")
|
|
_tls_reason = derived(lambda: _reason(lambda: check_egress(_tls_url, _g)))
|
|
check(f"plain http is refused, and refused BY THE TLS RULE - {_tls_reason!r}",
|
|
isinstance(_tls_reason, str) and "non-TLS" in _tls_reason)
|
|
check("a scheme that is neither is refused the same way, by TLS not by host",
|
|
"non-TLS" in str(derived(lambda: _reason(
|
|
lambda: check_egress("ftp://api.energy-charts.info/p", _g)))))
|
|
# ⚠️ M7b, WHICH I FOUND IN MY OWN FIX FOR M7 AND WHICH IS THE SAME DEFECT ONE
|
|
# LAYER DOWN. This check used to read `raises(..., SourceUnavailable)` and
|
|
# survived deleting the `if not sep` guard: with no guard,
|
|
# "api.energy-charts.info/p".partition("://") leaves scheme = the WHOLE string,
|
|
# so the TLS rule refuses it and the check named for the scheme guard passes.
|
|
# Refused either way - but the diagnostic an operator gets degrades from "not an
|
|
# absolute URL" to "uses 'api.energy-charts.info/p'" as a scheme, which is
|
|
# gibberish. Assert the reason, like M7 above.
|
|
_noscheme = derived(lambda: _reason(
|
|
lambda: check_egress("api.energy-charts.info/p", _g)))
|
|
check(f"a URL with no scheme is refused BY THE SCHEME GUARD, with a diagnostic "
|
|
f"that names the real problem - {_noscheme!r}",
|
|
isinstance(_noscheme, str) and "not an absolute URL" in _noscheme)
|
|
check("a userinfo prefix cannot smuggle a declared host past the check",
|
|
raises(lambda: check_egress(
|
|
"https://api.energy-charts.info@evil.example/p", _g),
|
|
SourceUnavailable))
|
|
check("an operator-supplied URL to another supplier's host is refused",
|
|
load("generic_dayahead",
|
|
{"price_feed_url": "https://api.eneco.be/eddy/prices"},
|
|
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))
|
|
).kind == "no_data")
|
|
_declared = {pid: cls.capabilities().egress_hosts
|
|
for pid, cls in PROVIDERS.items()}
|
|
check(f"every networked plugin declares at least one host, and manual_csv "
|
|
f"declares none {_declared}",
|
|
all(bool(h) for p, h in _declared.items() if p != "manual_csv")
|
|
and _declared["manual_csv"] == ())
|
|
check("the timeout the plugin passes to the transport is the configured one, "
|
|
"not a hardcoded default",
|
|
(lambda t: (load("generic_dayahead", {"price_timeout_s": 7},
|
|
transport=t), t.calls[0][1] == 7.0)[1])(
|
|
canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("11. injection price is a separate scalar (AC4, §9.3)")
|
|
check(f"the shipped default is config.yaml's "
|
|
f"{PRICE_OPTION_DEFAULTS['export_price_eur_kwh']}",
|
|
export_price_eur_kwh({}) == PRICE_OPTION_DEFAULTS["export_price_eur_kwh"])
|
|
check("an operator's value wins over it",
|
|
export_price_eur_kwh({"export_price_eur_kwh": 0.0}) == 0.0)
|
|
check("...including a negative one - Belgian injection has gone below zero",
|
|
export_price_eur_kwh({"export_price_eur_kwh": -0.02}) == -0.02)
|
|
check("it is NOT folded into the consumption series any plugin emits",
|
|
derived(lambda: all(export_price_eur_kwh({}) not in
|
|
[p.eur_per_kwh for p in r.points]
|
|
for r in READS.values())) is True)
|
|
check("...and no plugin can reach it to fold it in",
|
|
raises(lambda: PluginOptions({}).get("export_price_eur_kwh"),
|
|
FrameworkOption))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
print("12. the startup gate (validate.py rule 7)")
|
|
|
|
|
|
def vbase(**over):
|
|
o = {"setpoint_entity": "number.goodwe_setpoint", "max_w": 2000,
|
|
"soc_entity": "sensor.soc", "batt_entity": "sensor.batt",
|
|
"meter_source": "off",
|
|
"maintenance_charge_w": 2000, "maintenance_discharge_w": 2000}
|
|
o.update(over)
|
|
return o
|
|
|
|
|
|
def why(problems, needle):
|
|
return any(needle in p for p in problems)
|
|
|
|
|
|
check("the shipped price defaults start (price_provider is 'off')",
|
|
validate_startup_config(vbase(**{k: v for k, v in
|
|
PRICE_OPTION_DEFAULTS.items()}),
|
|
-5000, 5000) == [])
|
|
check("...and so does a configured provider on its own",
|
|
validate_startup_config(vbase(price_provider="generic_dayahead"),
|
|
-5000, 5000) == [])
|
|
check("an unknown provider id is refused rather than defaulted to 'off'",
|
|
why(validate_startup_config(vbase(price_provider="nordpool"), -5000, 5000),
|
|
"not a shipped plugin"))
|
|
# ⚠️ THE HA OVERLAP, DECIDED. The two entities keep working alone; combining
|
|
# them with a provider is what is refused, because the entities win silently.
|
|
_p = validate_startup_config(
|
|
vbase(price_provider="engie_dynamic", price_now_entity="sensor.now",
|
|
price_avg_entity="sensor.avg"), -5000, 5000)
|
|
check("a provider AND the Home Assistant price entities is refused - two price "
|
|
"sources on one site", len(_p) == 1)
|
|
check("...and the message says which one actually wins",
|
|
why(_p, "HOME ASSISTANT ENTITIES WIN"))
|
|
# TARIFF-03 added a second source with the same silent-loser shape.
|
|
_ps = validate_startup_config(
|
|
vbase(static_profile='{"periods": [{"from":"00:00","to":"24:00",'
|
|
'"days":"all","eur_per_kwh":0.31}]}',
|
|
price_now_entity="sensor.now", price_avg_entity="sensor.avg"),
|
|
-5000, 5000)
|
|
check("a static_profile AND the Home Assistant price entities is refused too - "
|
|
"main.py reads one of them for `cheap` and the loser is silent",
|
|
len(_ps) == 1 and why(_ps, "HOME ASSISTANT"))
|
|
check("the Home Assistant entities alone still start, so nothing is taken away "
|
|
"from a live install",
|
|
validate_startup_config(vbase(price_now_entity="sensor.now",
|
|
price_avg_entity="sensor.avg"),
|
|
-5000, 5000) == [])
|
|
# ⚠️ THIS TRIPWIRE FIRED, AND THIS IS WHAT IT LOOKS LIKE AFTERWARDS.
|
|
#
|
|
# TARIFF-02 left a check here asserting that NOTHING in app/ consumed
|
|
# `load_prices()`, because rule 7's stated reason - "the supplier you configured
|
|
# would decide nothing" - was true only for as long as that held. The
|
|
# instruction attached to it was: when this goes red, update the message in
|
|
# validate.py in the same commit, do not delete the check.
|
|
#
|
|
# TARIFF-03 landed `app/static_tariff.py`, whose `resolve_tariff()` consumes
|
|
# `load_prices()` and turns a series into a tariff plan, a cheap-window flag and
|
|
# an arbitrage verdict. The check went red, naming that file, and validate.py's
|
|
# message was rewritten in the same commit to say what is true now instead. So
|
|
# the tripwire is KEPT and RE-AIMED rather than removed, at the next thing that
|
|
# can go quietly wrong:
|
|
#
|
|
# * the consumer set is PINNED. A second consumer appearing - OPT-04 wiring a
|
|
# strategy in, most likely - goes red again, and whoever does that has to
|
|
# re-read rule 7's message against what their change makes true.
|
|
# * the EXPIRED PHRASE is asserted GONE. If the old wording is ever restored
|
|
# by a bad merge, this says so.
|
|
# * and the message has to keep naming its consumer, so it cannot decay into
|
|
# a refusal with no stated reason at all.
|
|
_APP = os.path.join(os.path.dirname(os.path.abspath(__file__)), "app")
|
|
# ⚠️ AST, NOT A SUBSTRING, AND THE FIRST DRAFT PROVED WHY. A text scan for
|
|
# "load_prices" matched the COMMENT in validate.py that explains this very
|
|
# tripwire, so the check went red against its own documentation - the same
|
|
# false-positive class as the plugin docstrings in section 4b, hit twice in one
|
|
# ticket. Comments and docstrings are not in an AST, so a name/import walk sees
|
|
# code and only code.
|
|
def _imports_or_calls(path, name):
|
|
try:
|
|
tree = ast.parse(open(path, encoding="utf-8").read())
|
|
except (OSError, SyntaxError):
|
|
return False
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.ImportFrom) and any(a.name == name
|
|
for a in node.names):
|
|
return True
|
|
if isinstance(node, ast.Name) and node.id == name:
|
|
return True
|
|
if isinstance(node, ast.Attribute) and node.attr == name:
|
|
return True
|
|
return False
|
|
|
|
|
|
_consumers = derived(lambda: sorted(
|
|
f for f in os.listdir(_APP)
|
|
if f.endswith(".py") and f != "price.py"
|
|
and _imports_or_calls(os.path.join(_APP, f), "load_prices")))
|
|
_EXPECTED_CONSUMERS = ["static_tariff.py"]
|
|
check(f"load_prices() is consumed by exactly the modules rule 7's message "
|
|
f"names {_consumers}. When a SECOND one appears this goes red, and the "
|
|
f"fix is to re-read that message against what the new consumer makes "
|
|
f"true - not to widen this list without doing so",
|
|
_consumers == _EXPECTED_CONSUMERS)
|
|
_RULE7 = derived(lambda: open(os.path.join(_APP, "validate.py"),
|
|
encoding="utf-8").read())
|
|
check("...and the phrase the tripwire expired - 'would decide nothing' about "
|
|
"the provider - is GONE from validate.py, so a bad merge that restores "
|
|
"the old reason is red rather than quiet",
|
|
isinstance(_RULE7, str)
|
|
and "supplier you configured would decide nothing" not in _RULE7)
|
|
check("...and the refusal still names the consumer that exists, so it cannot "
|
|
"decay into a rule with no stated reason",
|
|
isinstance(_RULE7, str) and "static_tariff.py" in _RULE7)
|
|
# ...and the other half: the registry IS on the startup path, which is what
|
|
# makes an unwired seam anchored rather than dead code.
|
|
check("...while validate.py does import the registry, so the seam is exercised "
|
|
"on every boot",
|
|
derived(lambda: "PROVIDERS" in open(
|
|
os.path.join(_APP, "validate.py"), encoding="utf-8").read()) is True)
|
|
|
|
check("manual_csv with no path is refused - the offline fallback with no file "
|
|
"is §11.4 reached by accident",
|
|
why(validate_startup_config(vbase(price_provider="manual_csv",
|
|
price_csv_path=""), -5000, 5000),
|
|
"price_csv_path"))
|
|
check("...and with a path it starts",
|
|
validate_startup_config(vbase(price_provider="manual_csv",
|
|
price_csv_path=CSV_PATH),
|
|
-5000, 5000) == [])
|
|
# ⚠️ price_provider 'off' must NEVER become a refusal: §9.2 puts a large share
|
|
# of the installed base on a fixed or static tariff, and §11.4 makes no-price
|
|
# data a supported state.
|
|
check("'off' is not refused, on any of the other configurations",
|
|
all(validate_startup_config(vbase(price_provider="off", **extra),
|
|
-5000, 5000) == []
|
|
for extra in ({}, {"price_now_entity": "sensor.now",
|
|
"price_avg_entity": "sensor.avg"},
|
|
{"price_csv_path": ""})))
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
print()
|
|
if fails:
|
|
print(f"{len(fails)} FAILED: " + "; ".join(fails))
|
|
sys.exit(1)
|
|
print("all checks passed")
|