Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
04292c4a08 | ||
|
|
86c43eba24 | ||
|
|
d57122ec74 | ||
|
|
4601682815 | ||
|
|
9234016419 | ||
|
|
b7272169b1 | ||
|
|
2b8b8cb97f | ||
|
|
8e589fee3f | ||
|
|
bdd7f18a54 | ||
|
|
a1376e44e4 | ||
|
|
e709554857 | ||
|
|
a30bde1ab1 |
@@ -2,6 +2,214 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
**Thirty days of the real meter, and the acceptance criterion three cards
|
||||
deferred is now a number (DATA-01).** TARIFF-01 and SAFETY-07 both merged with
|
||||
AC 3 - *"monthly peak held under the soft limit in >= 95 % of intervals over a
|
||||
30-day simulation replay"* - honestly unmet, on the belief that only two days of
|
||||
capture existed and nothing longer could be produced. That belief came from a
|
||||
true statement about the past being read as one about the future: HA's recorder
|
||||
still held the whole month. `scenarios/p1-net-30d-2026-08/` is 2026-08-01 to
|
||||
2026-08-30, **485,841 samples, 30/30 days read, 0 failed, 4.13 MB gzipped**, in
|
||||
the same schema as the two days that were already there.
|
||||
|
||||
`tools/export_p1_history.py` is the exporter, committed and runnable - the
|
||||
2026-08-25 one-off was not, and that is exactly why the data was thought
|
||||
unobtainable. It reads day-at-a-time, retries, and **never lets a failed read
|
||||
look like an empty hour**: a day whose read fails writes no file, is recorded as
|
||||
`"read": "FAILED"`, and the run exits non-zero; the entity is proved to exist
|
||||
before any empty day is believed. Nothing is interpolated, so the month's 13
|
||||
gaps (1291 s, 99.95 % covered) and its 3 `unavailable` samples stay visible.
|
||||
`--verify` re-fetches the two committed days by the same route and reproduces
|
||||
them **byte for byte** after decompression.
|
||||
|
||||
`tools/replay_ac3.py` runs the month. Over 30 days the criterion measures
|
||||
**98.89 % uncontrolled and 98.99 % guarded**, so it is met on that reading - and
|
||||
the uncontrolled figure is the finding: a month with no controller at all passes
|
||||
it. On the other reading of the same sentence it is not met - the guard takes
|
||||
August's peak from **5029.8 W to 4107.3 W**, still over the 3500 W soft limit,
|
||||
because once a month peak is set `CEILING_MONTH_PEAK` defends that peak rather
|
||||
than the cap. Neither number is hidden; both are in that directory's README.
|
||||
|
||||
The size decision is committed and measured, not absorbed: a `git bundle` of
|
||||
the whole history goes from **1.58 MB to 5.28 MB** (+3.70 MB, x3.3), which a
|
||||
Home Assistant supervisor clones on every install. Downsampling was not
|
||||
available - `meter_max_age_s` is 30 s, so any coarser resample converts real
|
||||
coverage into modelled meter-gap and moves the number being measured.
|
||||
|
||||
The thirty files sit in a SUBDIRECTORY on purpose: `test_tariff.py` part 9 and
|
||||
`test_peakguard.py` part 6 glob `scenarios/p1-net-*.jsonl.gz` non-recursively,
|
||||
so CI keeps replaying the two small days and does not grow ~50 s per PR. Both
|
||||
files' honesty notes are corrected to point at the month.
|
||||
|
||||
**OPT-04's AC4 is met: twelve months of real EPEX BE day-ahead prices are now in
|
||||
the repository, and the strategy has been replayed against them (DATA-02).**
|
||||
AC4 asked for "net-positive across 12 months of replayed prices". OPT-04 shipped
|
||||
it as NOT MET and replayed a synthetic year instead, because `scenarios/` held
|
||||
only P1 meter captures. That was the right call at the time and the wrong
|
||||
diagnosis of why: the criterion was being carried alongside a 30-day *meter*
|
||||
criterion as though both needed waiting, and EPEX BE day-ahead is **public data,
|
||||
not a capture**. It needed a fetch, not a month.
|
||||
|
||||
`scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json` — **8760 hourly points**,
|
||||
2025-09-01 to 2026-09-01 Brussels-local, in §9.3's canonical three-field shape.
|
||||
`scenarios/fetch_epex_be.py` produced it and `--verify` re-derives every claim it
|
||||
makes about itself without touching the network.
|
||||
|
||||
**Licensing was checked before anything was committed.** The source is
|
||||
Fraunhofer ISE's Energy-Charts API, which publishes the **BE** zone under
|
||||
**CC BY 4.0 from Bundesnetzagentur | SMARD.de** and other zones under
|
||||
private-use-only terms — so this zone is redistributable with attribution and the
|
||||
data is committed rather than left as a script plus a checksum. The fetcher
|
||||
**refuses to write a file** whose per-response `license_info` stops saying
|
||||
CC BY 4.0. Cross-checked against the upstream: SMARD's own `4996/DE` series
|
||||
matched this feed on all 96 quarter-hours of 2026-04-10, maximum difference
|
||||
0.00 EUR/MWh.
|
||||
|
||||
**The figures are raw wholesale and the file says so.** `"all_in": false`, with
|
||||
the markup/tax assumption carried in `provenance.all_in_assumption` as *numbers
|
||||
to apply* rather than numbers already applied — TARIFF-02's double-counting trap
|
||||
in data form. `test_arbitrage.py` section 7B applies it in one visible place and
|
||||
**asserts arithmetically** that its two additive terms cancel out of every daily
|
||||
spread, so the trade counts depend on the VAT multiplier alone and not on the
|
||||
guessed markup. The same replay is run on raw wholesale as well, so how much of
|
||||
the answer rests on the guess is reported (110 trades all-in, 88 raw).
|
||||
|
||||
**DST is handled by construction, not by a special case.** The fetcher iterates
|
||||
in UTC and renders through `price.brussels()`, so **2025-10-26 comes out 25
|
||||
points and 2026-03-29 comes out 23** — verified by counting per Brussels day
|
||||
rather than asserting 24, and by checking that October's 02:00 appears twice at
|
||||
`+02:00` and `+01:00` with two different prices while March's 02:00 does not
|
||||
appear at all. The window is bounded at *local* midnight: bounding it at 00:00
|
||||
UTC produced a 22-hour first day, and a 24-hour day silently missing from the
|
||||
end, both of which the point-and-gap accounting caught.
|
||||
|
||||
**The result, at the shipped default, whatever it is.** At
|
||||
`cycle_cost_threshold_eur` **0.10** widened by FCST-03's no-weather **1.5×** to a
|
||||
required 0.1500 EUR/kWh, the real year completes **110 round trips** and is
|
||||
**net-positive at +24.88 EUR** of benefit net of the cycle cost (2 kWh a trade).
|
||||
Unwidened it completes 173. **Nothing was tuned:** the shipped default and the
|
||||
2.5/1.5 factors are untouched, and every figure comes out of the same `decide()`.
|
||||
|
||||
**What the product owner's open decision now has under it.** The four
|
||||
independent reports that a static day/night spread of ~0.0840 does not clear
|
||||
0.10 **all stand** — a static-tariff site still never trades. What was missing
|
||||
was the other half: a *dynamic* site's daily spread clears the widened
|
||||
requirement on **121 of 365 days**, median daily all-in spread **0.1130**. The
|
||||
threshold at which a median day would clear is **0.0754**, and at that value the
|
||||
same year completes 156 trips for +35.42 EUR. Both numbers are reported; neither
|
||||
is applied.
|
||||
|
||||
**And the synthetic year stays, because the real one cannot grade §17.** Not one
|
||||
of the 365 real days has a negative *dearest* hour — 372 negative wholesale hours
|
||||
and 0 days where the strategy would have sold into one — so "0 discharges into a
|
||||
negative-price hour" over real prices is a count of something that could not
|
||||
happen, and deleting the guard would not change it. Section 7B asserts that
|
||||
degeneracy instead of quietly reporting the zero; section 7's generator is still
|
||||
the only thing here that puts a negative price where the strategy would sell.
|
||||
|
||||
**Size, measured rather than assumed.** Plain JSON: 893 KB in the working tree
|
||||
but **74.6 KB as a git object**, against 56.5 KB for a committed `.json.gz`. Git
|
||||
already compresses blobs, so gzipping saves 18 KB and costs a reviewer the
|
||||
ability to read the provenance block — including the licence claim they are
|
||||
meant to check — in the diff. Plain, and `load()` still accepts `.gz` for a
|
||||
future dataset that earns it.
|
||||
|
||||
**Nothing is wired into the add-on.** `generic_dayahead` remains the live fetch
|
||||
path; nothing under `app/` imports the script, and the dataset is for replay.
|
||||
|
||||
**The capacity-peak guard's forced discharge now has an EMS-side energy floor
|
||||
(SAFETY-24).** SAFETY-07 can command a discharge to hold a billed quarter down,
|
||||
and nothing below it consulted the state of charge — `soc_floor` belongs to the
|
||||
maintenance cycle, and freeze.py already recorded that for the grid-following
|
||||
loop "the BMS is the sole energy backstop". On the 2026-08-20 replay the guard
|
||||
asks for **878 Wh in a day**. Relying on the BMS is not absurd — it is a real
|
||||
floor and it works — but it is the pack's protection rather than the operator's,
|
||||
and a month of aggressive quarters leaves the battery repeatedly at that cut-out
|
||||
with no EMS-side record of why.
|
||||
|
||||
`decide()` now takes `soc` and `soc_floor`, both **required** and neither
|
||||
defaulted. Above the floor nothing changes. At or below it the guard files a
|
||||
**0 W** floor instead of a sized one — one-directional, so charging stays blocked
|
||||
and no discharge is capped — and reports `floor_limited` with the reason it
|
||||
bound. It is not a trip: every stop in §11.2's table still outranks the claim,
|
||||
and the floor can never be negative, so §9.1's "discharge and export are never
|
||||
capped by the peak guard" survives as a property of the object (swept: 0 negative
|
||||
floors across 1296 combinations of load, elapsed, battery, SoC and floor).
|
||||
|
||||
**Failing to shave is not silent.** A floor-limited cycle raises one notification
|
||||
per episode (`goodwe_peak_soc_floor`), distinct from the saturation alarm because
|
||||
the two ask the operator for opposite things, and renders its own status row.
|
||||
An **unreadable SoC is floor-limited, not a full battery** — a full battery is
|
||||
exactly the value that would disarm the floor, and this project has shipped that
|
||||
defaulting once before (`| float(100)` in the stage-5 loop).
|
||||
|
||||
**Verified by replay, not by argument.** The 2026-08-20 capture is driven against
|
||||
a modelled pack sized from the day's own measured appetite, so it genuinely
|
||||
exhausts: without the floor it reaches the BMS cut-out and sits there for 1227
|
||||
guard cycles; with it the pack stops at 20.0 % and never touches the BMS. The
|
||||
energy withheld equals the energy between the two floors to within one cycle at
|
||||
the rail. The honest half is reported too — the worst quarter goes from 4338 W to
|
||||
5030 W, because a floor-limited guard shaves less on purpose.
|
||||
|
||||
**Two new options, mirrored once.** `peak_soc_floor` (20) and `bms_soc_floor`
|
||||
(10), whose defaults live in `peakguard.PEAK_OPTION_DEFAULTS` — TARIFF-02's
|
||||
pattern — imported by `validate.py` and `main.py` rather than typed as literals,
|
||||
and pinned to `config.yaml` by a check that reads the file. CFG-03 still tracks
|
||||
three places. `validate.py` rule 8 refuses `peak_soc_floor <= bms_soc_floor` at
|
||||
startup, in CFG-01/MAINT-04's manner. `maintenance_soc_floor` is deliberately
|
||||
**not** reused and deliberately **not** renamed: it is a drain target, not a
|
||||
spend limit, and the rename is recorded as a decision to revisit if a third
|
||||
floor ever appears.
|
||||
|
||||
**Price arbitrage - the first strategy that trades on a price (OPT-04).**
|
||||
Four cards built the price stack and nothing traded on it. `app/arbitrage.py`
|
||||
is §10.1's priority 3: charge in the cheapest published hour, discharge in the
|
||||
dearest, **and only when the spread covers `cycle_cost_threshold_eur`**. It is
|
||||
off by default (`arbitrage_enabled: false`) - it is the first strategy in this
|
||||
add-on that commands the battery against the meter on purpose.
|
||||
|
||||
**It consumes the four cards rather than re-deciding anything.** Whether a
|
||||
trade is permitted at all, on which two hours, and against which widened
|
||||
threshold is `TariffPlan.arbitrage()`'s answer (TARIFF-03), and that answer
|
||||
cannot be obtained without first stating whether the peak guard is holding. So
|
||||
§9.1's "give peak shaving priority over arbitrage" is honoured
|
||||
**structurally**: there is no code path to a tradeable spread that skips the
|
||||
question. And because the claim is filed at `P_ARBITRAGE` (30) while peakguard
|
||||
files its floor at `P_PEAK` (50), the arbiter clamps an arbitrage charge even
|
||||
if that interlock were bypassed. Two independent mechanisms, neither
|
||||
re-implemented here.
|
||||
|
||||
**A guard that never ran is not a guard that is clear.** On `meter_source: off`
|
||||
there is no telegram stream, so the peak guard is never evaluated. That third
|
||||
state is carried as `None` and it **refuses**: charging at the rail creates the
|
||||
billed quantity while nothing measures the quarter it lands in. `validate.py`
|
||||
refuses that configuration at startup too, along with arbitrage armed on a site
|
||||
with no price source at all, and a SoC floor at or above the ceiling.
|
||||
|
||||
**§17: never export into a negative price.** The guard is on the hour being
|
||||
sold into, not on the spread - a day priced -0.30 ... -0.05 has a 0.25 EUR/kWh
|
||||
spread that clears any shipped threshold and a dearest hour that is still
|
||||
negative. Buying in a negative hour stays allowed; being paid to charge is the
|
||||
right trade.
|
||||
|
||||
**Prices are fetched in the background, never on the tick.** `resolve_tariff()`
|
||||
may open an HTTPS connection, and the control tick is what feeds the ESP32
|
||||
heartbeat - a supplier's TLS timeout there is a supplier able to push the
|
||||
add-on towards layer 1's 30 s veto. A separate task refreshes every 30 minutes;
|
||||
the tick reads an attribute and an age bound, and a plan whose refresh has
|
||||
stopped answering retires rather than standing as if it were current.
|
||||
|
||||
**What is honestly not measured.** The acceptance criterion asked for
|
||||
net-positivity across 12 months of *replayed* prices. This repository holds no
|
||||
price history - `scenarios/` is P1 meter data - so that criterion is **not met,
|
||||
and is not restated**. What `test_arbitrage.py` section 7 does instead is
|
||||
replay a synthetic year and assert the property that does not need real prices:
|
||||
every executed round trip cleared its own widened requirement, on all of them,
|
||||
across a year containing 522 hours priced below zero and zero discharges into
|
||||
one. And at the shipped 0.10 EUR/kWh threshold a realistic Belgian day/night
|
||||
site **does not trade at all** - reported, not tuned. That number, and the
|
||||
widening factors beside it, are one product decision and it is open.
|
||||
|
||||
**Provisional prices, and what may not rest on one (FCST-03).**
|
||||
Day-ahead prices arrive once a day, around 14–16 h CET. Until they do, §8.3 says
|
||||
the unknown hours use yesterday's profile and are flagged `provisional: true` —
|
||||
|
||||
+92
-10
@@ -420,7 +420,7 @@ help - see "Why the tuning is what it is".
|
||||
| `maintenance_start_hour` | 10 | Hour of day a due cycle begins |
|
||||
| `maintenance_discharge_w` | 2000 | Drain rate (exports the surplus). Above `max_w` it is **clamped** to `max_w` — the drain phase SETs the setpoint and the arbiter bounds it, saying so on the status page (`limited by supervised(max_w)`). Warned about, not refused: the cycle still completes, just slower |
|
||||
| `maintenance_charge_w` | 2000 | Charge ceiling, capped again by peak headroom. Must stay **at or below `max_w`** or the cheap-window shaping (`hi = -maintenance_charge_w`) contradicts the supervised limit (`lo = -max_w`), the arbiter resolves it to **0 W**, and maintenance silently never charges in a cheap window — **the add-on refuses to start** |
|
||||
| `maintenance_soc_floor` | 11 | Drain target — stay just above the inverter's own floor |
|
||||
| `maintenance_soc_floor` | 11 | Drain target — stay just above the inverter's own floor. ⚠️ **This is not `peak_soc_floor`.** It is the depth the monthly recalibration deliberately goes to, once, on purpose; `peak_soc_floor` is the depth a money-saving discharge may never reach on any day. The right values differ (11 vs 20 as shipped), which is why they are two keys |
|
||||
| `maintenance_soc_target` | 99 | Charge target |
|
||||
| `maintenance_hold_min` | 120 | Hold at full so the BMS can balance |
|
||||
|
||||
@@ -430,6 +430,8 @@ help - see "Why the tuning is what it is".
|
||||
|---|---|
|
||||
| `peak_forecast_entity` | Quarter-hour demand forecast, for capacity-tariff markets. Empty = no cap |
|
||||
| `peak_cap_w` | The site's capacity-tariff target |
|
||||
| `peak_soc_floor` (20) | **The percent below which the capacity-peak guard will not discharge the battery.** Not optional and not the same thing as `maintenance_soc_floor` — see below |
|
||||
| `bms_soc_floor` (10) | The pack's **own** cut-out percent, the number the BMS enforces whatever you set above. `peak_soc_floor` must be **strictly above** it or the add-on refuses to start |
|
||||
| `price_now_entity`, `price_avg_entity` | Dynamic tariff. Empty = never force a paid grid top-up |
|
||||
|
||||
On a capacity-tariff site the maintenance charge is capped by the headroom left
|
||||
@@ -530,11 +532,46 @@ projected over — raises one notification (`goodwe_peak_saturated`), because th
|
||||
is the one state where the money will be spent and shedding load is the only
|
||||
thing left that changes the bill.
|
||||
|
||||
**No new option.** The ceiling is `peak_cap_w`, which already shipped. The
|
||||
**No new option for the CEILING.** It is `peak_cap_w`, which already shipped. The
|
||||
coverage threshold is `meter_max_age_s`, the freeze predicate is
|
||||
`meter_freeze_s`, and the achievable discharge comes from `max_w` and the device
|
||||
rating — every one of them a key that already exists.
|
||||
|
||||
#### The energy floor under the forced discharge (`peak_soc_floor`)
|
||||
|
||||
The guard above can **command a discharge**, and until SAFETY-24 nothing in the
|
||||
add-on bounded how far down it took the pack — the BMS was the only backstop.
|
||||
On the 2026-08-20 replay the guard asks for **878 Wh in a day**.
|
||||
|
||||
That is not absurd: the BMS floor is real and it works. But it is *the pack's*
|
||||
protection, not *yours*. A month of aggressive quarters can leave the battery
|
||||
repeatedly parked at that cut-out with no record on this side of why, and the
|
||||
first symptom — a battery that is never available when you want it — looks
|
||||
exactly like the SoC drift the monthly maintenance cycle exists to correct.
|
||||
|
||||
- **`peak_soc_floor` (20 %)** is where the forced discharge stops. Below it the
|
||||
guard files a **0 W** floor instead of a sized one: charging stays blocked
|
||||
(that can only ever lower your offtake), and no further discharge is asked for.
|
||||
- **It does not stop the ordinary self-consumption loop.** That loop still has no
|
||||
SoC guard of its own. This bounds what the *peak guard* spends for money.
|
||||
- **When it binds you are told**, once per episode, with notification
|
||||
`goodwe_peak_soc_floor` and a status-page row reading `PEAK GUARD
|
||||
FLOOR-LIMITED`: the quarter **will** go over the ceiling, so shedding load is
|
||||
the only thing left that changes the bill. Failing to shave is never silent —
|
||||
the euro figure is the whole point of the guard.
|
||||
- **An unreadable SoC is treated as floor-limited, not as a full battery.** A
|
||||
full battery is exactly the value that would disarm this floor.
|
||||
- **`bms_soc_floor` (10 %)** is the pack's own cut-out — register `0x0560` on the
|
||||
reference GW5000S-BP, consistent with DoD 90. It exists so the add-on can
|
||||
**refuse to start** when `peak_soc_floor` is at or below it: such a floor can
|
||||
never be the thing that stops the discharge, and a guard that advertises a
|
||||
backstop it does not have is the one thing this project will not ship. Equal is
|
||||
refused too — equal buys zero margin.
|
||||
|
||||
Why 20 %: the reference pack is 13.8 kWh usable, so 10 points of margin over the
|
||||
BMS cut-out is about 1.6× the guard's worst measured day. If your pack is
|
||||
smaller, or your quarters are worse, raise it.
|
||||
|
||||
### The winding-loop alarm (the output clamp)
|
||||
|
||||
Every setpoint this add-on writes is clamped to `max_w` before it leaves the
|
||||
@@ -644,15 +681,16 @@ floor of 60 s that figure is 60 W and only the margin is left.
|
||||
|
||||
### Energy price providers
|
||||
|
||||
> **No *fetched* price moves energy in 0.5.x.** This release ships the
|
||||
> *plumbing*: the five provider plugins, the canonical all-in format they emit,
|
||||
> and the config that selects one. Turning a provider on fetches, validates and
|
||||
> publishes prices and changes no charging decision; the arbitrage strategy that
|
||||
> would use it is a later ticket.
|
||||
> **A fetched price moves energy only when you switch arbitrage on.** This
|
||||
> release ships the five provider plugins, the canonical all-in format they
|
||||
> emit, the config that selects one, and — new in OPT-04 — a price-arbitrage
|
||||
> strategy that trades on them. With `arbitrage_enabled: false`, which is the
|
||||
> shipped default, turning a provider on still fetches, validates and publishes
|
||||
> prices and changes no charging decision.
|
||||
>
|
||||
> The one exception is `static_profile` below. A static day/night tariff needs
|
||||
> no network, so its cheap window is computed on every cycle and does reach the
|
||||
> maintenance cycle's grid top-up.
|
||||
> `static_profile` below needs no network at all, so its cheap window is
|
||||
> computed on every cycle and reaches the maintenance cycle's grid top-up
|
||||
> whether or not arbitrage is armed.
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
@@ -804,6 +842,50 @@ to, and the site simply has no tariff until the feed returns. It is not required
|
||||
— the add-on warns rather than refusing — but filling it in is what makes that
|
||||
fallback work.
|
||||
|
||||
### Price arbitrage (`arbitrage_enabled`)
|
||||
|
||||
§10.1 priority 3: charge in the cheapest published hour, discharge in the
|
||||
dearest, **and only when the spread covers what the round trip costs.** Off by
|
||||
default — this is the first strategy in the add-on that commands the battery
|
||||
*against* the meter on purpose, so it is armed deliberately or not at all.
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
| `arbitrage_enabled` | `false` by default. Arms the strategy |
|
||||
| `cycle_cost_threshold_eur` | What one full round trip costs per kWh — pack wear plus conversion losses. A spread below this is not traded |
|
||||
| `arbitrage_soc_floor` | The reserve arbitrage may not spend, % |
|
||||
| `arbitrage_soc_ceiling` | The headroom it may not overfill, % |
|
||||
|
||||
**What it needs, and what the add-on refuses.** Arbitrage requires a price
|
||||
source (`price_provider` or `static_profile`) and a P1 meter (`meter_source`).
|
||||
Without a price source there is nothing to tell cheap from expensive; without
|
||||
P1 the capacity-tariff peak guard never runs, and §9.1 puts peak shaving above
|
||||
arbitrage — a charge that nothing measures is exactly what that guard exists to
|
||||
prevent. Both combinations are **refused at startup** rather than left looking
|
||||
armed.
|
||||
|
||||
**Peak shaving always wins.** The strategy cannot obtain a spread at all
|
||||
without first answering whether the peak guard is holding, and even if it
|
||||
could, the guard's floor is filed at a higher arbiter priority and clamps an
|
||||
arbitrage charge. Discharging is never capped by it — only offtake is billed.
|
||||
|
||||
**It never exports into a negative price** (§17). A negative hour is the best
|
||||
possible hour to *charge* in and that is still allowed; selling into one gives
|
||||
energy away and pays for the privilege.
|
||||
|
||||
**At the shipped threshold a static day/night site does not trade.** The
|
||||
realistic Belgian day/night spread is about 0.08 EUR/kWh and the default
|
||||
threshold is 0.10, so the honest answer on such a site is "no opportunity", and
|
||||
the status page says so with the numbers. Whether 0.10 is the right default is
|
||||
a product decision, not a bug: lower it only if you know what a cycle costs
|
||||
your pack.
|
||||
|
||||
**The prices are fetched in the background**, never on the control tick — a
|
||||
supplier's TLS timeout on that path would be a supplier able to push the add-on
|
||||
towards the firmware's 30 s failsafe. The plan refreshes every 30 minutes and
|
||||
retires itself if the refresh stops answering, at which point arbitrage stops
|
||||
with it.
|
||||
|
||||
### Site
|
||||
|
||||
| option | meaning |
|
||||
|
||||
@@ -36,6 +36,15 @@ P_SAFETY = 100 # stopped, inputs missing, shutting down, device limits
|
||||
P_MAINTENANCE = 60 # maintenance owns the actuator outright (drain, hold)
|
||||
P_PEAK = 50 # capacity-tariff peak shaving - costs real money
|
||||
P_MAINT_SHAPE = 40 # maintenance shaping the loop (charge-only)
|
||||
# OPT-04. §10.1 priority 3 - price arbitrage. ⚠️ ITS POSITION IS THE SPEC, NOT A
|
||||
# preference: P_PEAK (50) is ABOVE it so clause 2 lets the capacity-tariff floor
|
||||
# clamp an arbitrage charge, which is §9.1's "give peak shaving priority over
|
||||
# arbitrage whenever the two conflict" expressed as arithmetic rather than as a
|
||||
# rule somebody has to remember. P_MAINT_SHAPE (40) is above it so a speculative
|
||||
# spread cannot override the pack's balancing cycle. P_LOOP (10) is below it
|
||||
# because trading against the meter is the entire point - the grid-following
|
||||
# loop would otherwise cancel every charge the moment it created import.
|
||||
P_ARBITRAGE = 30 # price arbitrage - trades against the meter on purpose
|
||||
P_LOOP = 10 # ordinary grid-following
|
||||
|
||||
SET = "set"
|
||||
|
||||
@@ -0,0 +1,429 @@
|
||||
"""OPT-04 - §10.1 priority 3: charge cheap, discharge expensive, and only when
|
||||
the round trip is worth more than it costs.
|
||||
|
||||
WHERE THIS LIVES, AND WHY IT IS NOT WHERE THE CARD SAYS. The functional notes
|
||||
on the Trello card reference specs.md §2.2's `ems/optimizer/...` layout. That
|
||||
module layout does not exist and is not being built; the product owner's
|
||||
decision of 2026-08-30 is that everything ships in `goodwe_controller/app/`,
|
||||
beside price.py, static_tariff.py, peakguard.py and arbiter.py. The BEHAVIOUR
|
||||
§10.1 / §9.1 / §8.3 / §17 describes is unchanged and still authoritative.
|
||||
|
||||
|
||||
WHAT THIS MODULE IS, AND - MORE IMPORTANTLY - WHAT IT IS NOT
|
||||
============================================================
|
||||
Four cards landed before this one and this one CONSUMES all four. Almost every
|
||||
question an arbitrage strategy has to answer is already answered somewhere
|
||||
else, by an object built for it, and the single biggest way to get this ticket
|
||||
wrong is to answer any of them again here:
|
||||
|
||||
price.py (TARIFF-02) what a price IS, all-in, and what a product is.
|
||||
`Uncertainty`, `WIDEN_FACTOR`, `required_spread`.
|
||||
static_tariff.py (TARIFF-03) `TariffPlan.arbitrage()` - THE VERDICT. Whether
|
||||
a trade is permitted at all, on which two hours,
|
||||
against which widened threshold. Five refusal
|
||||
reasons live there, not here.
|
||||
price_forecast.py (FCST-03) provisional carrying, and the rule that no
|
||||
irreversible action rests on a provisional
|
||||
price (§8.3).
|
||||
peakguard.py (SAFETY-07) the capacity-tariff floor, filed as a `lo`
|
||||
LIMIT at P_PEAK on the arbiter.
|
||||
|
||||
So what is left for this file is genuinely small, and that is the point:
|
||||
|
||||
* WHICH HOUR IS IT NOW - is this instant the buy hour, the sell hour, or
|
||||
neither.
|
||||
* MAY THE BATTERY DO IT - state of charge, and §17's negative-price rule.
|
||||
* HOW HARD - one number, filed as one arbiter claim.
|
||||
|
||||
Everything else is delegated, and delegated by CONSTRUCTION rather than by
|
||||
convention: `TariffPlan.arbitrage()` will not hand over a spread at all unless
|
||||
it is told whether the peak guard is holding, what a cycle costs, and what is
|
||||
uncertain. There is no code path here that can reach a tradeable number without
|
||||
having answered §10.1's priority ordering first.
|
||||
|
||||
|
||||
AC3 - PEAK SHAVING WINS, AND IT WINS TWICE
|
||||
==========================================
|
||||
§9.1: "Give peak shaving priority over arbitrage whenever the two conflict."
|
||||
That is honoured through the two mechanisms that already exist, and NEITHER of
|
||||
them is re-implemented here:
|
||||
|
||||
1. THE INTERLOCK. `plan.arbitrage(peak_guard_active=...)` is keyword-only and
|
||||
REQUIRED. When the guard is engaged the verdict is a refusal carrying the
|
||||
§10.1 wording, so `decide()` below never sees a spread at all. It is not
|
||||
possible to write a caller here that skips the question - the argument has
|
||||
no default and the spread is unreachable without it.
|
||||
|
||||
2. THE ARBITER. Even if (1) were somehow bypassed, `claim()` files a SET at
|
||||
P_ARBITRAGE (30) and peakguard files a LIMIT at P_PEAK (50). arbiter.py
|
||||
clause 2 applies every limit whose priority is >= the winning set's, most
|
||||
restrictive wins - so the peak floor CLAMPS an arbitrage charge whatever
|
||||
this module thinks. 50 >= 30 is what makes that true, and it is why the
|
||||
priority below sits under P_PEAK rather than beside it.
|
||||
|
||||
Two independent mechanisms for one rule is deliberate. The first is the one
|
||||
that reports honestly; the second is the one that holds when the first is
|
||||
wrong.
|
||||
|
||||
⚠️ AND THE GUARD MUST BE ABLE TO SEE. On `meter_source: off` there is no
|
||||
telegram stream, so peakguard.decide() is never run and `peak_guard_active` is
|
||||
not FALSE - it is UNKNOWN. Charging a battery at max_w creates offtake, offtake
|
||||
is the billed quantity (§9.1), and doing it deliberately while nothing measures
|
||||
the quarter is the exact shape of failure §9.1 exists to prevent. So the third
|
||||
state is carried explicitly as `None` and it REFUSES. A failed read is not an
|
||||
observation, and "the guard did not run" is not "the guard is clear".
|
||||
|
||||
|
||||
§17's NEGATIVE-PRICE CASE, STATED ONCE
|
||||
======================================
|
||||
"Negative-price hours - arbitrage must not export." A negative price means the
|
||||
market pays the site to consume. Discharging the battery into such an hour
|
||||
gives away stored energy AND forgoes being paid to take more in; if the house
|
||||
load is covered it becomes export, which at a negative price is paid-for
|
||||
disposal. The buy side is untouched - a negative hour is the best possible hour
|
||||
to charge in, and this file's guard is deliberately asymmetric for that reason.
|
||||
|
||||
⚠️ THE GUARD IS ON THE HOUR BEING SOLD INTO, NOT ON THE SPREAD. A series whose
|
||||
prices are -0.30 and -0.05 has a 0.25 EUR/kWh spread that clears any shipped
|
||||
threshold, and its DEAREST hour is still negative. A guard written as
|
||||
"spread > 0" or "the prices are not all negative" passes that case and exports.
|
||||
|
||||
|
||||
WHAT `cycle_cost_threshold_eur` IS, AND WHAT THIS FILE REFUSES TO DO WITH IT
|
||||
===========================================================================
|
||||
§10.1 gates arbitrage on "expected round-trip benefit exceeds
|
||||
`cycle_cost_threshold_eur`". It ships at 0.10 EUR/kWh. On realistic Belgian
|
||||
day/night numbers the static spread is about 0.08 EUR/kWh, and FCST-03's
|
||||
widening factors (2.5 real-time, 1.5 no-weather) are honest policy defaults
|
||||
with nothing to derive them from.
|
||||
|
||||
⚠️ SO A STATIC-TARIFF SITE DOES NOT TRADE AT SHIPPED DEFAULTS, AND THAT IS
|
||||
REPORTED RATHER THAN TUNED AWAY. The threshold is a parameter here, the
|
||||
behaviour is correct at any value of it, and no number in this file was chosen
|
||||
to make a trade happen. Whether 0.10 is the right shipped default - and whether
|
||||
2.5/1.5 are - is one product decision and it belongs to the owner, not to this
|
||||
card. test_arbitrage.py section 7 reports what the shipped default does over a
|
||||
replayed year rather than adjusting anything so that it looks better.
|
||||
|
||||
|
||||
THE ACCOUNTING BASIS, AND THE TERM THAT IS DELIBERATELY ABSENT
|
||||
=============================================================
|
||||
The card's technical notes ask for `benefit = value_out(d) - cost_in(c)/η`, and
|
||||
then say of the round-trip efficiency: "η is NOT a §14 key - it needs one, or
|
||||
must be derived from measured charge/discharge energy in the state store. DO
|
||||
NOT HARDCODE IT." Both halves are honoured by not having the term at all:
|
||||
|
||||
* THE SELL SIDE IS PRICED AS DISPLACED IMPORT, NOT AS EXPORT. The two hours
|
||||
compared are both all-in CONSUMPTION prices, which is the basis the notes
|
||||
require ("credit displacement at all-in import prices, not at export price,
|
||||
or the number flatters itself"). `export_price_eur_kwh` is therefore not
|
||||
read here at all - §9.3's near-zero Belgian injection price is precisely
|
||||
why the economically sound trade on this market is charge-cheap-to-
|
||||
displace-expensive-import, and §17's guard below is the same fact stated as
|
||||
a rule.
|
||||
|
||||
* AND THERE IS NO η, BECAUSE THERE IS NOTHING HONEST TO PUT IN IT. No config
|
||||
key exists, the state store carries no measured charge/discharge energy
|
||||
pair to derive one from, and a hardcoded 0.85 is exactly what the notes
|
||||
forbid - it would also DOUBLE-COUNT, because `cycle_cost_threshold_eur` is
|
||||
already defined as what a round trip costs per kWh and conversion loss is
|
||||
part of that cost. When a measured η exists it belongs beside the threshold
|
||||
as a second term, and this file's arithmetic is one subtraction, in one
|
||||
place, ready for it.
|
||||
|
||||
|
||||
HYSTERESIS (§10.2)
|
||||
==================
|
||||
The switching decision is "which hour is it", and an hour is 60 minutes wide,
|
||||
so the action cannot chatter within one. The other two terms are monotone
|
||||
inside an hour in the only direction that matters: a charge raises SoC towards
|
||||
the ceiling and a discharge lowers it towards the floor, so each can stop once
|
||||
and cannot restart in the same hour. No timer is needed and none is invented.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
from .arbiter import P_ARBITRAGE, Claim
|
||||
from .price import Absence, price_option
|
||||
|
||||
_LOG = logging.getLogger("goodwe.arbitrage")
|
||||
|
||||
SOURCE = "arbitrage"
|
||||
|
||||
# §10.2 reason codes. Written out because the action log is a user-facing
|
||||
# artefact and a code assembled from an f-string cannot be grepped for.
|
||||
CHARGE = "ARBITRAGE_CHARGE"
|
||||
DISCHARGE = "ARBITRAGE_DISCHARGE"
|
||||
|
||||
# Why nothing was done. These are the reasons this MODULE owns; the five
|
||||
# refusals that belong to the verdict itself are carried through verbatim from
|
||||
# static_tariff.py rather than restated, so a refusal's wording has exactly one
|
||||
# author.
|
||||
IDLE_NO_PLAN = "no tariff plan has been resolved yet"
|
||||
IDLE_NOT_COVERED = (
|
||||
"NOT COVERED: meter_source is off, so the capacity-tariff peak guard never "
|
||||
"runs and nothing measures the quarter this charge would land in. §9.1 puts "
|
||||
"peak shaving above arbitrage, and an unmeasured peak cannot be shaved - "
|
||||
"so the honest answer is that arbitrage is not covered on this install, not "
|
||||
"that the guard is clear")
|
||||
IDLE_NO_SOC = (
|
||||
"state of charge is not being read, and a battery whose fill nobody knows "
|
||||
"cannot be told to empty or to fill")
|
||||
IDLE_OFF_PEAK_HOUR = "this hour is neither the buy hour nor the sell hour"
|
||||
NEGATIVE_PRICE = (
|
||||
"§17: the sell hour is priced BELOW ZERO, so discharging into it would give "
|
||||
"energy away and pay for the privilege. Arbitrage must not export at a "
|
||||
"negative price")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ArbitrageDecision:
|
||||
"""What the strategy concluded this cycle, with what it concluded it from.
|
||||
|
||||
⚠️ NO FIELD HAS A DEFAULT, and that is TEST-14's rule rather than a style
|
||||
choice: an outcome type's required field quietly gaining a default is a
|
||||
mutation that survived review in three files in this repository. The
|
||||
signature is scanned in test_price_forecast.py alongside `Gate`'s and
|
||||
`Arbitrage`'s constructors, so a default added here later goes red without
|
||||
anybody remembering to write a case for it.
|
||||
|
||||
`action` is None whenever nothing is to be done, and `power_w` is then None
|
||||
too - not 0.0. Zero watts is a COMMAND (it parks the battery); "no claim"
|
||||
is the absence of one, and the arbiter must be free to let the loop win.
|
||||
"""
|
||||
|
||||
action: str | None # CHARGE | DISCHARGE | None
|
||||
power_w: float | None # signed: negative charges, positive discharges
|
||||
reason: str
|
||||
spread_eur_kwh: float | None
|
||||
required_eur_kwh: float | None
|
||||
price_now_eur_kwh: float | None
|
||||
|
||||
@property
|
||||
def trading(self) -> bool:
|
||||
return self.action is not None
|
||||
|
||||
def explain(self) -> str:
|
||||
if not self.trading:
|
||||
return f"arbitrage idle - {self.reason}"
|
||||
return self.reason
|
||||
|
||||
|
||||
def _idle(reason, *, spread=None, required=None, price=None) -> ArbitrageDecision:
|
||||
if not reason or not isinstance(reason, str):
|
||||
raise ValueError(
|
||||
"an idle arbitrage decision needs a reason a human can act on - "
|
||||
"§10.2 requires every plan change to carry one, and 'arbitrage did "
|
||||
"nothing' with no reason is indistinguishable from a bug.")
|
||||
return ArbitrageDecision(action=None, power_w=None, reason=reason,
|
||||
spread_eur_kwh=spread, required_eur_kwh=required,
|
||||
price_now_eur_kwh=price)
|
||||
|
||||
|
||||
def enabled(opts: dict) -> bool:
|
||||
"""Is the strategy switched on at all? Off in the shipped configuration.
|
||||
|
||||
⚠️ OFF BY DEFAULT ON PURPOSE. Every other card in this repository ships its
|
||||
machinery inert and lets an operator arm it; this one MOVES ENERGY against
|
||||
the meter on a real battery, so it is the last one that should be an
|
||||
exception to that.
|
||||
"""
|
||||
v = price_option(opts, "arbitrage_enabled")
|
||||
return v is True or str(v).strip().lower() in ("true", "yes", "on", "1")
|
||||
|
||||
|
||||
def threshold_eur_kwh(opts: dict) -> float:
|
||||
"""§10.1's `cycle_cost_threshold_eur`, from the one place option defaults live."""
|
||||
v = float(price_option(opts, "cycle_cost_threshold_eur"))
|
||||
if not math.isfinite(v) or v < 0:
|
||||
raise ValueError(
|
||||
f"cycle_cost_threshold_eur is {v!r}. It is what a round trip COSTS, "
|
||||
f"in EUR/kWh, so it is non-negative and finite - a negative "
|
||||
f"threshold is one every spread clears, which is arbitrage with the "
|
||||
f"brake removed.")
|
||||
return v
|
||||
|
||||
|
||||
def soc_bounds(opts: dict) -> tuple:
|
||||
"""(floor, ceiling) in percent. The reserve arbitrage may not spend."""
|
||||
return (float(price_option(opts, "arbitrage_soc_floor")),
|
||||
float(price_option(opts, "arbitrage_soc_ceiling")))
|
||||
|
||||
|
||||
def guard_state(peak_decision):
|
||||
"""The peak guard's answer as the three-state `decide()` takes: T / F / None.
|
||||
|
||||
THE ONE PLACE THIS CONJUNCTION IS WRITTEN, and it is public for the reason
|
||||
`peakguard.shaping_suspended()`, `watchdog.loop_is_following()` and
|
||||
`freeze.looks_frozen()` are: the check that grades it has to assert on THE
|
||||
SHIPPED EXPRESSION. A predicate transcribed into its own test is one copy
|
||||
plus a future disagreement about what the check means, and the copy that
|
||||
drifts is always the grading one - this repository has three separate
|
||||
records of exactly that, one of which left every suite green while a
|
||||
default install parked at 0 W in the quarter that sets the month's bill.
|
||||
|
||||
⚠️ `None` IS NOT `False` AND COLLAPSING THE TWO IS THE WHOLE HAZARD. On
|
||||
`meter_source: off` main.py never runs peakguard.decide() - there is no
|
||||
telegram stream, so there is no open quarter to project - and the decision
|
||||
it hands here is None. That is "the guard never ran", not "the guard is
|
||||
clear", and answering False would arm a strategy whose charging creates the
|
||||
billed quantity (§9.1) while nothing measures the quarter it lands in.
|
||||
"""
|
||||
if peak_decision is None:
|
||||
return None
|
||||
return bool(peak_decision.engaged)
|
||||
|
||||
|
||||
def _point_at(plan, when: datetime):
|
||||
"""The price point covering `when`, or None when the series does not cover it.
|
||||
|
||||
⚠️ None IS NOT A PRICE AND IS NOT ZERO. `TariffPlan` raises `Absence` here
|
||||
rather than answering 0.0, for price.py's reason - "no data" becoming "free"
|
||||
is the defect the whole outcome-type family exists to remove - and this
|
||||
turns that into "do nothing", never into "it is cheap".
|
||||
"""
|
||||
try:
|
||||
return plan.read.match(series=lambda r: r.at(when),
|
||||
no_data=lambda _r: None)
|
||||
except Absence:
|
||||
return None
|
||||
|
||||
|
||||
def decide(plan, now: datetime, *, peak_guard_active, soc,
|
||||
min_spread_eur_kwh, uncertainty, charge_w, discharge_w,
|
||||
soc_floor_pct, soc_ceiling_pct) -> ArbitrageDecision:
|
||||
"""Should the battery trade this cycle, and how hard.
|
||||
|
||||
⚠️ `peak_guard_active` IS A THREE-STATE ARGUMENT AND ALL THREE ARE REAL.
|
||||
True (the guard is holding), False (it ran and is clear), and None (it never
|
||||
ran, because nothing measures this install's quarter). See the module note:
|
||||
None refuses, because the alternative is a strategy that creates the billed
|
||||
quantity while the thing that would have limited it is switched off.
|
||||
|
||||
⚠️ EVERY OTHER ARGUMENT IS KEYWORD-ONLY AND HAS NO DEFAULT, for
|
||||
`TariffPlan.arbitrage()`'s reason. A defaulted `soc_floor_pct` is a reserve
|
||||
floor this module asserted rather than one the operator configured, and the
|
||||
first time it is wrong the battery is at 0 %.
|
||||
"""
|
||||
if peak_guard_active is None:
|
||||
return _idle(IDLE_NOT_COVERED)
|
||||
if plan is None:
|
||||
return _idle(IDLE_NO_PLAN)
|
||||
|
||||
verdict = plan.arbitrage(peak_guard_active=peak_guard_active,
|
||||
min_spread_eur_kwh=min_spread_eur_kwh,
|
||||
uncertainty=uncertainty)
|
||||
return verdict.match(
|
||||
refused=lambda v: _idle(v.reason),
|
||||
allowed=lambda v: _act(plan, now, v, soc, charge_w, discharge_w,
|
||||
soc_floor_pct, soc_ceiling_pct))
|
||||
|
||||
|
||||
def _act(plan, now, v, soc, charge_w, discharge_w, floor_pct, ceiling_pct):
|
||||
"""The permitted case: which hour is this, and may the pack do it."""
|
||||
here = _point_at(plan, now)
|
||||
if here is None:
|
||||
return _idle(
|
||||
f"the series carries a tradeable spread but no price for this "
|
||||
f"instant ({now.isoformat()}), which is §11.4 for this hour rather "
|
||||
f"than a reason to reach for a neighbouring one",
|
||||
spread=v.spread_eur_kwh, required=v.required_spread_eur_kwh)
|
||||
|
||||
price = here.eur_per_kwh
|
||||
buying = here.start == v.cheapest.start
|
||||
selling = here.start == v.dearest.start
|
||||
if not (buying or selling):
|
||||
return _idle(IDLE_OFF_PEAK_HOUR, spread=v.spread_eur_kwh,
|
||||
required=v.required_spread_eur_kwh, price=price)
|
||||
|
||||
# ⚠️ BELT AND BRACES ON §8.3, AND IT IS NOT DEAD CODE. `_firm_or_die` makes
|
||||
# the two points on the verdict firm at construction, but the point matched
|
||||
# HERE is whatever `at()` returned for this instant, and the identity of the
|
||||
# two rests on the series being the one the verdict was built from. An hour
|
||||
# that is provisional is one nobody published a price for, and no
|
||||
# irreversible action may rest on it.
|
||||
if here.provisional:
|
||||
return _idle(
|
||||
f"§8.3: the price for this hour ({here.start.isoformat()}) is "
|
||||
f"CARRIED FORWARD, not published, and no irreversible action may be "
|
||||
f"taken on a provisional price",
|
||||
spread=v.spread_eur_kwh, required=v.required_spread_eur_kwh,
|
||||
price=price)
|
||||
|
||||
if soc is None:
|
||||
return _idle(IDLE_NO_SOC, spread=v.spread_eur_kwh,
|
||||
required=v.required_spread_eur_kwh, price=price)
|
||||
|
||||
common = dict(spread=v.spread_eur_kwh, required=v.required_spread_eur_kwh,
|
||||
price=price)
|
||||
|
||||
if selling:
|
||||
# §17. THE NAMED REGRESSION CASE. Checked BEFORE the SoC question, so a
|
||||
# negative price reports the reason that actually forbids the trade
|
||||
# rather than one that happens to fire first on some other site.
|
||||
if price < 0:
|
||||
return _idle(f"{NEGATIVE_PRICE} - this hour is "
|
||||
f"{price:+.4f} EUR/kWh", **common)
|
||||
if soc <= floor_pct:
|
||||
return _idle(
|
||||
f"the sell hour is here at {price:.4f} EUR/kWh but the pack is "
|
||||
f"at {soc:g} %, at or under the {floor_pct:g} % reserve floor "
|
||||
f"arbitrage may not spend (§10.1 priority 4)", **common)
|
||||
return ArbitrageDecision(
|
||||
action=DISCHARGE, power_w=abs(float(discharge_w)),
|
||||
reason=(f"{DISCHARGE}: this is the dearest firm hour at "
|
||||
f"{price:.4f} EUR/kWh, {v.spread_eur_kwh:.4f} above the "
|
||||
f"cheapest and clear of the {v.required_spread_eur_kwh:.4f} "
|
||||
f"this cycle had to cover; discharging "
|
||||
f"{abs(float(discharge_w)):.0f} W from {soc:g} %"),
|
||||
spread_eur_kwh=v.spread_eur_kwh,
|
||||
required_eur_kwh=v.required_spread_eur_kwh,
|
||||
price_now_eur_kwh=price)
|
||||
|
||||
if soc >= ceiling_pct:
|
||||
return _idle(
|
||||
f"the buy hour is here at {price:.4f} EUR/kWh but the pack is at "
|
||||
f"{soc:g} %, at or over the {ceiling_pct:g} % ceiling - there is "
|
||||
f"nowhere to put the cheap energy", **common)
|
||||
return ArbitrageDecision(
|
||||
action=CHARGE, power_w=-abs(float(charge_w)),
|
||||
reason=(f"{CHARGE}: this is the cheapest firm hour at "
|
||||
f"{price:.4f} EUR/kWh, {v.spread_eur_kwh:.4f} below the dearest "
|
||||
f"and clear of the {v.required_spread_eur_kwh:.4f} this cycle "
|
||||
f"had to cover; charging {abs(float(charge_w)):.0f} W from "
|
||||
f"{soc:g} %"),
|
||||
spread_eur_kwh=v.spread_eur_kwh,
|
||||
required_eur_kwh=v.required_spread_eur_kwh,
|
||||
price_now_eur_kwh=price)
|
||||
|
||||
|
||||
def claim(decision: ArbitrageDecision) -> Claim | None:
|
||||
"""The decision as one arbiter claim, or None when nothing is to be traded.
|
||||
|
||||
A SET at P_ARBITRAGE, and the priority is the whole §10.1 ordering:
|
||||
|
||||
P_PEAK 50 > this, so peakguard's `lo` floor CLAMPS an arbitrage
|
||||
charge (arbiter clause 2 applies every limit at or
|
||||
above the winner's priority). AC3, mechanically.
|
||||
P_MAINT_SHAPE 40 > this, so a maintenance charge window's `hi` shaping is
|
||||
not overridden by a price opportunity. §12.4's "money
|
||||
outranks the schedule" is about the PEAK guard, which
|
||||
is measured and billed; a speculative spread is not
|
||||
that, and letting it out-rank the pack's balancing
|
||||
cycle would be this card quietly rewriting §12.
|
||||
P_LOOP 10 < this, because trading against the meter is the point:
|
||||
the grid-following loop would otherwise cancel every
|
||||
charge the moment it created import.
|
||||
|
||||
⚠️ NONE MEANS NO CLAIM, NOT A 0 W CLAIM. Filing `Claim.set(..., 0.0)` here
|
||||
would out-rank the loop with a park order every cycle the strategy had
|
||||
nothing to say, i.e. it would silently disable self-consumption on any site
|
||||
that armed arbitrage.
|
||||
"""
|
||||
if not decision.trading or decision.power_w is None:
|
||||
return None
|
||||
return Claim.set(SOURCE, P_ARBITRAGE, float(decision.power_w),
|
||||
decision.action)
|
||||
@@ -28,10 +28,11 @@ import contextlib
|
||||
import json
|
||||
import logging
|
||||
import signal
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import aiohttp
|
||||
|
||||
from . import arbitrage
|
||||
from .arbiter import (
|
||||
P_LOOP, P_MAINT_SHAPE, P_MAINTENANCE, P_PEAK, P_SAFETY, Claim, resolve,
|
||||
)
|
||||
@@ -48,7 +49,8 @@ from .maintenance import IDLE, MaintConfig, Maintenance
|
||||
from .mqtt import MqttPublisher
|
||||
from . import peakguard
|
||||
from .p1 import P1Ingest, build_source, is_enabled
|
||||
from .static_tariff import static_status
|
||||
from .price import Uncertainty, price_option
|
||||
from .static_tariff import resolve_tariff, static_status
|
||||
from .tariff import PeakStore, PeakTracker
|
||||
from .validate import ConfigRefused, refuse, validate_startup_config
|
||||
from .watchdog import SignWatchdog, SignWatchdogConfig, loop_is_following
|
||||
@@ -57,6 +59,34 @@ from . import web
|
||||
OPTIONS_PATH = "/data/options.json"
|
||||
_LOG = logging.getLogger("goodwe")
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# OPT-04: THE BACKGROUND TARIFF REFRESH, AND WHY IT IS NOT OPTIONAL
|
||||
#
|
||||
# `static_tariff.resolve_tariff()` may perform an HTTPS fetch. The control tick
|
||||
# is what feeds the ESP32 heartbeat, and layer 1 commands 0 W if that heartbeat
|
||||
# stops for ~30 s - so a supplier's TLS timeout on the tick is a SUPPLIER able
|
||||
# to push this add-on into the failsafe. Every module in the price stack says
|
||||
# so in its own words (static_tariff.static_plan, resolve_tariff, main's
|
||||
# read_inputs) and this is the task those notes were waiting for.
|
||||
#
|
||||
# The tick therefore never awaits a fetch. It reads `self.tariff_plan`, an
|
||||
# attribute this task assigns, and nothing else.
|
||||
#
|
||||
# ⚠️ THE CADENCE IS NOT A GUESS AT HOW OFTEN PRICES CHANGE. A day-ahead feed
|
||||
# publishes ONCE, at 14-16 h CET, and §8.3's carried hours become firm at that
|
||||
# moment; 30 minutes is short enough that the switch is picked up inside the
|
||||
# hour it happens and long enough to be a polite 48 fetches a day. Nothing in
|
||||
# the control law depends on it.
|
||||
TARIFF_REFRESH_S = 1800.0
|
||||
# ⚠️ AND THE AGE BOUND IS NOT THE SAME QUESTION. A refresh that FAILS still
|
||||
# stores a plan - resolve_tariff() degrades to the §11.4 static fallback, which
|
||||
# is provisional and refuses arbitrage on its own. This bound covers the other
|
||||
# case: the refresh task itself dying, leaving a plan that was true hours ago
|
||||
# standing as if it were current. Three missed refreshes is the threshold, for
|
||||
# `p1.max_age_s`'s reason - one missed poll is not a fault, a run of them is.
|
||||
# A failed read is not an observation, and neither is a stale one.
|
||||
TARIFF_MAX_AGE_S = 3 * TARIFF_REFRESH_S
|
||||
|
||||
|
||||
def load_options() -> dict:
|
||||
try:
|
||||
@@ -199,14 +229,25 @@ class Controller:
|
||||
# the action log and the notification fire on transitions rather than
|
||||
# once per cycle for a quarter of an hour.
|
||||
#
|
||||
# ⚠️ NO CONFIG OF ITS OWN, for the reason the clamp watchdog gives ~90
|
||||
# lines up and TARIFF-01 repeats: the ceiling is `peak_cap_w`, already
|
||||
# shipped and already validated, and every other term is read off
|
||||
# something the arbiter or the averager already owns. CFG-03 still
|
||||
# tracks three places, not four.
|
||||
# ⚠️ THE CEILING HAS NO CONFIG OF ITS OWN, for the reason the clamp
|
||||
# watchdog gives ~90 lines up and TARIFF-01 repeats: it is `peak_cap_w`,
|
||||
# already shipped and already validated, and every other term of the
|
||||
# projection is read off something the arbiter or the averager already
|
||||
# owns.
|
||||
#
|
||||
# ⚠️ SAFETY-24'S FLOOR DOES HAVE ONE, AND IT COULD NOT NOT HAVE. How far
|
||||
# down the guard may spend the pack for money is a number only the site
|
||||
# knows, and refusing one set below the pack's own cut-out means knowing
|
||||
# that cut-out too. So this card adds exactly two keys and reads their
|
||||
# defaults from `peakguard.PEAK_OPTION_DEFAULTS` - TARIFF-02's mirror
|
||||
# pattern - rather than typing literals here. THAT is what keeps CFG-03
|
||||
# at three places: the divergence validate.py's rule 6 documents
|
||||
# (`maintenance_charge_w` 2000 there, 2500 here) is exactly what a
|
||||
# literal in this file would repeat.
|
||||
self.peak_guard = None
|
||||
self._peak_engaged = False
|
||||
self._peak_saturated = False
|
||||
self._peak_floor_limited = False
|
||||
self._maint_shape_suspended = False
|
||||
# Set by amain() once the transport is built, so the status page can show
|
||||
# what only the transport knows (homewizard_local's unchanged_s). Stays
|
||||
@@ -232,6 +273,16 @@ class Controller:
|
||||
# window we are in and why it is or is not cheap - so "Cheap window:
|
||||
# no" is an evidenced answer rather than a bare no.
|
||||
self.tariff_words = "no static profile"
|
||||
|
||||
# OPT-04. The tariff plan the BACKGROUND task last resolved, and when.
|
||||
# cycle() reads these two attributes and never awaits a fetch - see
|
||||
# TARIFF_REFRESH_S. `None` means no refresh has completed yet, which is
|
||||
# not "no prices" and certainly not "free": arbitrage.decide() reports
|
||||
# it as its own reason and does nothing.
|
||||
self.tariff_plan = None
|
||||
self.tariff_at = None
|
||||
self.arb = None
|
||||
self._arb_action = None
|
||||
self.last_write = None
|
||||
self.last_write_at = None
|
||||
self.last_write_ok = None
|
||||
@@ -494,6 +545,17 @@ class Controller:
|
||||
cap_w=float(self.o.get("peak_cap_w", 3500)),
|
||||
reading=self.peak.peak(now),
|
||||
achievable_w=min(max_w, abs(float(dev_top))),
|
||||
# ⚠️ SAFETY-24. `self.soc` IS THE MEASURED READING AND MAY BE None,
|
||||
# and it is handed over as None rather than defaulted here. An early
|
||||
# draft of the stage-5 loop read SoC as `| float(100)` and thereby
|
||||
# invented a full battery (NOTES.md 4.4b); a full battery is
|
||||
# precisely the value that disarms this floor. decide() treats None
|
||||
# as floor-limited, which is the direction that costs a shave rather
|
||||
# than the pack.
|
||||
soc=self.soc,
|
||||
soc_floor=float(self.o.get(
|
||||
"peak_soc_floor",
|
||||
peakguard.PEAK_OPTION_DEFAULTS["peak_soc_floor"])),
|
||||
max_hold_s=self.p1.averager.max_hold_s,
|
||||
frozen=looks_frozen(getattr(self.p1_source, "unchanged_s", None),
|
||||
standing, max_w, self.freeze_wd.cfg),
|
||||
@@ -514,6 +576,64 @@ class Controller:
|
||||
if self.peak_guard is not None:
|
||||
self.log_event(self.peak_guard.explain())
|
||||
|
||||
# --- OPT-04: price arbitrage (§10.1 priority 3) ----------------------
|
||||
# ONE `set` claim at P_ARBITRAGE, and app/arbitrage.py carries the whole
|
||||
# argument. What is decided HERE and nowhere else is what the strategy
|
||||
# is allowed to see:
|
||||
#
|
||||
# ⚠️ IT IS PLACED AFTER THE PEAK BLOCK BECAUSE IT READS ITS CONCLUSION.
|
||||
# `engaged_now` is this cycle's peak verdict, computed ~20 lines up.
|
||||
# Evaluated before it, this would hand the strategy the PREVIOUS cycle's
|
||||
# answer - one cycle of lag on the interlock that implements §9.1's
|
||||
# "peak shaving outranks arbitrage", inside a 900 s quarter.
|
||||
#
|
||||
# ⚠️ AND THE THIRD STATE IS NOT A TIDINESS. Without P1 the guard is not
|
||||
# CLEAR, it never RAN, and `None` says so. arbitrage.decide() refuses on
|
||||
# it, because charging at max_w creates the billed quantity while
|
||||
# nothing is measuring the quarter it lands in. `False` here would be
|
||||
# this line asserting a measurement that was never taken - the defect
|
||||
# `PeakReading` and `quarter_ceiling_w()` both exist to remove.
|
||||
#
|
||||
# ⚠️ THE THREE-STATE COLLAPSE LIVES IN arbitrage.guard_state(), NOT
|
||||
# HERE, and that is not tidiness. It is the same lesson
|
||||
# `shaping_suspended()` records: inline, this expression was reachable
|
||||
# by no check, and mutating it back to a bare `engaged_now` left every
|
||||
# suite in this repository green while a `meter_source: off` install
|
||||
# traded blind. Read guard_state() before touching this line.
|
||||
#
|
||||
# ⚠️ NO FETCH ON THIS PATH. `fresh_tariff_plan()` is an attribute read
|
||||
# and an age comparison; the HTTPS call lives in run_tariff_refresh().
|
||||
# See TARIFF_REFRESH_S: a supplier's TLS timeout here would be a
|
||||
# supplier able to push this add-on towards layer 1's 30 s veto.
|
||||
#
|
||||
# ⚠️ AND THE UNCERTAINTY SET IS NOT EMPTY BY DEFAULT. §11.4's
|
||||
# no-weather row widens the threshold, and this add-on carries no
|
||||
# weather forecast at all - §8.1's PV forecast is not built. Passing
|
||||
# `frozenset()` would be main.py asserting a forecast is healthy when
|
||||
# there is not one, which is exactly what static_tariff's `uncertainty`
|
||||
# argument has no default in order to prevent.
|
||||
self.arb = None
|
||||
if arbitrage.enabled(self.o):
|
||||
floor_pct, ceiling_pct = arbitrage.soc_bounds(self.o)
|
||||
self.arb = arbitrage.decide(
|
||||
self.fresh_tariff_plan(now), now,
|
||||
peak_guard_active=arbitrage.guard_state(self.peak_guard),
|
||||
soc=self.soc,
|
||||
min_spread_eur_kwh=arbitrage.threshold_eur_kwh(self.o),
|
||||
uncertainty=frozenset({Uncertainty.NO_WEATHER}),
|
||||
charge_w=max_w, discharge_w=max_w,
|
||||
soc_floor_pct=floor_pct, soc_ceiling_pct=ceiling_pct)
|
||||
arb_claim = arbitrage.claim(self.arb)
|
||||
if arb_claim is not None:
|
||||
claims.append(arb_claim)
|
||||
# Edge-logged, for the peak guard's reason: the action log holds 40
|
||||
# lines and a trade lasts a whole hour, so a line per cycle would
|
||||
# flush the context that explains it. The transition is what an
|
||||
# operator needs - "it started buying at 03:00", not 240 copies.
|
||||
if self.arb.action != self._arb_action:
|
||||
self._arb_action = self.arb.action
|
||||
self.log_event(self.arb.explain())
|
||||
|
||||
# --- maintenance -----------------------------------------------------
|
||||
if result.owns_setpoint:
|
||||
claims.append(Claim.set("maintenance", P_MAINTENANCE,
|
||||
@@ -682,6 +802,16 @@ class Controller:
|
||||
# method assigned above and nothing since has touched.
|
||||
await self.check_peak_saturated()
|
||||
|
||||
# --- SAFETY-24's alarm, and it is the OTHER way the money gets spent --
|
||||
# Same boundary argument as check_peak_saturated(): it grades
|
||||
# `self.peak_guard`, which cycle() assigned above and nothing since has
|
||||
# touched. Kept as its own method rather than folded into that one,
|
||||
# because the two states are mutually exclusive and want OPPOSITE
|
||||
# operator actions - saturation says "shed load, the inverter is already
|
||||
# flat out"; this says "the pack is at its floor and this quarter is not
|
||||
# being shaved at all".
|
||||
await self.check_peak_floor_limited()
|
||||
|
||||
# --- TARIFF-01, and it is LAST on purpose -----------------------------
|
||||
# Banking a quarter is bookkeeping about a moment that has already
|
||||
# happened; a slow SD card must never sit between the meter and the
|
||||
@@ -826,6 +956,70 @@ class Controller:
|
||||
f"thing that changes the bill."),
|
||||
})
|
||||
|
||||
async def check_peak_floor_limited(self) -> None:
|
||||
"""SAFETY-24. Alarm when the EMS SoC floor stops the guard shaving.
|
||||
|
||||
⚠️ FAILING TO SHAVE MUST NOT BE SILENT, AND THAT IS THE WHOLE POINT.
|
||||
The guard exists for one euro figure - the capaciteitstarief bills the
|
||||
month's single worst quarter - so a cycle in which it CHOSE not to shave
|
||||
is a cycle in which money is being spent on purpose. Under-shaving
|
||||
quietly would leave the operator with a bill and a status page full of
|
||||
green.
|
||||
|
||||
The distinction from check_peak_saturated() is the operator's action,
|
||||
which is why these are two notifications and not one:
|
||||
|
||||
saturated the guard asked for everything the inverter has and it
|
||||
was not enough. SHED LOAD - the controller has already
|
||||
done all it can, and it is still discharging.
|
||||
floor-limited the guard asked for NOTHING, because the pack is at or
|
||||
under the EMS floor, or its SoC could not be read.
|
||||
Nothing is being discharged for this quarter at all.
|
||||
|
||||
They cannot both be true: peakguard.decide()'s floor-limited branch
|
||||
returns before the `achievable_w` clamp that sets `saturated`.
|
||||
|
||||
Edge-triggered, and de-asserting arms it again - FreezeWatchdog's shape,
|
||||
for FreezeWatchdog's reason: one notification per episode, not one per
|
||||
cycle for a quarter of an hour.
|
||||
"""
|
||||
g = self.peak_guard
|
||||
limited = bool(g is not None and g.floor_limited)
|
||||
was, self._peak_floor_limited = self._peak_floor_limited, limited
|
||||
if not limited or was:
|
||||
return
|
||||
# No prefix: `reason` already opens with PEAK GUARD FLOOR-LIMITED, and
|
||||
# a second copy is the kind of stutter that makes a 40-line action log
|
||||
# read like boilerplate. The saturated path needs its prefix because its
|
||||
# reason carries the word in the middle of a sentence, not at the front.
|
||||
self.log_event(g.reason)
|
||||
_LOG.error(
|
||||
"SAFETY-24 - %s. The controller is STILL RUNNING and charging is "
|
||||
"still blocked, but it is not discharging to hold this quarter, so "
|
||||
"the quarter will set or raise the month's billed capacity peak. "
|
||||
"Shed load, or lower peak_soc_floor if this pack can afford it.",
|
||||
g.reason)
|
||||
# A fixed notification_id, and a DIFFERENT one from every other alarm
|
||||
# here - check_meter_frozen() gives the reasoning. A battery at its
|
||||
# energy floor is not a saturated inverter and is not a frozen meter,
|
||||
# and an operator who has to dismiss three identical-looking cards
|
||||
# learns to dismiss the one that mattered.
|
||||
await self.hass.call("persistent_notification", "create", {
|
||||
"notification_id": "goodwe_peak_soc_floor",
|
||||
"title": "GoodWe: the capacity-peak guard has run out of battery",
|
||||
"message": (
|
||||
f"{g.reason}.\n\nTHIS QUARTER WILL GO OVER THE "
|
||||
f"{g.ceiling_w:.0f} W ceiling and will set or raise the "
|
||||
f"month's billed capaciteitstarief peak, because the peak guard "
|
||||
f"is not allowed to discharge any further: {g.soc_floor_reason}."
|
||||
f" This is not a controller fault - it is the EMS-side energy "
|
||||
f"floor (peak_soc_floor) doing what it is set to do, and it sits "
|
||||
f"deliberately above the pack's own BMS cut-out. Charging stays "
|
||||
f"blocked. Shedding load is the only thing that changes the bill "
|
||||
f"now; if this pack can afford a deeper discharge, lower "
|
||||
f"peak_soc_floor."),
|
||||
})
|
||||
|
||||
async def check_meter_frozen(self) -> None:
|
||||
"""SAFETY-18. Alarm on a meter that answers forever without changing.
|
||||
|
||||
@@ -866,6 +1060,57 @@ class Controller:
|
||||
})
|
||||
|
||||
# -- tasks ---------------------------------------------------------------
|
||||
# -- OPT-04: the background tariff refresh -------------------------------
|
||||
def fresh_tariff_plan(self, now):
|
||||
"""The stored plan if it is still current, else None. NEVER a stale one.
|
||||
|
||||
⚠️ THIS IS THE ONE PLACE THE AGE IS JUDGED, and it is public for
|
||||
`loop_is_following()`'s reason: a check has to be able to assert on the
|
||||
shipped expression rather than on a transcription that drifts. Both
|
||||
terms are required - a plan with no timestamp is one nothing stored, and
|
||||
a timestamped plan older than TARIFF_MAX_AGE_S is a reading from a task
|
||||
that has stopped answering.
|
||||
"""
|
||||
if self.tariff_plan is None or self.tariff_at is None:
|
||||
return None
|
||||
if (now - self.tariff_at).total_seconds() > TARIFF_MAX_AGE_S:
|
||||
return None
|
||||
return self.tariff_plan
|
||||
|
||||
async def run_tariff_refresh(self) -> None:
|
||||
"""Resolve the tariff off the control path, forever. NEVER on the tick.
|
||||
|
||||
⚠️ IT CATCHES EVERYTHING AND KEEPS GOING, for MqttPublisher's reason:
|
||||
a supplier's outage, a certificate expiry, a malformed feed - all of it
|
||||
is pricing, and the battery does not care. What it must never do is
|
||||
raise out of here and take the task down, because then the plan freezes
|
||||
at its last value and looks current forever. That is what
|
||||
`fresh_tariff_plan()`'s age bound is the backstop for, and this is the
|
||||
half that keeps the task alive so the bound rarely matters.
|
||||
|
||||
⚠️ AND A FAILURE DOES NOT CLEAR THE PLAN. `resolve_tariff()` already has
|
||||
a §11.4 answer for a dead feed - degrade to the static profile, mark it
|
||||
provisional, arbitrage refuses - and that answer is better than the
|
||||
absence of one. Zeroing the plan here would replace a coarse tariff with
|
||||
no tariff, which is the opposite of what §11.4 asks for.
|
||||
"""
|
||||
horizon = float(price_option(self.o, "price_horizon_h"))
|
||||
while not self.stopping:
|
||||
now = datetime.now(timezone.utc)
|
||||
try:
|
||||
plan = await resolve_tariff(
|
||||
self.o, now, now + timedelta(hours=horizon), now=now)
|
||||
self.tariff_plan, self.tariff_at = plan, now
|
||||
_LOG.info("tariff refreshed: %s", plan.words)
|
||||
except Exception as err: # noqa: BLE001
|
||||
_LOG.warning(
|
||||
"tariff refresh failed (%s); keeping the plan resolved at "
|
||||
"%s. It retires on its own after %.0f s and arbitrage "
|
||||
"stops with it.", err, self.tariff_at, TARIFF_MAX_AGE_S)
|
||||
with contextlib.suppress(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(self.stop_event.wait(),
|
||||
timeout=TARIFF_REFRESH_S)
|
||||
|
||||
async def run_control(self) -> None:
|
||||
entity = self.o.get("setpoint_entity", "")
|
||||
if entity:
|
||||
@@ -1116,6 +1361,13 @@ class Controller:
|
||||
elif _pg.saturated:
|
||||
out.append({"ok": False, "warn": False,
|
||||
"text": "PEAK GUARD SATURATED - " + _pg.reason})
|
||||
elif _pg.floor_limited:
|
||||
# SAFETY-24. NOT ok, like saturation: in both states the money
|
||||
# is going to be spent and a human is the only actuator left. A
|
||||
# separate row from saturation because the two ask for opposite
|
||||
# things - see check_peak_floor_limited().
|
||||
out.append({"ok": False, "warn": False,
|
||||
"text": _pg.explain()})
|
||||
elif _pg.blind:
|
||||
out.append({"ok": False, "warn": True, "text": _pg.explain()})
|
||||
else:
|
||||
@@ -1243,6 +1495,11 @@ class Controller:
|
||||
("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")),
|
||||
("Cheap window", "yes" if self.cheap else "no"),
|
||||
("Tariff", self.tariff_words),
|
||||
# OPT-04. The strategy's own sentence, not a re-derivation of it -
|
||||
# every refusal it can make already carries a reason a human can
|
||||
# act on, and "Arbitrage: no" would throw all of them away.
|
||||
("Arbitrage", "off (arbitrage_enabled)" if self.arb is None
|
||||
else self.arb.explain()),
|
||||
]
|
||||
return {
|
||||
"banner": banner, "level": level, "rows": rows, "checks": checks,
|
||||
@@ -1370,6 +1627,19 @@ async def amain() -> None:
|
||||
|
||||
tasks = [asyncio.create_task(controller.run_control())]
|
||||
|
||||
# OPT-04. The tariff refresh runs as its OWN long-lived task, for the
|
||||
# reason P1 ingestion does and then some: `resolve_tariff()` may open an
|
||||
# HTTPS connection to a supplier, and the control tick is what feeds the
|
||||
# ESP32 heartbeat. A fetch on that path is a supplier's TLS timeout able
|
||||
# to push this add-on towards layer 1's 30 s veto. See TARIFF_REFRESH_S.
|
||||
#
|
||||
# ⚠️ STARTED ONLY WHEN ARBITRAGE IS ARMED. Nothing else in the add-on
|
||||
# reads the resolved plan - `cheap` still comes from static_status() on
|
||||
# the tick, which is pure - so on a default install this task would be
|
||||
# network traffic in aid of an attribute nobody reads.
|
||||
if arbitrage.enabled(opts):
|
||||
tasks.append(asyncio.create_task(controller.run_tariff_refresh()))
|
||||
|
||||
# P1 ingestion runs as its own long-lived task. ⚠️ It must not be driven
|
||||
# off the control loop: telegrams arrive every ~5 s and the loop would
|
||||
# decimate them, so the 15-minute average - the capacity-tariff billing
|
||||
|
||||
@@ -246,19 +246,100 @@ validate.py are untouched by this ticket, exactly as TARIFF-01 left them, and
|
||||
mismatch trap 14's own warning box describes.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
⚠️ WHAT THIS GUARD DOES NOT HAVE: AN ENERGY BACKSTOP. SEE SAFETY-24.
|
||||
SAFETY-24: THE ENERGY BACKSTOP THIS GUARD SHIPPED WITHOUT
|
||||
|
||||
The graded branch can FORCE DISCHARGE, and nothing in this file or in the loop
|
||||
below it consults the state of charge - `soc_floor` belongs to the maintenance
|
||||
cycle, and freeze.py already records that for the grid-following loop "the BMS
|
||||
is the sole energy backstop". On the 2026-08-20 replay the guard asks for 878 Wh
|
||||
in a day. Against a pack sitting near 9 % that is not nothing, and the guard will
|
||||
keep asking as long as the meter says the quarter is over the ceiling.
|
||||
SAFETY-07 left this file saying, in its own words, that the graded branch can
|
||||
FORCE DISCHARGE and that nothing below it consults the state of charge - "the
|
||||
BMS is the sole energy backstop" (freeze.py). On the 2026-08-20 replay the guard
|
||||
asks for 878 Wh in a day, and it keeps asking for as long as the meter says the
|
||||
quarter is over the ceiling. That is the behaviour working as designed, and it
|
||||
is unbounded from below by anything in the EMS.
|
||||
|
||||
That is deliberate for THIS card and it is filed as SAFETY-24. It is also the
|
||||
reason this must not go to a client install before SAFETY-24 lands: on the
|
||||
reference site the BMS is a known quantity and an operator is present, and on a
|
||||
client site neither is true.
|
||||
⚠️ STATE THE RISK FAIRLY, because overstating it is how a real one gets ignored.
|
||||
Relying on the BMS is not absurd: it is a genuine floor and it works. But it is
|
||||
THE PACK's protection, not the OPERATOR's. A month of aggressive quarters can
|
||||
leave the battery repeatedly parked at that floor with no EMS-side record of
|
||||
why, and the first symptom - a pack that is never available when it is wanted -
|
||||
is indistinguishable from the SoC drift MAINT-01 exists to correct.
|
||||
|
||||
So `decide()` now takes `soc` and `soc_floor`, and `discharge_permitted()` is
|
||||
the one place the question is answered. Four things about the shape of it:
|
||||
|
||||
1. IT IS NOT A TRIP, AND IT MUST NOT BECOME ONE. SAFETY-07's claim is a LIMIT
|
||||
and every stop in 11.2's table still outranks it. This floor is about
|
||||
ENERGY - how far down the guard may spend the pack for money - not about
|
||||
last-resort safety. The action when it binds is therefore to stop ASKING
|
||||
for more discharge, not to command anything.
|
||||
|
||||
2. THE ACTION IS `floor_w = 0.0`, WHICH IS THE BLIND BRANCH'S ACTION FOR THE
|
||||
BLIND BRANCH'S REASON. Zero is one-directional: it forbids CHARGING, which
|
||||
can only ever lower delivered offtake, and it forbids nothing else. It
|
||||
cannot pull the loop off a discharge it is already doing, because a `lo`
|
||||
limit structurally cannot cap discharge - the property this module's whole
|
||||
shape rests on, and one a NEGATIVE floor would destroy. Nothing here can
|
||||
produce a negative floor: the only two values `floor_w` ever takes are 0.0
|
||||
and `min(wanted, achievable)` with `wanted > 0` already established.
|
||||
|
||||
⚠️ AND IT DOES NOT BOUND THE LOOP. Self-consumption may still drain the
|
||||
pack past this floor, exactly as SAFETY-18 recorded. That is not fixed
|
||||
here and must not be claimed: this card bounds THE PEAK GUARD's OWN forced
|
||||
discharge, which is the thing SAFETY-07 introduced. A loop-side SoC guard
|
||||
is a different lever with a different right value.
|
||||
|
||||
3. IT IS GRADED ONLY WHERE IT IS NOT DEGENERATE. The SoC question is asked
|
||||
AFTER `wanted > 0` is established - i.e. only where a forced discharge is
|
||||
actually being asked for. Asked earlier it would grade a quantity that
|
||||
governs nothing, and would report "floor-limited" on a quiet afternoon
|
||||
where the guard was never going to discharge anyway.
|
||||
|
||||
4. AN UNREADABLE SoC IS NOT A FULL BATTERY. `soc=None` is floor-limited, not
|
||||
permitted. This is `PeakReading`'s lesson in the one place it bites
|
||||
hardest: a full battery is exactly the value that would disarm this floor,
|
||||
so a failed read defaulting to one would silently restore the unbounded
|
||||
discharge this card exists to remove. Both new arguments are REQUIRED
|
||||
keyword-only and neither has a default, which is TEST-14 read as a rule
|
||||
rather than as a story: a required field that gains a default is how three
|
||||
separate files in this repo shipped a quiet fallback.
|
||||
|
||||
AND FAILING TO SHAVE MUST NOT BE SILENT. The whole point of the guard is the
|
||||
euro figure, so a floor-limited cycle carries `floor_limited=True` and main.py
|
||||
raises ONE notification on the edge, saying the quarter will be exceeded - the
|
||||
same shape, and the same argument, as `saturated`. The two are mutually
|
||||
exclusive by construction: a floor-limited decision asks for nothing, so it
|
||||
cannot have run out of inverter.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
CFG-03: TWO NEW OPTIONS, MIRRORED ONCE, AND WHY BOTH ARE NEEDED
|
||||
|
||||
SAFETY-07 added none and said so. This card cannot: an EMS-side floor is a
|
||||
number only the site knows, and refusing one set below the pack's own cut-out
|
||||
means knowing that cut-out too.
|
||||
|
||||
* `peak_soc_floor` - the EMS floor. ⚠️ NOT `maintenance_soc_floor`. That is
|
||||
the maintenance cycle's DRAIN TARGET: the depth a monthly recalibration
|
||||
deliberately goes to, once, under supervision. This is the depth a
|
||||
money-saving discharge may never reach, on any day of the month. Sharing
|
||||
one key would couple two features whose right values differ and would widen
|
||||
CFG-03 rather than narrow it.
|
||||
* `bms_soc_floor` - the pack's own cut-out (register 0x0560 on the reference
|
||||
unit, 10 %). It is here so validate.py rule 8 can REFUSE a `peak_soc_floor`
|
||||
at or below it. That configuration is CFG-01 rule (a): the operator states
|
||||
a floor, the BMS enforces a different one, and the status page reports an
|
||||
energy backstop that can never bind.
|
||||
|
||||
Both defaults are mirrored ONCE, in `PEAK_OPTION_DEFAULTS` below, which is
|
||||
TARIFF-02's `PRICE_OPTION_DEFAULTS` pattern - imported by validate.py and by
|
||||
main.py, pinned to config.yaml by test_peakguard.py rather than transcribed.
|
||||
This module still reads no option itself: `decide()` takes both as ARGUMENTS,
|
||||
makes no `.get()` call, and the mirror is a default table, not a read.
|
||||
|
||||
⚠️ A DECISION RECORDED, NOT A RENAME DONE. Now that a second SoC floor exists,
|
||||
`maintenance_soc_floor` is the ambiguous name of the two - it reads like "the
|
||||
SoC floor" and is not. Renaming it is an option migration on a live install for
|
||||
a clarity gain, so the decision recorded here is: KEEP IT, and carry the
|
||||
distinction in DOCS.md and in this comment. Revisit if a THIRD floor appears -
|
||||
a loop-side guard would be the third - at which point the migration buys enough
|
||||
to be worth its cost.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
AND IT READS NO PER-PHASE QUANTITY, ON PURPOSE
|
||||
@@ -287,8 +368,44 @@ BLIND_FROZEN = "SAFETY-18: the meter is answering without changing"
|
||||
CEILING_CAP = "peak_cap_w"
|
||||
CEILING_MONTH_PEAK = "the month's banked peak"
|
||||
|
||||
# SAFETY-24. Why the forced discharge may not be sized. Reasons, not booleans,
|
||||
# for BLIND_*'s reason: the status page, the action log and the operator
|
||||
# notification all have to say WHICH, and "floor-limited" alone is unactionable.
|
||||
FLOOR_SOC_UNREADABLE = ("the battery state of charge was not measured this "
|
||||
"cycle, and an unreadable SoC is not a full battery")
|
||||
FLOOR_AT_SOC_FLOOR = "state of charge is at or below the peak guard's SoC floor"
|
||||
|
||||
SOURCE = "peak"
|
||||
|
||||
# ⚠️ CFG-03. THE ONE IN-CODE MIRROR OF config.yaml FOR THIS CARD'S OPTIONS, and
|
||||
# it is TARIFF-02's `PRICE_OPTION_DEFAULTS` pattern for TARIFF-02's reason: the
|
||||
# add-on already carries option defaults in three places (config.yaml's
|
||||
# `options:`, config.yaml's `schema:`, and main.py's literal `.get()`
|
||||
# fallbacks), and a fourth is the open ticket. validate.py and main.py both
|
||||
# import THIS dict rather than typing a literal, and test_peakguard.py pins it
|
||||
# to config.yaml by READING the file - so a drift goes red instead of shipping.
|
||||
#
|
||||
# ⚠️ NOT A READ. `decide()` takes both values as arguments and this module makes
|
||||
# no `.get()` call at all; the dict is a default table for its two callers.
|
||||
PEAK_OPTION_DEFAULTS = {
|
||||
# The EMS-side floor on SAFETY-07's forced discharge, in percent.
|
||||
#
|
||||
# 20 is DERIVED from the two numbers this repository already has, not
|
||||
# chosen. The reference pack is 300 Ah at 51.2 V with DoD 90 - 13.8 kWh
|
||||
# usable, the figure validate.py's rule 6 note already quotes - and the
|
||||
# 2026-08-20 replay models 878 Wh of guard discharge in a day, which is
|
||||
# 6.4 % of that. A floor 10 points above the pack's own 10 % cut-out
|
||||
# therefore leaves the guard roughly 1.6 of its worst observed days of room
|
||||
# before it stops, which is the smallest margin that lets the guard do its
|
||||
# job on a bad day and still never reach the BMS.
|
||||
"peak_soc_floor": 20,
|
||||
# The pack's OWN cut-out, in percent: what the BMS enforces regardless of
|
||||
# anything here. On the reference unit this is register 0x0560, read as 10,
|
||||
# and it is consistent with DoD 90 on a full pack. It exists so validate.py
|
||||
# rule 8 can refuse a `peak_soc_floor` at or below it - see that rule.
|
||||
"bms_soc_floor": 10,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PeakDecision:
|
||||
@@ -309,9 +426,19 @@ class PeakDecision:
|
||||
excess_w: float = 0.0
|
||||
wanted_w: float | None = None # the floor before `achievable_w` clamped it
|
||||
saturated: bool = False
|
||||
# SAFETY-24. True when a forced discharge WAS wanted and the EMS-side SoC
|
||||
# floor refused to size it. Mutually exclusive with `saturated` by
|
||||
# construction: this branch asks for nothing, so it cannot have run out of
|
||||
# inverter. `soc_floor_reason` says WHICH of the two ways it bound.
|
||||
floor_limited: bool = False
|
||||
soc: float | None = None
|
||||
soc_floor: float | None = None
|
||||
soc_floor_reason: str | None = None
|
||||
reason: str = ""
|
||||
|
||||
def explain(self) -> str:
|
||||
if self.floor_limited:
|
||||
return self.reason
|
||||
if not self.engaged:
|
||||
if self.projected_w is None:
|
||||
return f"peak guard idle - {self.reason}"
|
||||
@@ -383,7 +510,44 @@ def blindness(projection, max_hold_s, frozen: bool) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def decide(projection, *, batt_w, cap_w, reading, achievable_w,
|
||||
def discharge_permitted(soc, soc_floor) -> str | None:
|
||||
"""SAFETY-24. May the guard size a forced discharge, or why may it not?
|
||||
|
||||
Returns None when it may, and otherwise the REASON it may not - which the
|
||||
decision, the status page, the action log and the operator notification all
|
||||
carry, because "floor-limited" on its own tells nobody what to do.
|
||||
|
||||
THE ONE PLACE THIS QUESTION IS ANSWERED, and it is public for the reason
|
||||
`freeze.looks_frozen()`, `watchdog.loop_is_following()` and
|
||||
`shaping_suspended()` are: a safety predicate transcribed into its own check
|
||||
is one copy plus a future disagreement about what the check means, and the
|
||||
copy that drifts is always the grading one.
|
||||
|
||||
⚠️ `soc is None` IS NOT PERMISSION. A failed read is not an observation, and
|
||||
a FULL BATTERY is exactly the value that would disarm this floor - so a
|
||||
`float(soc or 100)` here would restore the unbounded discharge this card
|
||||
removes, silently, on the one install whose SoC sensor is misnamed. There is
|
||||
deliberately no default argument on either parameter.
|
||||
|
||||
⚠️ AND `soc_floor is None` IS NOT PERMISSION EITHER, for the same reason one
|
||||
step further out: a caller that has no floor configured has not thereby been
|
||||
granted an unbounded one.
|
||||
|
||||
Boundary: `<=`, not `<`. A pack sitting exactly ON the floor has already
|
||||
spent everything this guard was allowed to spend; permitting one more cycle
|
||||
there would make the floor mean "one cycle below the floor".
|
||||
"""
|
||||
if soc is None:
|
||||
return FLOOR_SOC_UNREADABLE
|
||||
if soc_floor is None:
|
||||
return ("no peak guard SoC floor is configured, so there is nothing to "
|
||||
"bound a forced discharge with")
|
||||
if float(soc) <= float(soc_floor):
|
||||
return FLOOR_AT_SOC_FLOOR
|
||||
return None
|
||||
|
||||
|
||||
def decide(projection, *, batt_w, cap_w, reading, achievable_w, soc, soc_floor,
|
||||
max_hold_s=None, frozen: bool = False) -> PeakDecision:
|
||||
"""The whole guard. Pure: no clock, no I/O, no state carried between calls.
|
||||
|
||||
@@ -395,6 +559,13 @@ def decide(projection, *, batt_w, cap_w, reading, achievable_w,
|
||||
claim can NEVER produce `lo > hi` in the resolver - a
|
||||
contradiction there resolves to 0 W, which would park the
|
||||
battery in the exact minute it should be discharging hardest.
|
||||
`soc` SAFETY-24. MEASURED state of charge, percent. None when the
|
||||
sensor for it did not answer - which is floor-limited, not
|
||||
permission.
|
||||
`soc_floor` SAFETY-24. The percent below which this guard may not spend
|
||||
the pack for money. REQUIRED, and deliberately without a
|
||||
default: a default here would be CFG-03's fourth defaults
|
||||
site and TEST-14's quiet fallback in one line.
|
||||
"""
|
||||
ceiling, from_where = quarter_ceiling_w(cap_w, reading)
|
||||
ceil_top = abs(float(achievable_w))
|
||||
@@ -454,12 +625,41 @@ def decide(projection, *, batt_w, cap_w, reading, achievable_w,
|
||||
wanted_w=wanted,
|
||||
reason="the load is under the level this quarter can afford")
|
||||
|
||||
# SAFETY-24 - AND ONLY HERE. `wanted > 0` is now established, so a forced
|
||||
# discharge really is being asked for and the SoC is the quantity that
|
||||
# decides whether it may be sized. Asked one line earlier it would grade a
|
||||
# number that governs nothing on every quiet cycle of the day, and would
|
||||
# report "floor-limited" over a guard that was never going to discharge.
|
||||
denied = discharge_permitted(soc, soc_floor)
|
||||
if denied is not None:
|
||||
# 0.0, not `wanted`, and not a negative number. Zero forbids CHARGING -
|
||||
# one-directional, cannot cost pack energy - and forbids nothing else;
|
||||
# a `lo` limit structurally cannot cap the discharge the loop is already
|
||||
# doing. See the module note: this bounds THE GUARD's forced discharge,
|
||||
# not the grid-following loop, which still has no SoC guard.
|
||||
return PeakDecision(
|
||||
engaged=True, floor_w=0.0, ceiling_w=ceiling,
|
||||
ceiling_from=from_where, projected_w=projected, excess_w=ex,
|
||||
wanted_w=wanted, floor_limited=True, soc=soc, soc_floor=soc_floor,
|
||||
soc_floor_reason=denied,
|
||||
reason=(
|
||||
f"PEAK GUARD FLOOR-LIMITED: the quarter projects "
|
||||
f"{projected:.0f} W against a {ceiling:.0f} W ceiling "
|
||||
f"({from_where}) and {wanted:+.0f} W of discharge would hold "
|
||||
f"it, but {denied}"
|
||||
+ ("" if soc is None else f" ({float(soc):.0f} %")
|
||||
+ ("" if soc is None or soc_floor is None
|
||||
else f" vs {float(soc_floor):.0f} %")
|
||||
+ ("" if soc is None else ")")
|
||||
+ " - commanding no forced discharge, so THIS QUARTER WILL GO "
|
||||
"OVER THE CEILING. Charging stays blocked (floor 0 W)"))
|
||||
|
||||
floor = min(wanted, ceil_top)
|
||||
saturated = floor < wanted
|
||||
return PeakDecision(
|
||||
engaged=True, floor_w=floor, ceiling_w=ceiling, ceiling_from=from_where,
|
||||
blind=None, projected_w=projected, excess_w=ex, wanted_w=wanted,
|
||||
saturated=saturated,
|
||||
saturated=saturated, soc=soc, soc_floor=soc_floor,
|
||||
reason=(
|
||||
f"PEAK GUARD: quarter projects {projected:.0f} W against a "
|
||||
f"{ceiling:.0f} W ceiling ({from_where}) with "
|
||||
@@ -513,5 +713,13 @@ def claim(decision: PeakDecision) -> Claim | None:
|
||||
"""
|
||||
if not decision.engaged or decision.floor_w is None:
|
||||
return None
|
||||
# Three reasons, not two. A `lo = 0 W` claim reaches the arbiter from THREE
|
||||
# different states now - blind, floor-limited, and a genuine floor that
|
||||
# happens to be zero - and the arbiter's explain() is what an operator reads
|
||||
# when the battery stops charging. Spelling them the same way would make the
|
||||
# SAFETY-24 case indistinguishable from the SAFETY-18 one on the one line
|
||||
# that gets read.
|
||||
return Claim.limit(SOURCE, P_PEAK, lo=float(decision.floor_w),
|
||||
reason=("blind" if decision.blind else "quarter peak"))
|
||||
reason=("blind" if decision.blind
|
||||
else "SoC floor" if decision.floor_limited
|
||||
else "quarter peak"))
|
||||
|
||||
@@ -607,6 +607,26 @@ PRICE_OPTION_DEFAULTS = {
|
||||
# test_price.py section 1. A second defaults dict in the new module would be
|
||||
# a fourth place option defaults live, which is exactly the open ticket.
|
||||
"static_profile": "",
|
||||
# OPT-04. §10.1 priority 3, and §14's `optimizer.cycle_cost_threshold_eur`.
|
||||
#
|
||||
# ⚠️ THEY LIVE HERE FOR CFG-03's SAKE, exactly as `static_profile` does, and
|
||||
# FCST-03's note above `WIDEN_FACTOR` names this as the place they would
|
||||
# land. `arbitrage_soc_floor` / `arbitrage_soc_ceiling` are the reserve
|
||||
# arbitrage may not spend and the headroom it may not overfill; they are
|
||||
# NOT the maintenance SoC keys, which describe a balancing cycle rather
|
||||
# than a trading reserve and would silently couple two unrelated policies.
|
||||
#
|
||||
# ⚠️ AND `cycle_cost_threshold_eur` IS 0.10 BECAUSE §10.1's NAME FOR IT SAYS
|
||||
# SO, NOT BECAUSE IT MAKES ANYTHING TRADE. On realistic Belgian day/night
|
||||
# numbers the static spread is ~0.08 EUR/kWh, so a static-tariff site does
|
||||
# NOT trade at this default. That is a reported result (test_arbitrage.py
|
||||
# section 7), and whether 0.10 - or FCST-03's 2.5/1.5 - is right is one
|
||||
# product decision that belongs to the owner. No number here was chosen to
|
||||
# make arbitrage happen.
|
||||
"arbitrage_enabled": False,
|
||||
"cycle_cost_threshold_eur": 0.10,
|
||||
"arbitrage_soc_floor": 20,
|
||||
"arbitrage_soc_ceiling": 95,
|
||||
}
|
||||
|
||||
# ⚠️ The keys the FRAMEWORK owns. A plugin that reads one of these is a plugin
|
||||
|
||||
@@ -131,6 +131,25 @@ NOT READY banner already names the problem, (b) does not apply.
|
||||
installed base is on a fixed or static tariff. Refusing to start without a
|
||||
price feed would lock those sites out of the add-on entirely.
|
||||
|
||||
8. `peak_soc_floor` at or below `bms_soc_floor`. [(a)]
|
||||
SAFETY-24, and the same shape as 1, 4, 5 and 6. The peak guard can COMMAND
|
||||
a discharge to hold a billed quarter down (SAFETY-07), and `peak_soc_floor`
|
||||
is the EMS-side bound on how far down it may take the pack. The BMS has its
|
||||
own cut-out and enforces it regardless. Set the EMS floor at or below that
|
||||
cut-out and THE BMS WINS: the EMS bound can never be the thing that stops
|
||||
the discharge, so the add-on carries an energy backstop that is incapable
|
||||
of binding while its status page reports one - rule 5's "a disarmed guard
|
||||
that advertises itself as armed", on the guard whose failure mode is a pack
|
||||
that is quietly never available when it is wanted.
|
||||
|
||||
⚠️ `<=`, NOT `<`, and the equality case is the interesting one. At exactly
|
||||
equal, the EMS floor and the pack's cut-out act at the same percent, so the
|
||||
EMS floor buys ZERO margin - it can never prevent the pack reaching the BMS
|
||||
limit, which is the entire thing this option exists to prevent. That is not
|
||||
"close to the rail", it is the disarmed case with an extra step. Rule 6
|
||||
permits equality because equality there leaves one reachable point that
|
||||
works; here it leaves none.
|
||||
|
||||
WHAT IS DELIBERATELY *NOT* REFUSED, because it is untidy rather than unsafe:
|
||||
|
||||
- `integrator_max_w` above `max_w`. DOCS.md is right that the only thing the
|
||||
@@ -151,12 +170,18 @@ WHAT IS DELIBERATELY *NOT* REFUSED, because it is untidy rather than unsafe:
|
||||
- `maintenance_discharge_w` above `max_w`. Clamped, not contradicted, and the
|
||||
clamp names itself in the arbiter's explain() - see rule 6. Warned about,
|
||||
not refused.
|
||||
- `peak_soc_floor` below `maintenance_soc_floor`, or above it. They are
|
||||
DIFFERENT QUANTITIES (SAFETY-24: a drain target versus a spend limit) and
|
||||
every ordering of the two is a legitimate site configuration. Refusing on
|
||||
an ordering would be inventing a relationship the system does not have.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from .arbitrage import enabled as arbitrage_on
|
||||
from .p1 import is_enabled, validate_meter_options
|
||||
from .price import PROVIDERS
|
||||
from .peakguard import PEAK_OPTION_DEFAULTS
|
||||
from .price import PRICE_OPTION_DEFAULTS, PROVIDERS
|
||||
from .static_tariff import ProfileInvalid, parse_static_profile
|
||||
|
||||
_LOG = logging.getLogger("goodwe.config")
|
||||
@@ -369,6 +394,36 @@ def validate_startup_config(opts: dict, dev_min=None, dev_max=None) -> list[str]
|
||||
# --- 7. two price sources on one site ------------------------------------
|
||||
problems.extend(_price_problems(opts))
|
||||
|
||||
# --- 8. an EMS energy floor the BMS does not already enforce -------------
|
||||
# ⚠️ THE FALLBACKS ARE peakguard.PEAK_OPTION_DEFAULTS, NOT LITERALS, and
|
||||
# that is CFG-03 read as a rule rather than as a complaint. Rule 6's own
|
||||
# comment documents a LIVE divergence in this repo caused by typing 2000
|
||||
# here and 2500 in main.py; this card refuses to add a second instance of
|
||||
# it. The dict is the single in-code mirror, pinned to config.yaml by
|
||||
# test_peakguard.py, and main.py reads the same one.
|
||||
soc_floor = _number(opts, "peak_soc_floor",
|
||||
PEAK_OPTION_DEFAULTS["peak_soc_floor"])
|
||||
bms_floor = _number(opts, "bms_soc_floor",
|
||||
PEAK_OPTION_DEFAULTS["bms_soc_floor"])
|
||||
if soc_floor is None or bms_floor is None:
|
||||
problems.append(
|
||||
"peak_soc_floor and bms_soc_floor must both be numbers (got "
|
||||
f"{opts.get('peak_soc_floor')!r} / {opts.get('bms_soc_floor')!r})")
|
||||
elif soc_floor <= bms_floor:
|
||||
problems.append(
|
||||
f"peak_soc_floor is {soc_floor:.0f} % but the pack's own cut-out "
|
||||
f"(bms_soc_floor) is {bms_floor:.0f} %, so SAFETY-07's forced "
|
||||
"discharge has no EMS-side bound at all - THE BMS WINS, and it is "
|
||||
"the pack's protection rather than yours. The guard would report a "
|
||||
"SoC floor on its status page while being incapable of ever being "
|
||||
"the thing that stops the discharge, and the symptom of that is a "
|
||||
"battery repeatedly sitting at the BMS limit with no record of why "
|
||||
f"- which looks exactly like the SoC drift the maintenance cycle "
|
||||
f"exists to correct. Raise peak_soc_floor above {bms_floor:.0f} % "
|
||||
f"(the shipped default is "
|
||||
f"{PEAK_OPTION_DEFAULTS['peak_soc_floor']:.0f} %), or correct "
|
||||
"bms_soc_floor if it does not match this pack.")
|
||||
|
||||
# --- warnings: real, but not worth refusing over -------------------------
|
||||
integ = opts.get("integrator_max_w")
|
||||
integ_w = None if not integ else _number(opts, "integrator_max_w", None)
|
||||
@@ -453,14 +508,21 @@ def _price_problems(opts: dict) -> list[str]:
|
||||
# arbitrage verdict, so the seam is no longer inert.
|
||||
# * main.py reads the STATIC side of that module every cycle
|
||||
# (static_status()), which makes static_profile a real writer of `cheap`.
|
||||
# * main.py does NOT yet call resolve_tariff() on the control tick, and
|
||||
# that is deliberate: a provider fetch is network I/O and the tick is
|
||||
# what feeds the ESP32 heartbeat. So a configured PROVIDER still shapes
|
||||
# no energy decision on its own; a configured STATIC PROFILE does.
|
||||
# * main.py does NOT call resolve_tariff() on the control tick, and that
|
||||
# is still deliberate: a provider fetch is network I/O and the tick is
|
||||
# what feeds the ESP32 heartbeat. OPT-04 put it on a BACKGROUND REFRESH
|
||||
# instead (Controller.run_tariff_refresh), and the tick reads the plan
|
||||
# that task last stored - an attribute read, with nothing to block on.
|
||||
# * so a configured PROVIDER now DOES shape an energy decision, but only
|
||||
# through app/arbitrage.py and only while `arbitrage_enabled` is on.
|
||||
# With it off - the shipped configuration - the series is still fetched,
|
||||
# validated and published, and still moves nothing.
|
||||
#
|
||||
# Hence two refusals below rather than one, and neither claims more than
|
||||
# that list. When OPT-04 puts resolve_tariff() on a background refresh, the
|
||||
# first message's middle clause is what changes - not the rule.
|
||||
# Hence three refusals below rather than two, and none of them claims more
|
||||
# than that list. ⚠️ THE MIDDLE CLAUSE OF THE FIRST MESSAGE IS WHAT OPT-04
|
||||
# CHANGED, exactly as the note it replaced predicted - the RULE is
|
||||
# untouched, because the reason it exists (main.py reads the Home Assistant
|
||||
# pair FIRST and the other source is silent) is unchanged.
|
||||
ha_price = [k for k in ("price_now_entity", "price_avg_entity")
|
||||
if str(opts.get(k, "") or "").strip()]
|
||||
if pid != "off" and ha_price:
|
||||
@@ -469,10 +531,12 @@ def _price_problems(opts: dict) -> list[str]:
|
||||
f"{'is' if len(ha_price) == 1 else 'are'} set, which is two price "
|
||||
"sources on one site. THE HOME ASSISTANT ENTITIES WIN: they are "
|
||||
"what main.py reduces to the `cheap` flag the maintenance cycle "
|
||||
"reads. The provider series IS consumed now - app/static_tariff.py "
|
||||
"builds a tariff plan and an arbitrage verdict from it - but it is "
|
||||
"not what main.py acts on, so the supplier you configured would "
|
||||
"shape no energy decision while the status page named it. Clear "
|
||||
"reads, and read_inputs() never looks at the other source once "
|
||||
"they are set. The provider series is consumed - "
|
||||
"app/static_tariff.py builds a tariff plan and an arbitrage "
|
||||
"verdict, and app/arbitrage.py trades on it off a background "
|
||||
"refresh - but the `cheap` flag those entities own is a separate "
|
||||
"decision they would silently keep. Clear "
|
||||
"price_now_entity and price_avg_entity to use the "
|
||||
f"{pid!r} plugin, or set price_provider to 'off' to keep the "
|
||||
"Home Assistant sensors.")
|
||||
@@ -528,6 +592,67 @@ def _price_problems(opts: dict) -> list[str]:
|
||||
"with no tariff at all rather than with a coarse one. Not refused, "
|
||||
"but filling it in is what makes that row work.", pid)
|
||||
|
||||
# ⚠️ OPT-04. THE THREE WAYS `arbitrage_enabled: true` CAN BE A LIE, refused
|
||||
# rather than logged, because a strategy that is armed in the UI and cannot
|
||||
# fire is indistinguishable from one that fires and finds no opportunity -
|
||||
# which is the §14 warning box's case exactly. None of these can break an
|
||||
# existing install: the option is new and ships `false`.
|
||||
# ⚠️ CHECKED WHETHER OR NOT ARBITRAGE IS ARMED. A threshold that is not a
|
||||
# non-negative number is a configuration error the moment it is typed, and
|
||||
# deferring the complaint until somebody turns the strategy on would report
|
||||
# it at the worst possible moment - the one where energy starts moving.
|
||||
thresh = _number(opts, "cycle_cost_threshold_eur",
|
||||
PRICE_OPTION_DEFAULTS["cycle_cost_threshold_eur"])
|
||||
if thresh is None or thresh < 0:
|
||||
problems.append(
|
||||
f"cycle_cost_threshold_eur is "
|
||||
f"{opts.get('cycle_cost_threshold_eur')!r} and must be a "
|
||||
f"non-negative number of EUR/kWh - it is what one round trip COSTS "
|
||||
f"(§10.1). A negative threshold is one that every spread clears, "
|
||||
f"including a spread of nothing, which is arbitrage with the brake "
|
||||
f"removed rather than arbitrage tuned aggressively.")
|
||||
|
||||
if arbitrage_on(opts):
|
||||
if pid == "off" and not str(opts.get("static_profile", "") or "").strip():
|
||||
problems.append(
|
||||
"arbitrage_enabled is on but price_provider is 'off' and "
|
||||
"static_profile is empty, so this site has NO PRICES AT ALL. "
|
||||
"§10.1's priority 3 is 'charge cheap, discharge expensive' and "
|
||||
"there is nothing here to tell cheap from expensive, so the "
|
||||
"strategy would refuse on every cycle forever while the "
|
||||
"configuration said it was running. Configure a price_provider "
|
||||
"or a static_profile, or turn arbitrage_enabled off.")
|
||||
if not is_enabled(opts):
|
||||
problems.append(
|
||||
"arbitrage_enabled is on but meter_source is 'off', so there "
|
||||
"is no P1 telegram stream, the capacity-tariff peak guard "
|
||||
"never runs, and nothing measures the 15-minute average this "
|
||||
"strategy's charging would land in. §9.1 gives peak shaving "
|
||||
"priority over arbitrage; an unmeasured peak cannot be shaved, "
|
||||
"so arbitrage would be deliberately creating the billed "
|
||||
"quantity with the one guard that limits it switched off. "
|
||||
"app/arbitrage.py refuses to trade in that state anyway - this "
|
||||
"refuses the configuration that asks it to. Set meter_source, "
|
||||
"or turn arbitrage_enabled off.")
|
||||
floor = _number(opts, "arbitrage_soc_floor",
|
||||
PRICE_OPTION_DEFAULTS["arbitrage_soc_floor"])
|
||||
ceiling = _number(opts, "arbitrage_soc_ceiling",
|
||||
PRICE_OPTION_DEFAULTS["arbitrage_soc_ceiling"])
|
||||
if floor is None or ceiling is None:
|
||||
problems.append(
|
||||
f"arbitrage_soc_floor {opts.get('arbitrage_soc_floor')!r} and "
|
||||
f"arbitrage_soc_ceiling {opts.get('arbitrage_soc_ceiling')!r} "
|
||||
f"must both be numbers - they are percentages of pack charge.")
|
||||
elif floor >= ceiling:
|
||||
problems.append(
|
||||
f"arbitrage_soc_floor is {floor:g} % and arbitrage_soc_ceiling "
|
||||
f"is {ceiling:g} %, so there is no band left between them: "
|
||||
f"every state of charge is either at or under the floor (no "
|
||||
f"discharge) or at or over the ceiling (no charge), and the "
|
||||
f"strategy can never act. The floor is the reserve it may not "
|
||||
f"spend and the ceiling is the headroom it may not overfill, "
|
||||
f"so the floor must be the SMALLER number.")
|
||||
|
||||
if pid == "manual_csv" and not str(opts.get("price_csv_path", "") or "").strip():
|
||||
problems.append(
|
||||
"price_provider is 'manual_csv' but price_csv_path is empty, so "
|
||||
|
||||
@@ -163,6 +163,37 @@ options:
|
||||
peak_forecast_entity: ""
|
||||
peak_cap_w: 3500
|
||||
|
||||
# --- SAFETY-24: the energy backstop under SAFETY-07's forced discharge -----
|
||||
# The peak guard can COMMAND a discharge to hold a billed quarter under
|
||||
# peak_cap_w. Until this card it was bounded from below by nothing but the
|
||||
# pack's own BMS cut-out - a real floor, but the PACK's protection rather than
|
||||
# the OPERATOR's. On the 2026-08-20 replay the guard asks for 878 Wh in a day.
|
||||
#
|
||||
# peak_soc_floor is the percent below which that discharge is not sized. When
|
||||
# it binds the guard reports FLOOR-LIMITED, keeps charging blocked (a 0 W
|
||||
# floor), and the add-on raises ONE notification saying the quarter will be
|
||||
# exceeded - failing to shave is never silent, because the euro figure is the
|
||||
# whole point of the guard.
|
||||
#
|
||||
# ⚠️ THIS IS NOT maintenance_soc_floor AND MUST NOT BE MERGED WITH IT.
|
||||
# maintenance_soc_floor is the DRAIN TARGET of the monthly recalibration: the
|
||||
# depth the cycle deliberately goes to, once a month, on purpose. This is the
|
||||
# depth a money-saving discharge may never reach on any day. The right values
|
||||
# differ, and sharing one key would couple two features that want to move
|
||||
# independently.
|
||||
#
|
||||
# ⚠️ 20 IS DERIVED, NOT PICKED. The reference pack is 13.8 kWh usable and the
|
||||
# worst modelled guard day is 878 Wh, i.e. 6.4 % - so 10 points above the
|
||||
# pack's own cut-out leaves the guard ~1.6 of its worst observed days of room
|
||||
# and still never reaches the BMS.
|
||||
peak_soc_floor: 20
|
||||
# The pack's OWN cut-out percent - what the BMS enforces whatever is set
|
||||
# above. Register 0x0560 reads 10 on the reference GW5000S-BP, consistent with
|
||||
# DoD 90. It is here so validate.py can REFUSE a peak_soc_floor at or below
|
||||
# it: such a floor can never bind, the BMS wins, and the status page would
|
||||
# report an energy backstop that does not exist.
|
||||
bms_soc_floor: 10
|
||||
|
||||
# ⚠️ THE OLD PRICE PATH, AND IT IS NOT DEAD YET. These two entities are the
|
||||
# add-on delegating the whole price question to Home Assistant: main.py reads
|
||||
# both and derives ONE BOOLEAN, `cheap = now <= avg`, which is consumed by
|
||||
@@ -240,6 +271,38 @@ options:
|
||||
# a second series: no shipped supplier publishes a dynamic injection curve.
|
||||
export_price_eur_kwh: 0.04
|
||||
|
||||
# --- price arbitrage (OPT-04, specs §10.1 priority 3 / §17) ---------------
|
||||
# Charge in the cheapest published hour, discharge in the dearest, and ONLY
|
||||
# when the spread covers what the round trip costs. Off in the shipped
|
||||
# configuration: this is the first strategy in the add-on that moves energy
|
||||
# against the meter on purpose, so it is armed deliberately or not at all.
|
||||
#
|
||||
# ⚠️ IT NEEDS A PRICE SOURCE AND A P1 METER, and validate.py refuses the
|
||||
# combinations that cannot work rather than letting them look armed. With
|
||||
# `price_provider: off` and no `static_profile` there is no series to trade
|
||||
# on; with `meter_source: off` the capacity-tariff peak guard never runs, and
|
||||
# §9.1 puts peak shaving above arbitrage - a charge nothing measures is the
|
||||
# exact failure that guard exists to prevent.
|
||||
#
|
||||
# ⚠️ AT THE SHIPPED THRESHOLD A STATIC DAY/NIGHT SITE DOES NOT TRADE. The
|
||||
# realistic Belgian day/night spread is about 0.08 EUR/kWh and the threshold
|
||||
# below is 0.10, so the verdict is "the firm spread does not cover this
|
||||
# cycle's cost" and nothing happens - correctly. That is a product decision
|
||||
# (this number, and app/price.py's widening factors) and it is open with the
|
||||
# owner; the code is correct at any value and no number was picked to make a
|
||||
# trade appear.
|
||||
arbitrage_enabled: false
|
||||
# §10.1's `cycle_cost_threshold_eur`: what one full round trip costs per kWh
|
||||
# - the wear on the pack plus the conversion losses. A spread below this is
|
||||
# a cycle that loses money and pack life, so it is not taken.
|
||||
cycle_cost_threshold_eur: 0.10
|
||||
# The reserve arbitrage may not spend, and the headroom it may not overfill.
|
||||
# Deliberately NOT the maintenance_soc_* keys: those describe a monthly
|
||||
# balancing cycle, and pointing a trading strategy at them would couple two
|
||||
# unrelated policies through one number.
|
||||
arbitrage_soc_floor: 20
|
||||
arbitrage_soc_ceiling: 95
|
||||
|
||||
# --- site ------------------------------------------------------------------
|
||||
estop_fitted: false
|
||||
log_level: info
|
||||
@@ -311,6 +374,12 @@ schema:
|
||||
|
||||
peak_forecast_entity: str?
|
||||
peak_cap_w: int(500,15000)
|
||||
# SAFETY-24. No "off" value: a guard that can command a discharge does not get
|
||||
# a supported way to have no energy bound at all. The ranges OVERLAP on
|
||||
# purpose - the schema cannot express "above bms_soc_floor", so validate.py
|
||||
# rule 8 is what refuses the pair, with a message naming both numbers.
|
||||
peak_soc_floor: int(5,60)
|
||||
bms_soc_floor: int(0,50)
|
||||
price_now_entity: str?
|
||||
price_avg_entity: str?
|
||||
|
||||
@@ -337,5 +406,15 @@ schema:
|
||||
# bottoms out at 0 would make that site inexpressible.
|
||||
export_price_eur_kwh: float(-1,1)
|
||||
|
||||
# OPT-04. ⚠️ The threshold's lower bound is 0 and not negative: it is a COST,
|
||||
# and a negative one is a threshold every spread clears - arbitrage with the
|
||||
# brake removed. app/arbitrage.py refuses it in code too, because
|
||||
# validate_startup_config() is also sim/inject.py's entry point and can be
|
||||
# handed any dict at all.
|
||||
arbitrage_enabled: bool
|
||||
cycle_cost_threshold_eur: float(0,1)
|
||||
arbitrage_soc_floor: int(5,95)
|
||||
arbitrage_soc_ceiling: int(10,100)
|
||||
|
||||
estop_fitted: bool
|
||||
log_level: list(trace|debug|info|warning|error)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,551 @@
|
||||
#!/usr/bin/env python3
|
||||
"""DATA-02: fetch 12 months of EPEX BE day-ahead prices, in §9.3's shape.
|
||||
|
||||
python3 scenarios/fetch_epex_be.py --start 2025-09-01 --end 2026-09-01 \
|
||||
--out scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json
|
||||
|
||||
Run from `goodwe_controller/`. Rewrites the output file from scratch; the
|
||||
committed dataset is exactly what this produces, and `--verify` re-derives every
|
||||
claim the file makes about itself without going to the network.
|
||||
|
||||
|
||||
WHAT THIS IS NOT
|
||||
================
|
||||
NOT wired into the add-on, and it must never be. `price.py`'s `GenericDayAhead`
|
||||
is the live path and it has its own egress check; this is a one-shot archival
|
||||
fetch that produces a file for REPLAY, run by hand by a developer. Nothing under
|
||||
`app/` imports it.
|
||||
|
||||
|
||||
THE SOURCE, AND WHY THIS ONE
|
||||
============================
|
||||
`https://api.energy-charts.info/price?bzn=BE` - Fraunhofer ISE's Energy-Charts
|
||||
API. No key, no registration.
|
||||
|
||||
⚠️ THE LICENCE WAS CHECKED BEFORE ANY DATA WAS COMMITTED, because for this
|
||||
source it is NOT uniform across bidding zones. The API's own OpenAPI document
|
||||
(`https://api.energy-charts.info/openapi.json`, /price) splits the zones in two:
|
||||
|
||||
"The data for the following bidding zones is licensed as CC BY 4.0 from
|
||||
Bundesnetzagentur | SMARD.de and is published without changes:
|
||||
AT, BE (Belgium), CH, CZ, DE-LU, ..."
|
||||
|
||||
"The data for the other bidding zones is for private and internal use only.
|
||||
The utilization of any data, whether in its raw or derived form, for
|
||||
external or commercial purposes is expressly prohibited. Should you require
|
||||
licensing for market-related data, please direct your inquiries to the
|
||||
original data providers, including but not limited to EPEX SPOT SE."
|
||||
|
||||
BE is in the FIRST list, so redistribution is permitted with attribution and the
|
||||
dataset is committed rather than left as a fetch script and a checksum. Every
|
||||
response also carries the licence per-response in `license_info`, and this
|
||||
script REFUSES to write a file if that field ever stops saying CC BY 4.0 - a
|
||||
licence checked once by a human and assumed forever is a licence that changes
|
||||
silently. Had BE been in the second list, the correct output of this card would
|
||||
have been this script and a checksum and no data.
|
||||
|
||||
Attribution, carried in the dataset's own `provenance` block and reproduced
|
||||
here: data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts
|
||||
(Fraunhofer ISE), https://energy-charts.info, published without changes.
|
||||
|
||||
Cross-checked against the upstream on 2026-08-31: SMARD's own chart_data series
|
||||
`4996/DE` over 2026-04-10 matched this API's BE series on all 96 quarter-hours
|
||||
with a maximum absolute difference of 0.00 EUR/MWh. "Published without changes"
|
||||
is a claim worth checking once; `--verify` does not repeat it, because it is a
|
||||
claim about the SOURCE and not about the file.
|
||||
|
||||
|
||||
RAW WHOLESALE, AND THE ASSUMPTION THAT IS NOT BAKED IN
|
||||
======================================================
|
||||
⚠️ WHAT COMES BACK IS RAW WHOLESALE, EUR/MWh, DAY-AHEAD AUCTION CLEARING PRICE.
|
||||
It is NOT all-in: no supplier markup, no distribution or transmission tariff, no
|
||||
levies, no VAT. §9.3's canonical point is all-in, and TARIFF-02's whole trap is
|
||||
a figure that already contains a component being handed to something that adds
|
||||
it again.
|
||||
|
||||
So this file does NOT add one. `eur_per_kwh` in the output is the wholesale
|
||||
figure converted from EUR/MWh, `provenance.all_in` is `false` beside it, and the
|
||||
markup/tax assumption a consumer needs lives in `provenance.all_in_assumption`
|
||||
as NUMBERS TO APPLY rather than numbers already applied. test_arbitrage.py
|
||||
section 7 applies them where a reader can watch it happen.
|
||||
|
||||
The assumption's SHAPE is the part that is not arbitrary, and it decides which
|
||||
conclusions depend on the guessed numbers at all:
|
||||
|
||||
all_in(h) = (wholesale(h) + markup + network_and_levies) * (1 + vat)
|
||||
|
||||
* The two ADDITIVE terms cancel out of a spread: max - min is unchanged by
|
||||
them. So how often a day clears `cycle_cost_threshold_eur` does NOT depend
|
||||
on the markup guess - only on the VAT multiplier.
|
||||
* They do NOT cancel out of §17's negative-price guard, which compares ONE
|
||||
hour's all-in price against zero. A wholesale hour at -0.02 EUR/kWh is
|
||||
comfortably positive all-in once ~0.12 of fixed charges is added.
|
||||
|
||||
Both are asserted arithmetically in section 7 rather than believed, and the
|
||||
trade count is reported on the raw spread as well, so a reader can see exactly
|
||||
how much of the answer rests on the guess.
|
||||
|
||||
The numbers themselves (2026 Belgian residential dynamic contract, order of
|
||||
magnitude) are a STATED GUESS, labelled as one everywhere they appear. They are
|
||||
not a measurement and this card had nothing to measure them with.
|
||||
|
||||
|
||||
HOURS, QUARTER-HOURS, AND THE SEAM IN THE MIDDLE OF THE YEAR
|
||||
============================================================
|
||||
⚠️ THE SOURCE CHANGES RESOLUTION PART-WAY THROUGH ANY RECENT 12 MONTHS. EPEX
|
||||
day-ahead moved to a 15-minute market time unit in October 2025 and the feed
|
||||
follows: 2025-09-10 comes back as 24 hourly points, 2025-12-10 as 96
|
||||
quarter-hourly ones. An importer that assumed either one silently loses three
|
||||
quarters of the year, or reads a quarter as an hour.
|
||||
|
||||
The canonical format is hourly, so quarter-hours are aggregated: an hour's price
|
||||
is the unweighted mean of its four quarters. That is a STATED APPROXIMATION of
|
||||
the hourly product, not the hourly product, and it errs in the direction that
|
||||
matters here - averaging discards intra-hour variation, so the observed spread
|
||||
is a LOWER bound on what a 15-minute strategy would see.
|
||||
`provenance.resolution` records how many hours came from each source shape.
|
||||
|
||||
⚠️ AND A PARTIAL HOUR IS DROPPED, NOT AVERAGED OVER WHAT ARRIVED. An hour with
|
||||
three of four quarters is an hour nobody published a complete price for; the
|
||||
mean of three would be an invented number carrying no flag. Dropped hours are
|
||||
listed in `provenance.gaps` with their timestamps, because a missing hour in a
|
||||
feed is not a zero price, and a zero price is exactly the value that would make
|
||||
everything look free.
|
||||
|
||||
|
||||
SIZE: WHY THIS IS PLAIN JSON AND NOT A `.gz`, MEASURED
|
||||
======================================================
|
||||
The directory's existing captures are gzipped, so the default would have been to
|
||||
gzip this too. Measured on the committed file instead of assumed:
|
||||
|
||||
working tree, plain JSON 893 171 bytes
|
||||
what GIT actually stores for it (zlib, loose object) 74 590 bytes
|
||||
what git would store for a committed `.json.gz` 56 504 bytes
|
||||
|
||||
⚠️ GIT ALREADY COMPRESSES BLOBS, so gzipping buys 18 KB - 24 % of an object
|
||||
that is already 75 KB, in a repository whose three existing P1 captures cost
|
||||
27 KB, 136 KB and 140 KB. Eighteen kilobytes is not worth making the provenance
|
||||
block unreadable in a pull request, and the licence claim in that block is
|
||||
precisely the thing a reviewer of this card has to read and check. Plain JSON
|
||||
also greps, diffs line-by-line when a future refetch changes something, and is
|
||||
what `--verify` opens. `load()` still handles `.gz` for a future dataset large
|
||||
enough to want it; this one is not.
|
||||
|
||||
The P1 captures are gzipped for the opposite reason and it is not an
|
||||
inconsistency: they are hundreds of thousands of telegram lines that no human
|
||||
reads, and there is no metadata in them to review.
|
||||
|
||||
|
||||
TIME
|
||||
====
|
||||
Every instant is carried as UTC and rendered with `app.price.brussels()`, the
|
||||
module that already owns the EU DST rule without a tz database. There is no
|
||||
second answer to that question here and there must not be: the point of
|
||||
`brussels()` is that the 02:00-03:00 hour which happens twice in October is two
|
||||
instants, and a 24-slot local-hour importer collapses them.
|
||||
|
||||
So a year is 8760 hours only if it contains no transition, or contains both and
|
||||
they cancel - 25 in October, 23 in March. `--verify` therefore asserts the
|
||||
PER-DAY counts, not the total, because the total is exactly the number a broken
|
||||
importer also produces.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from app.price import brussels # noqa: E402
|
||||
|
||||
API = "https://api.energy-charts.info/price"
|
||||
BZN = "BE"
|
||||
LICENCE_MUST_CONTAIN = "CC BY 4.0"
|
||||
UTC = timezone.utc
|
||||
|
||||
ATTRIBUTION = ("Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, "
|
||||
"retrieved via the Energy-Charts API (Fraunhofer ISE), "
|
||||
"https://energy-charts.info. Published without changes.")
|
||||
|
||||
# ⚠️ A GUESS, LABELLED AS ONE, AND APPLIED BY THE CONSUMER - NOT HERE.
|
||||
# Order-of-magnitude 2026 Belgian residential dynamic contract. Nothing in this
|
||||
# repository measured any part of it.
|
||||
ALL_IN_ASSUMPTION = {
|
||||
"form": "all_in = (wholesale + markup + network_and_levies) * (1 + vat)",
|
||||
"markup_eur_per_kwh": 0.020,
|
||||
"network_and_levies_eur_per_kwh": 0.100,
|
||||
"vat_fraction": 0.06,
|
||||
"status": "ASSUMPTION, NOT MEASURED. Stated so a consumer applies it "
|
||||
"deliberately; the committed eur_per_kwh has none of it applied.",
|
||||
"note": "The two additive terms cancel out of any spread, so an arbitrage "
|
||||
"trade count depends on vat_fraction only. They do NOT cancel out "
|
||||
"of the negative-price guard (17), which compares one hour's "
|
||||
"all-in price against zero.",
|
||||
}
|
||||
|
||||
|
||||
PACE_S = 3.0 # between chunks, unconditionally
|
||||
RETRIES = 6
|
||||
|
||||
|
||||
def _get(url: str) -> dict:
|
||||
"""One GET, backing off on 429 and 5xx.
|
||||
|
||||
⚠️ THIS IS A FREE PUBLIC SERVICE AND IT RATE-LIMITS. The first run of this
|
||||
script was refused with 429 on its fifth chunk. Backing off is the polite
|
||||
behaviour; it is also the correct one, because the alternative - catching
|
||||
the error and carrying on - would produce a file that is silently missing a
|
||||
month and says nothing about it. A refused request is not an empty month.
|
||||
"""
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": "goodwe-addon/DATA-02 (one-shot archival fetch)"})
|
||||
wait = 5.0
|
||||
for attempt in range(1, RETRIES + 1):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
return json.loads(r.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in (429, 500, 502, 503, 504) or attempt == RETRIES:
|
||||
raise
|
||||
print(f" HTTP {e.code}; backing off {wait:.0f}s "
|
||||
f"(attempt {attempt}/{RETRIES})", file=sys.stderr)
|
||||
time.sleep(wait)
|
||||
wait *= 2
|
||||
raise SystemExit(f"unreachable: {url}")
|
||||
|
||||
|
||||
def local_midnight_utc(day: str) -> datetime:
|
||||
"""The INSTANT at which Brussels midnight begins on `day` (YYYY-MM-DD).
|
||||
|
||||
⚠️ THE WINDOW IS LOCAL, NOT UTC, AND THE FIRST RUN OF THIS SCRIPT PROVED WHY.
|
||||
Bounding it at 00:00 UTC starts the year at 02:00 CEST, which makes the first
|
||||
Brussels day 22 hours long - and 22 is not one of the two lengths a
|
||||
transition produces, so `report()` refused the file. A dataset whose first
|
||||
day is short is a dataset whose "days per year" arithmetic is quietly wrong
|
||||
everywhere downstream.
|
||||
|
||||
Derived from `brussels()` by trying the only two offsets Belgium has and
|
||||
keeping the one that round-trips, rather than by restating the EU rule here.
|
||||
"""
|
||||
for hours in (1, 2):
|
||||
guess = datetime.fromisoformat(f"{day}T00:00:00").replace(
|
||||
tzinfo=timezone(timedelta(hours=hours))).astimezone(UTC)
|
||||
if brussels(guess).isoformat() == f"{day}T00:00:00+0{hours}:00":
|
||||
return guess
|
||||
raise SystemExit(f"no Brussels midnight for {day!r} - brussels() disagrees "
|
||||
f"with itself, which is a bug in price.py, not here.")
|
||||
|
||||
|
||||
def _months(start: datetime, end: datetime):
|
||||
"""[start, end) in month-sized chunks, so a bad chunk names its own window."""
|
||||
a = start
|
||||
while a < end:
|
||||
b = min(end, (a.replace(day=1) + timedelta(days=32)).replace(day=1))
|
||||
yield a, b
|
||||
a = b
|
||||
|
||||
|
||||
def fetch(start: datetime, end: datetime):
|
||||
"""Raw {unix_second: EUR/MWh} over [start, end), deduplicated.
|
||||
|
||||
⚠️ A `null` price is DROPPED, never coerced. The API returns null for an
|
||||
interval it holds no result for, and `or 0.0` is the version of this line
|
||||
that ships a year of free electricity.
|
||||
"""
|
||||
seen, licences, nulls = {}, Counter(), 0
|
||||
for a, b in _months(start, end):
|
||||
# ⚠️ THE API'S `end` IS AN INCLUSIVE LOCAL DAY, AND THE WINDOW'S IS AN
|
||||
# EXCLUSIVE INSTANT. Ask for the day containing the LAST hour of the
|
||||
# chunk, derived from the instant rather than by subtracting a calendar
|
||||
# day from the bound: `b - 1 day` drops the final local day whenever the
|
||||
# window ends at local midnight, which is exactly how this window ends.
|
||||
# The first run lost 2026-08-31 that way, silently - 8736 points instead
|
||||
# of 8760 - and the points-plus-gaps accounting in report() is what
|
||||
# noticed. A day nobody asked for is not a day with no prices.
|
||||
first = brussels(a).date().isoformat()
|
||||
last = brussels(b - timedelta(hours=1)).date().isoformat()
|
||||
url = f"{API}?bzn={BZN}&start={first}&end={last}"
|
||||
d = _get(url)
|
||||
lic = str(d.get("license_info", ""))
|
||||
licences[lic] += 1
|
||||
if LICENCE_MUST_CONTAIN not in lic:
|
||||
raise SystemExit(
|
||||
f"REFUSING TO WRITE. {url} came back under licence {lic!r}, "
|
||||
f"which does not say {LICENCE_MUST_CONTAIN!r}. The BE zone was "
|
||||
f"redistributable when this card ran; if that has changed, the "
|
||||
f"right output is this script and a checksum, not the data.")
|
||||
unit = str(d.get("unit", ""))
|
||||
if "MWh" not in unit:
|
||||
raise SystemExit(
|
||||
f"REFUSING TO WRITE. {url} reports unit {unit!r}, not EUR/MWh. "
|
||||
f"A silent unit change is a factor of 1000 on every price.")
|
||||
for t, p in zip(d["unix_seconds"], d["price"]):
|
||||
if p is None:
|
||||
nulls += 1
|
||||
continue
|
||||
if start.timestamp() <= t < end.timestamp():
|
||||
seen[int(t)] = float(p)
|
||||
print(f" {a.date()} .. {b.date()} {len(d['price']):5d} intervals",
|
||||
file=sys.stderr)
|
||||
time.sleep(PACE_S)
|
||||
return seen, licences, nulls
|
||||
|
||||
|
||||
def to_hourly(raw: dict):
|
||||
"""Hourly means keyed by UTC hour. Returns (hours, gaps, shape_counts).
|
||||
|
||||
An hour is emitted only when every interval the source used for it arrived:
|
||||
1 of 1 where the source was hourly, 4 of 4 where it was quarter-hourly.
|
||||
Anything else is a gap and is named as one.
|
||||
"""
|
||||
buckets = defaultdict(dict)
|
||||
for t, p in raw.items():
|
||||
buckets[t - (t % 3600)][t % 3600] = p
|
||||
|
||||
hours, gaps, shapes = {}, [], Counter()
|
||||
for h in sorted(buckets):
|
||||
got = buckets[h]
|
||||
if set(got) == {0}:
|
||||
shapes["hourly"] += 1
|
||||
hours[h] = got[0]
|
||||
elif set(got) == {0, 900, 1800, 2700}:
|
||||
shapes["quarter_hourly_mean"] += 1
|
||||
hours[h] = sum(got.values()) / 4.0
|
||||
else:
|
||||
gaps.append({
|
||||
"hour_utc": datetime.fromtimestamp(h, UTC).isoformat(),
|
||||
"intervals_present": sorted(got),
|
||||
"why": "incomplete hour - dropped rather than averaged over "
|
||||
"what arrived; a partial read is not an observation",
|
||||
})
|
||||
return hours, gaps, shapes
|
||||
|
||||
|
||||
def canonical(hours: dict) -> list:
|
||||
"""9.3 points. THREE FIELDS, rendered at the Brussels offset for the instant."""
|
||||
return [{"start": brussels(datetime.fromtimestamp(h, UTC)).isoformat(),
|
||||
"eur_per_kwh": round(hours[h] / 1000.0, 6) + 0.0,
|
||||
"provisional": False}
|
||||
for h in sorted(hours)]
|
||||
|
||||
|
||||
def expected_hours(start: datetime, end: datetime) -> int:
|
||||
"""Whole hours in the window. Derived, rather than 8760 typed in."""
|
||||
return int((end - start).total_seconds() // 3600)
|
||||
|
||||
|
||||
def per_local_day(points) -> Counter:
|
||||
"""Points per BRUSSELS calendar day - the count that exposes a DST bug."""
|
||||
n = Counter()
|
||||
for p in points:
|
||||
n[p["start"][:10]] += 1
|
||||
return n
|
||||
|
||||
|
||||
def digest(points) -> str:
|
||||
"""A checksum over the POINTS, so reformatting the envelope does not move it."""
|
||||
h = hashlib.sha256()
|
||||
for p in points:
|
||||
h.update(f"{p['start']}|{p['eur_per_kwh']:.6f}|{int(p['provisional'])}\n"
|
||||
.encode())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def load(path: str) -> dict:
|
||||
op = gzip.open if path.endswith(".gz") else open
|
||||
with op(path, "rt", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def report(doc, *, start=None, end=None) -> int:
|
||||
"""Re-derive every claim the file makes about itself. Returns failures."""
|
||||
pts = doc["prices"]
|
||||
prov = doc["provenance"]
|
||||
bad = 0
|
||||
|
||||
def ck(name, cond):
|
||||
nonlocal bad
|
||||
print(f" {'ok ' if cond else 'FAIL'} {name}")
|
||||
if not cond:
|
||||
bad += 1
|
||||
|
||||
ck(f"the checksum over the {len(pts)} points matches the recorded one "
|
||||
f"({prov['sha256_of_points'][:16]}...)",
|
||||
digest(pts) == prov["sha256_of_points"])
|
||||
ck("every point has exactly the three canonical fields and nothing else",
|
||||
all(set(p) == {"start", "eur_per_kwh", "provisional"} for p in pts))
|
||||
ck("every start carries a UTC offset, and it is +01:00 or +02:00 - a naive "
|
||||
"timestamp here is a wall clock plus an assumption",
|
||||
bool(pts) and all(p["start"][-6:] in ("+01:00", "+02:00") for p in pts))
|
||||
ck("the series is strictly increasing in time, so no instant is duplicated",
|
||||
all(datetime.fromisoformat(a["start"]) < datetime.fromisoformat(b["start"])
|
||||
for a, b in zip(pts, pts[1:])))
|
||||
ck("consecutive points are exactly one hour apart AS INSTANTS, so the two "
|
||||
"02:00s of the October day are two hours rather than one",
|
||||
all((datetime.fromisoformat(b["start"])
|
||||
- datetime.fromisoformat(a["start"])) == timedelta(hours=1)
|
||||
for a, b in zip(pts, pts[1:])))
|
||||
|
||||
days = per_local_day(pts)
|
||||
odd = {d: n for d, n in days.items() if n != 24}
|
||||
ck(f"exactly two Brussels days are not 24 hours long, and they are the "
|
||||
f"transitions: {sorted(odd.items())}",
|
||||
sorted(odd.values()) == [23, 25] and len(odd) == 2)
|
||||
spring = [d for d, n in odd.items() if n == 23]
|
||||
autumn = [d for d, n in odd.items() if n == 25]
|
||||
ck(f"the 23-hour day {spring} is in March and the 25-hour day {autumn} is "
|
||||
f"in October, which is the direction the EU rule actually runs",
|
||||
len(spring) == 1 and len(autumn) == 1
|
||||
and spring[0][5:7] == "03" and autumn[0][5:7] == "10")
|
||||
if autumn:
|
||||
twice = [p["start"] for p in pts
|
||||
if p["start"].startswith(autumn[0] + "T02")]
|
||||
ck(f"...and 02:00 on {autumn[0]} occurs TWICE at two different offsets "
|
||||
f"{twice} - exactly the case a 24-slot local-hour array collapses",
|
||||
len(twice) == 2 and twice[0][-6:] == "+02:00"
|
||||
and twice[1][-6:] == "+01:00")
|
||||
ck(f"...and the two carry DIFFERENT prices, so the 25th hour is a real "
|
||||
f"observation and not the same one written down twice",
|
||||
len({p["eur_per_kwh"] for p in pts
|
||||
if p["start"].startswith(autumn[0] + "T02")}) == 2)
|
||||
if spring:
|
||||
ck(f"...and 02:00 on {spring[0]} does not occur at all, because that "
|
||||
f"wall clock reading never happened in Belgium",
|
||||
not any(p["start"].startswith(spring[0] + "T02") for p in pts))
|
||||
|
||||
if start and end:
|
||||
want = expected_hours(start, end)
|
||||
ck(f"{len(pts)} points for a window of {want} whole hours, and the "
|
||||
f"{len(prov['gaps'])} recorded gaps account for the difference "
|
||||
f"exactly", len(pts) + len(prov["gaps"]) == want)
|
||||
|
||||
# ⚠️ A ZERO IS NOT AUTOMATICALLY A DEFECT HERE, AND SAYING IT WAS WOULD BE
|
||||
# THE WRONG LESSON. The Belgian market really does clear at 0.00 EUR/MWh in
|
||||
# solar-glut hours; this window contains such hours. What must be impossible
|
||||
# is a zero that arrived by COERCION - a dropped read turned into a free
|
||||
# hour. That is excluded structurally rather than by outlawing the value:
|
||||
# nulls are dropped, incomplete hours become gaps, and the accounting above
|
||||
# shows points + gaps covering every hour of the window, so every emitted
|
||||
# point stands for intervals the source actually published.
|
||||
zeros = [p["start"] for p in pts if p["eur_per_kwh"] == 0.0]
|
||||
ck(f"the {len(zeros)} hours priced at exactly zero match the count recorded "
|
||||
f"at build time, so no zero appeared between the fetch and the file",
|
||||
len(zeros) == prov["exact_zero_hours"])
|
||||
ck("...and none of them is a coerced read: every emitted hour is backed by "
|
||||
"published intervals, which is what the points-plus-gaps accounting "
|
||||
"above establishes",
|
||||
len(prov["gaps"]) == 0 or all("intervals_present" in g
|
||||
for g in prov["gaps"]))
|
||||
ck("the figures are declared RAW WHOLESALE, and the all-in assumption is "
|
||||
"recorded as an assumption rather than applied",
|
||||
prov["all_in"] is False
|
||||
and "ASSUMPTION" in prov["all_in_assumption"]["status"])
|
||||
return bad
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--start", default="2025-09-01")
|
||||
ap.add_argument("--end", default="2026-09-01",
|
||||
help="exclusive; both bounds are BRUSSELS-local midnights")
|
||||
ap.add_argument("--out", default=None)
|
||||
ap.add_argument("--verify", default=None,
|
||||
help="re-derive the claims of an existing file; no network")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.verify:
|
||||
doc = load(a.verify)
|
||||
print(f"verifying {a.verify}")
|
||||
s = datetime.fromisoformat(doc["provenance"]["window_utc"]["start"])
|
||||
e = datetime.fromisoformat(doc["provenance"]["window_utc"]["end"])
|
||||
bad = report(doc, start=s, end=e)
|
||||
print("OK" if not bad else f"{bad} FAILED")
|
||||
return 1 if bad else 0
|
||||
|
||||
start = local_midnight_utc(a.start)
|
||||
end = local_midnight_utc(a.end)
|
||||
out = a.out or f"scenarios/epex-be-dayahead-{a.start}_{a.end}.json"
|
||||
|
||||
print(f"fetching {BZN} day-ahead {start.date()} .. {end.date()}",
|
||||
file=sys.stderr)
|
||||
raw, licences, nulls = fetch(start, end)
|
||||
hours, gaps, shapes = to_hourly(raw)
|
||||
pts = canonical(hours)
|
||||
|
||||
doc = {
|
||||
"provenance": {
|
||||
"card": "DATA-02",
|
||||
"what": "EPEX BE day-ahead auction clearing prices, hourly",
|
||||
"source": f"{API}?bzn={BZN} (Energy-Charts API, Fraunhofer ISE)",
|
||||
"upstream": "Bundesnetzagentur | SMARD.de",
|
||||
"licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
|
||||
"licence_checked": "before committing anything; the API's OpenAPI "
|
||||
"document lists BE among the CC BY 4.0 zones and "
|
||||
"the remaining zones as private/internal use "
|
||||
"only, so redistribution of THIS zone is "
|
||||
"permitted with attribution",
|
||||
"licence_as_returned": sorted(licences),
|
||||
"attribution": ATTRIBUTION,
|
||||
"cross_check": "SMARD chart_data 4996/DE over 2026-04-10 matched "
|
||||
"this feed on all 96 quarter-hours, max abs "
|
||||
"difference 0.00 EUR/MWh (2026-08-31)",
|
||||
"retrieved_utc": datetime.now(UTC).replace(microsecond=0).isoformat(),
|
||||
"fetched_by": "scenarios/fetch_epex_be.py",
|
||||
"window_utc": {"start": start.isoformat(), "end": end.isoformat()},
|
||||
"unit": "EUR per kWh (source publishes EUR/MWh; divided by 1000)",
|
||||
"all_in": False,
|
||||
"all_in_note": "RAW WHOLESALE. No supplier markup, no network or "
|
||||
"transmission tariff, no levies, no VAT. 9.3's "
|
||||
"canonical point is all-in and this is NOT; the "
|
||||
"consumer applies the assumption below, visibly.",
|
||||
"all_in_assumption": ALL_IN_ASSUMPTION,
|
||||
"resolution": {
|
||||
"note": "EPEX day-ahead moved to a 15-minute market time unit "
|
||||
"in Oct 2025, part-way through this window.",
|
||||
"hours_from_hourly_source": shapes["hourly"],
|
||||
"hours_from_quarter_hourly_mean": shapes["quarter_hourly_mean"],
|
||||
"aggregation": "unweighted mean of the four quarters; a stated "
|
||||
"approximation of the hourly product, and a "
|
||||
"LOWER bound on the spread a 15-minute strategy "
|
||||
"would have seen",
|
||||
},
|
||||
"timezone": "starts rendered at the Brussels offset by "
|
||||
"app.price.brussels(); instants carried in UTC, so the "
|
||||
"transition days come out 23 and 25 points by "
|
||||
"construction rather than by a special case",
|
||||
"nulls_dropped": nulls,
|
||||
"gaps": gaps,
|
||||
"gaps_note": "hours the source did not publish completely. Dropped, "
|
||||
"never zero-filled: a missing hour is not a free hour.",
|
||||
"points": len(pts),
|
||||
"exact_zero_hours": sum(1 for p in pts if p["eur_per_kwh"] == 0.0),
|
||||
"negative_hours": sum(1 for p in pts if p["eur_per_kwh"] < 0),
|
||||
"zero_note": "REAL CLEARINGS AT 0.00 EUR/MWh, not dropped reads. "
|
||||
"Nulls are dropped and incomplete hours become gaps, "
|
||||
"so points + gaps == hours in the window and no point "
|
||||
"stands in for something that failed to arrive.",
|
||||
"sha256_of_points": digest(pts),
|
||||
},
|
||||
"prices": pts,
|
||||
}
|
||||
|
||||
with open(out, "w", encoding="utf-8", newline="\n") as f:
|
||||
json.dump(doc, f, indent=1, ensure_ascii=False)
|
||||
f.write("\n")
|
||||
print(f"wrote {out} {len(pts)} points {os.path.getsize(out)} bytes",
|
||||
file=sys.stderr)
|
||||
print(f"verifying {out}")
|
||||
return 1 if report(doc, start=start, end=end) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,167 @@
|
||||
# 30 days of P1 net power — 2026-08-01 … 2026-08-30
|
||||
|
||||
`sensor.p1_meter_active_power` out of Home Assistant's recorder, one gzipped
|
||||
JSONL file per **local** day (Europe/Brussels), same schema as the two files one
|
||||
directory up:
|
||||
|
||||
{"t": "2026-08-22T22:00:00+00:00", "w": -25.0}
|
||||
|
||||
**485,841 samples, 30/30 days read, 0 days failed, 4.13 MB gzipped.**
|
||||
Produced by `tools/export_p1_history.py`; `manifest.json` here carries the
|
||||
per-day counts, sha256 of every file, and every gap.
|
||||
|
||||
python3 tools/export_p1_history.py --days 30 --end-date 2026-08-30 \
|
||||
--out-dir scenarios/p1-net-30d-2026-08
|
||||
|
||||
## Why this exists
|
||||
|
||||
[[TARIFF-01]] AC 3 and [[SAFETY-07]] AC 3 both ask for *"monthly peak held under
|
||||
the soft limit in >= 95 % of intervals over a 30-day simulation replay"*, and
|
||||
both were merged with the criterion honestly unmet because only two days of
|
||||
capture existed. `test_tariff.py`'s docstring still recorded the reason as
|
||||
"nothing longer was ever captured" — which was true about the past and was
|
||||
quietly read as a statement about the future. It was not one. Probed on
|
||||
2026-08-31, HA's recorder answered a 20-minute window with 230 samples at 30
|
||||
days back and 0 at 45, so the whole month was still there and was exported the
|
||||
same day. **The window rolls: 2026-08-01 leaves the recorder around 2026-09-15.**
|
||||
|
||||
## The size decision — COMMITTED, deliberately, at 4.13 MB
|
||||
|
||||
Three options were on the table. This one was taken and the cost is stated
|
||||
rather than absorbed, and it is measured rather than estimated - a
|
||||
`git bundle` of the whole history before and after this commit:
|
||||
|
||||
release/1.0 @ 86c43eb 1.58 MB
|
||||
DATA-01 5.28 MB +3.70 MB, x3.3
|
||||
|
||||
A full clone of this repository therefore more than triples, and a Home
|
||||
Assistant supervisor clones it on every add-on install and update. That is the
|
||||
price; the paragraphs below are why it was judged worth paying rather than
|
||||
avoided by making the data worse or making it disappear.
|
||||
|
||||
* **Downsampling is not available.** It is not a matter of taste: the shipped
|
||||
`meter_max_age_s` is **30 s**, which is the age past which the controller's
|
||||
own ingest stops counting a held reading as coverage. Any resample coarser
|
||||
than 30 s therefore converts real coverage into modelled meter-gap and changes
|
||||
the very number AC 3 measures. Resampling at or under 30 s saves almost
|
||||
nothing against a 5 s cadence.
|
||||
* **"Outside the repo with a pointer" is what already happened, and it is why
|
||||
this card exists.** The two committed days came from the infra repo's private
|
||||
`sim/scenarios/`, the one-off export script was thrown away, and three cards
|
||||
then deferred an acceptance criterion on the belief that the data could not be
|
||||
produced again. `test_tariff.py` states the general rule verbatim: *a
|
||||
regression fixture CI cannot see is not a regression fixture.* CI's token is
|
||||
minted for this repository only.
|
||||
* **So: committed** — but in a SUBDIRECTORY, which is load-bearing. Both
|
||||
`test_tariff.py` part 9 and `test_peakguard.py` part 6 do
|
||||
`(HERE / "scenarios").glob("p1-net-*.jsonl.gz")`, which does not recurse. Drop
|
||||
these thirty files beside the two and CI grows sixty full-day
|
||||
replays - about 50 s of pure compute - on every PR forever, `test_tariff.py`'s
|
||||
`check(... replay_days < 30)` starts failing, and both files' honesty notes
|
||||
become false. In `p1-net-30d-2026-08/` they are visible to CI, invisible to
|
||||
the globs, and the two committed days keep their job as the small fixtures the
|
||||
suites grade against.
|
||||
|
||||
They are also written once and never edited, so git stores one already-deflated
|
||||
blob per day with no delta churn.
|
||||
|
||||
## Gaps are visible, never interpolated
|
||||
|
||||
Nothing is filled, carried forward or resampled. A hole in the recorder is a
|
||||
hole here — consecutive `t` values further apart than the cadence — which is
|
||||
exactly what the controller's freshness rule already reads as *no coverage*, so
|
||||
a missing hour replays as a missing hour rather than as an hour of whatever the
|
||||
last sample said.
|
||||
|
||||
**13 gaps over 30 s, 1291 s in total: 99.950 % of the month is covered.**
|
||||
All of them are listed in `manifest.json` and by
|
||||
`python3 tools/replay_ac3.py <dir> --report-gaps`. The longest is 263 s on
|
||||
2026-08-11.
|
||||
|
||||
Three samples in the month carried the state `unavailable` rather than a number
|
||||
(2026-08-18 02:03:54Z and 04:10:41Z, 2026-08-25 01:52:02Z). There is no float to
|
||||
write and the committed schema has no slot for a non-number, so they are absent
|
||||
from the series and recorded individually in `manifest.json` under
|
||||
`non_numeric_states`. Each one sits inside a gap that is already listed, which
|
||||
is the honest rendering: the meter was not reporting a power then.
|
||||
|
||||
**A failed read is never an empty hour.** If a day's read fails, the exporter
|
||||
writes NO FILE, records `"read": "FAILED"` with the error, and exits non-zero —
|
||||
so a missing day is unreadable-and-known rather than silently zero. The entity
|
||||
is also proved to exist through `/api/states` before any day is trusted, because
|
||||
a typo would otherwise answer `200 []` thirty times and produce a beautifully
|
||||
consistent dataset of nothing. On this run: 30 read, 0 failed, 0 empty.
|
||||
|
||||
## AC 3, measured
|
||||
|
||||
cd goodwe_controller
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
485841 samples, 2879 closed quarters (30.0 days' worth)
|
||||
month 2026-08: peak 5029.8 W uncontrolled -> 4107.3 W guarded
|
||||
|
||||
TARIFF-01 AC 3 (UNCONTROLLED): 2847/2879 = 98.89 % of quarters at or under 3500 W
|
||||
SAFETY-07 AC 3 (GUARDED): 2850/2879 = 98.99 %
|
||||
guard cycles 143810, engaged on the measurement 1542, blind 195,
|
||||
saturated 0, modelled discharge 8060 Wh
|
||||
|
||||
2879 rather than 2880 quarters: the last quarter of 2026-08-30 closes only when
|
||||
a telegram past its boundary arrives, and that telegram is 2026-08-31's.
|
||||
|
||||
### What the numbers actually say — three findings, not a pass
|
||||
|
||||
1. **AC 3 is met on the ">= 95 % of intervals" reading, and the criterion turns
|
||||
out to be nearly insensitive.** The UNCONTROLLED month — no controller, no
|
||||
peak shaving, nothing running — already scores 98.89 %. A criterion that a
|
||||
house passes by doing nothing does not discriminate between a working guard
|
||||
and an absent one. That is a finding about the acceptance criterion, and it
|
||||
is reported rather than filed as a win.
|
||||
2. **On the other reading of the same sentence, AC 3 is NOT met.** "Monthly peak
|
||||
held under the soft limit" taken as *the peak itself* fails: the guard pulls
|
||||
August's peak from **5029.8 W to 4107.3 W** (-922.5 W, -18.3 %), which is
|
||||
real money on a Belgian capacity tariff, but 4107.3 W is still over the
|
||||
3500 W soft limit. Both readings are printed by the tool; neither is hidden.
|
||||
3. **The guard stops defending the soft limit once the month's peak is set, and
|
||||
that is by design.** `quarter_ceiling_w` uses `CEILING_MONTH_PEAK`: the
|
||||
billed quantity is the month's maximum, so a quarter under an
|
||||
already-recorded peak costs nothing extra and the guard does not spend the
|
||||
battery on it. The month's 32 over-cap quarters fall on four nights
|
||||
(2026-08-17, -18, -20, -21); **31 of them are pulled down** — 7494 W of
|
||||
quarter-average removed in total, 8060 Wh of modelled discharge — but only
|
||||
**3 land under 3500 W**, because by then the ceiling being defended is the
|
||||
running month peak (~4107 W), not the soft limit. The single untouched
|
||||
quarter, 2026-08-18T08:15Z at 4039.7 W, is the same rule seen from the other
|
||||
side: the running peak was already 4833 W, so 4039.7 W added nothing to the
|
||||
bill and the guard correctly stayed out. AC 3's wording asks the guard for
|
||||
something its own tariff logic deliberately does not optimise.
|
||||
|
||||
Also worth an eye: **195 blind engagements** over 143,810 cycles — the guard
|
||||
acting on a projection it could not fully cover. On the two committed days that
|
||||
count was 5. It scales with the gaps above.
|
||||
|
||||
## Verification of the exporter
|
||||
|
||||
`tools/export_p1_history.py --verify` re-fetches a committed day by the exact
|
||||
route used here and diffs it against the file:
|
||||
|
||||
python3 tools/export_p1_history.py \
|
||||
--verify scenarios/p1-net-2026-08-20.jsonl.gz \
|
||||
scenarios/p1-net-2026-08-23.jsonl.gz
|
||||
|
||||
Both days come back **byte-identical after decompression** — 865,024 B /
|
||||
15,600 samples and 894,387 B / 16,351 samples — and the copies of those two days
|
||||
inside this directory are byte-identical to the committed ones too. The gzip
|
||||
CONTAINERS differ by design: a gzip header carries an mtime and a stored
|
||||
filename, and this exporter writes mtime 0 so its own output is reproducible,
|
||||
where the 2026-08-25 one-off wrote a wall-clock mtime. Content equality is the
|
||||
claim; container equality would only assert that two files were written at the
|
||||
same second.
|
||||
|
||||
`tools/replay_ac3.py` is checked the same way rather than trusted: pointed at
|
||||
`scenarios/` it must reproduce `test_peakguard.py` part 6's published totals,
|
||||
and `--expect` makes it a hard check.
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
|
||||
reproduces 189/190 quarters, the 5029.8 W billed quarter, the 3485.3 W guarded
|
||||
result and the 878 Wh of modelled discharge that part 6 prints.
|
||||
@@ -0,0 +1,767 @@
|
||||
{
|
||||
"generated": "2026-08-31T08:42:40.614548+00:00",
|
||||
"generator": "tools/export_p1_history.py",
|
||||
"source": "http://192.168.2.158:8123/api/history/period",
|
||||
"entity": "sensor.p1_meter_active_power",
|
||||
"timezone": "Europe/Brussels",
|
||||
"gap_threshold_s": 30.0,
|
||||
"format": "{\"t\": <ISO-8601 UTC>, \"w\": <float watts>} per line, gzip",
|
||||
"days": [
|
||||
{
|
||||
"date": "2026-08-01",
|
||||
"window": [
|
||||
"2026-08-01T00:00:00+02:00",
|
||||
"2026-08-02T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1128060,
|
||||
"attempts": 1,
|
||||
"samples": 16511,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-07-31T22:00:00+00:00",
|
||||
"last": "2026-08-01T21:59:56.268499+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.3,
|
||||
"mean_cadence_s": 5.233,
|
||||
"file": "p1-net-2026-08-01.jsonl.gz",
|
||||
"bytes_gz": 140531,
|
||||
"bytes_raw": 913203,
|
||||
"sha256_raw": "120de2a0b7eb552281dac4bc1f8fd76d947ba72d1f5b426d9dcdd85ec103bd06",
|
||||
"sha256_gz": "e09bdee349dd3668a073b7fd77974e639c269ec37e73e81a9706cadde083f064"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-02",
|
||||
"window": [
|
||||
"2026-08-02T00:00:00+02:00",
|
||||
"2026-08-03T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1115387,
|
||||
"attempts": 1,
|
||||
"samples": 16303,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-01T22:00:00+00:00",
|
||||
"last": "2026-08-02T21:59:55.191360+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.2,
|
||||
"mean_cadence_s": 5.3,
|
||||
"file": "p1-net-2026-08-02.jsonl.gz",
|
||||
"bytes_gz": 141404,
|
||||
"bytes_raw": 903234,
|
||||
"sha256_raw": "9ab4891c0abf9d621fa528848fc3a874687843804768e8aeab6a88f7bb04b679",
|
||||
"sha256_gz": "94774139b3e19d04bcda3ac5ea988dd4d645e99004f90b917c6b0bf3fbac21cb"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-03",
|
||||
"window": [
|
||||
"2026-08-03T00:00:00+02:00",
|
||||
"2026-08-04T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1123668,
|
||||
"attempts": 1,
|
||||
"samples": 16424,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-02T22:00:00+00:00",
|
||||
"last": "2026-08-03T21:59:55.255808+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.3,
|
||||
"mean_cadence_s": 5.261,
|
||||
"file": "p1-net-2026-08-03.jsonl.gz",
|
||||
"bytes_gz": 144202,
|
||||
"bytes_raw": 909942,
|
||||
"sha256_raw": "13f8dd904a31b5b67fcfde04f6afc49c34c3a9504617278d79d87516fcd6441f",
|
||||
"sha256_gz": "e1bc0cb0e271cbc8d896aee97aa0f5c18e49e78905d1aaf137904a768047c7bc"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-04",
|
||||
"window": [
|
||||
"2026-08-04T00:00:00+02:00",
|
||||
"2026-08-05T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1108582,
|
||||
"attempts": 1,
|
||||
"samples": 16286,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-03T22:00:00+00:00",
|
||||
"last": "2026-08-04T21:59:58.729138+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.7,
|
||||
"mean_cadence_s": 5.305,
|
||||
"file": "p1-net-2026-08-04.jsonl.gz",
|
||||
"bytes_gz": 136505,
|
||||
"bytes_raw": 896650,
|
||||
"sha256_raw": "a65cd9d1ceda61e5a40043edb401c2154feea2b36bd3221710df808d778eabce",
|
||||
"sha256_gz": "c19b19ac890f4caad4140cb27d2fe6f3e972545f664127e5d9f457e711ad8c88"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-05",
|
||||
"window": [
|
||||
"2026-08-05T00:00:00+02:00",
|
||||
"2026-08-06T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1136556,
|
||||
"attempts": 1,
|
||||
"samples": 16655,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-04T22:00:00+00:00",
|
||||
"last": "2026-08-05T21:59:56.189152+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.2,
|
||||
"mean_cadence_s": 5.188,
|
||||
"file": "p1-net-2026-08-05.jsonl.gz",
|
||||
"bytes_gz": 142561,
|
||||
"bytes_raw": 919827,
|
||||
"sha256_raw": "dd327685c6f322bfaa49bc5bf228301e790e5a96fd7c23c487b845e44a962db4",
|
||||
"sha256_gz": "5ff7a1336c49ba38b8058820181dd8ed9c2b8db41b5a3a52412a5a3d8c515332"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-06",
|
||||
"window": [
|
||||
"2026-08-06T00:00:00+02:00",
|
||||
"2026-08-07T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124419,
|
||||
"attempts": 1,
|
||||
"samples": 16412,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-05T22:00:00+00:00",
|
||||
"last": "2026-08-06T21:59:57.192711+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.2,
|
||||
"mean_cadence_s": 5.265,
|
||||
"file": "p1-net-2026-08-06.jsonl.gz",
|
||||
"bytes_gz": 142313,
|
||||
"bytes_raw": 910849,
|
||||
"sha256_raw": "7cd1fe8315e24dc6391693c49d6306b0c0cf9b6c3a478e457da921001cea930c",
|
||||
"sha256_gz": "9108f169d645b9da07d4ae310666a1365ea07275ff311cd60adad1b702edbf11"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-07",
|
||||
"window": [
|
||||
"2026-08-07T00:00:00+02:00",
|
||||
"2026-08-08T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1087531,
|
||||
"attempts": 1,
|
||||
"samples": 15952,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-06T22:00:00+00:00",
|
||||
"last": "2026-08-07T21:59:58.195053+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.2,
|
||||
"mean_cadence_s": 5.416,
|
||||
"file": "p1-net-2026-08-07.jsonl.gz",
|
||||
"bytes_gz": 135979,
|
||||
"bytes_raw": 879941,
|
||||
"sha256_raw": "e0f67a9214980e2fd4893208cb402165065dadce2105ce1581c6b1d49952f39a",
|
||||
"sha256_gz": "b70b84f3743d1d84055c911bcd5589cb8f0371e8916bde741308c5bb50702c29"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-08",
|
||||
"window": [
|
||||
"2026-08-08T00:00:00+02:00",
|
||||
"2026-08-09T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1058569,
|
||||
"attempts": 1,
|
||||
"samples": 15468,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-07T22:00:00+00:00",
|
||||
"last": "2026-08-08T21:59:52.805887+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86392.8,
|
||||
"mean_cadence_s": 5.586,
|
||||
"file": "p1-net-2026-08-08.jsonl.gz",
|
||||
"bytes_gz": 134025,
|
||||
"bytes_raw": 857271,
|
||||
"sha256_raw": "0be14ca7593b4c7d68570115cc26e24f6a26ab87fe3c168a8b942b167e8d4f68",
|
||||
"sha256_gz": "d93dcc8ca3eae03af13b9ae36acd5ab1eaa1e7ee93c237d386e8ded1c61f5906"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-09",
|
||||
"window": [
|
||||
"2026-08-09T00:00:00+02:00",
|
||||
"2026-08-10T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1059349,
|
||||
"attempts": 1,
|
||||
"samples": 15508,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-08T22:00:00+00:00",
|
||||
"last": "2026-08-09T21:59:58.178998+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.2,
|
||||
"mean_cadence_s": 5.572,
|
||||
"file": "p1-net-2026-08-09.jsonl.gz",
|
||||
"bytes_gz": 135697,
|
||||
"bytes_raw": 857531,
|
||||
"sha256_raw": "f4793799173c502fcee5f78e2d9b0dcc984b5d2d8bd2fc0db0cd0f8d403733b3",
|
||||
"sha256_gz": "2fea6c5c8b1452d72ad5aa3b0ae9755bb2ea08071514c08b1a3b1155ef457cf1"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-10",
|
||||
"window": [
|
||||
"2026-08-10T00:00:00+02:00",
|
||||
"2026-08-11T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1066721,
|
||||
"attempts": 1,
|
||||
"samples": 15594,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-09T22:00:00+00:00",
|
||||
"last": "2026-08-10T21:59:59.170056+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86399.2,
|
||||
"mean_cadence_s": 5.541,
|
||||
"file": "p1-net-2026-08-10.jsonl.gz",
|
||||
"bytes_gz": 136682,
|
||||
"bytes_raw": 863785,
|
||||
"sha256_raw": "ff5c3fa419505127a51acc56dfc13e8437e4ec2c13a11c8d8b2d6c3e20e2410e",
|
||||
"sha256_gz": "4fd30e495f4e6d1c8501d05c23f9891ac34364d86a9c55ac825d52b2ebb31023"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-11",
|
||||
"window": [
|
||||
"2026-08-11T00:00:00+02:00",
|
||||
"2026-08-12T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1056281,
|
||||
"attempts": 1,
|
||||
"samples": 15371,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-10T22:00:00+00:00",
|
||||
"last": "2026-08-11T21:59:58.608150+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-11T08:00:14.121653+00:00",
|
||||
"to": "2026-08-11T08:02:54.427007+00:00",
|
||||
"seconds": 160.305
|
||||
},
|
||||
{
|
||||
"from": "2026-08-11T17:04:18.517592+00:00",
|
||||
"to": "2026-08-11T17:08:41.759777+00:00",
|
||||
"seconds": 263.242
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86398.6,
|
||||
"mean_cadence_s": 5.621,
|
||||
"file": "p1-net-2026-08-11.jsonl.gz",
|
||||
"bytes_gz": 133711,
|
||||
"bytes_raw": 856244,
|
||||
"sha256_raw": "67c2a740019f8908405b96b3436cd953af27baead2957acc7c1dcf10b37ac94e",
|
||||
"sha256_gz": "4f8c553edc9051644bd7baa659f6f92a8b5f5ce12a0410c8b5235d1e6c3084c9"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-12",
|
||||
"window": [
|
||||
"2026-08-12T00:00:00+02:00",
|
||||
"2026-08-13T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124945,
|
||||
"attempts": 1,
|
||||
"samples": 16432,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-11T22:00:00+00:00",
|
||||
"last": "2026-08-12T21:59:58.648966+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.6,
|
||||
"mean_cadence_s": 5.258,
|
||||
"file": "p1-net-2026-08-12.jsonl.gz",
|
||||
"bytes_gz": 136903,
|
||||
"bytes_raw": 911115,
|
||||
"sha256_raw": "73c8a1626640226ddffb32d564557d1cbe57a15367ea1ba3632934e15a1413dc",
|
||||
"sha256_gz": "acec87065b7de2c605db00504ec5a9b3c99584620dfb3b67194e2cf9464d6a5f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-13",
|
||||
"window": [
|
||||
"2026-08-13T00:00:00+02:00",
|
||||
"2026-08-14T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1126742,
|
||||
"attempts": 1,
|
||||
"samples": 16473,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-12T22:00:00+00:00",
|
||||
"last": "2026-08-13T21:59:57.562043+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.6,
|
||||
"mean_cadence_s": 5.245,
|
||||
"file": "p1-net-2026-08-13.jsonl.gz",
|
||||
"bytes_gz": 136221,
|
||||
"bytes_raw": 912379,
|
||||
"sha256_raw": "1d9a16c529174bd8c74625a742c46d98de4acf7a045f83df8bdbac22fa86e8e7",
|
||||
"sha256_gz": "adfef36f629decf54c625f2dbfe6b4dab9441218c42557c90ac7f6d331943a59"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-14",
|
||||
"window": [
|
||||
"2026-08-14T00:00:00+02:00",
|
||||
"2026-08-15T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124413,
|
||||
"attempts": 1,
|
||||
"samples": 16427,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-13T22:00:00+00:00",
|
||||
"last": "2026-08-14T21:59:55.611487+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.6,
|
||||
"mean_cadence_s": 5.26,
|
||||
"file": "p1-net-2026-08-14.jsonl.gz",
|
||||
"bytes_gz": 135570,
|
||||
"bytes_raw": 910648,
|
||||
"sha256_raw": "dc1417881e54fe215e9fcd4ed643d2482a277d61f8f27f25e2ff1e2cba29f7a8",
|
||||
"sha256_gz": "d4eff781cd16a83fd1be9e643fee5bb5d5897e7d45bb9865972a7304d92c84a4"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-15",
|
||||
"window": [
|
||||
"2026-08-15T00:00:00+02:00",
|
||||
"2026-08-16T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124461,
|
||||
"attempts": 1,
|
||||
"samples": 16516,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-14T22:00:00+00:00",
|
||||
"last": "2026-08-15T21:59:56.581991+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.6,
|
||||
"mean_cadence_s": 5.231,
|
||||
"file": "p1-net-2026-08-15.jsonl.gz",
|
||||
"bytes_gz": 137670,
|
||||
"bytes_raw": 909539,
|
||||
"sha256_raw": "f07a242faed68313512e0363d914846522ff839315ad80b55689072ac74e8a69",
|
||||
"sha256_gz": "082314c6003c2208752acc4614b9809f06735d01e216b935ea94faf578c5598e"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-16",
|
||||
"window": [
|
||||
"2026-08-16T00:00:00+02:00",
|
||||
"2026-08-17T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1127871,
|
||||
"attempts": 1,
|
||||
"samples": 16552,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-15T22:00:00+00:00",
|
||||
"last": "2026-08-16T21:59:59.527710+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86399.5,
|
||||
"mean_cadence_s": 5.22,
|
||||
"file": "p1-net-2026-08-16.jsonl.gz",
|
||||
"bytes_gz": 138347,
|
||||
"bytes_raw": 912481,
|
||||
"sha256_raw": "a44e5d961600a539ddf824d5d211ef319833666b7598e5939a6cdf2cb1e3581f",
|
||||
"sha256_gz": "8f6511e52f5be9187f422021abc93a5fa8b698b4c785877d69d15dd6de106fd8"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-17",
|
||||
"window": [
|
||||
"2026-08-17T00:00:00+02:00",
|
||||
"2026-08-18T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1133721,
|
||||
"attempts": 1,
|
||||
"samples": 16627,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-16T22:00:00+00:00",
|
||||
"last": "2026-08-17T21:59:59.685699+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-17T17:27:00.599126+00:00",
|
||||
"to": "2026-08-17T17:28:20.389628+00:00",
|
||||
"seconds": 79.791
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86399.7,
|
||||
"mean_cadence_s": 5.197,
|
||||
"file": "p1-net-2026-08-17.jsonl.gz",
|
||||
"bytes_gz": 140214,
|
||||
"bytes_raw": 917356,
|
||||
"sha256_raw": "70cce79b3732f292f7f58f0b2655bdb4a00a20a6530f1f50dac62ef5f039788b",
|
||||
"sha256_gz": "e2f9b461c4db7902e1a3d30b439ebc11ebcd9618a72d12f10e5131f38fb437bf"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-18",
|
||||
"window": [
|
||||
"2026-08-18T00:00:00+02:00",
|
||||
"2026-08-19T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1074019,
|
||||
"attempts": 1,
|
||||
"samples": 15699,
|
||||
"non_numeric_states": [
|
||||
{
|
||||
"t": "2026-08-18T02:03:54.417106+00:00",
|
||||
"state": "unavailable"
|
||||
},
|
||||
{
|
||||
"t": "2026-08-18T04:10:41.786299+00:00",
|
||||
"state": "unavailable"
|
||||
}
|
||||
],
|
||||
"first": "2026-08-17T22:00:00+00:00",
|
||||
"last": "2026-08-18T21:59:54.501766+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-18T02:02:59.633845+00:00",
|
||||
"to": "2026-08-18T02:04:38.694366+00:00",
|
||||
"seconds": 99.061
|
||||
},
|
||||
{
|
||||
"from": "2026-08-18T04:09:54.292049+00:00",
|
||||
"to": "2026-08-18T04:11:38.500238+00:00",
|
||||
"seconds": 104.208
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86394.5,
|
||||
"mean_cadence_s": 5.504,
|
||||
"file": "p1-net-2026-08-18.jsonl.gz",
|
||||
"bytes_gz": 135894,
|
||||
"bytes_raw": 869570,
|
||||
"sha256_raw": "68cd44ac56099ee38f0bba4c32d7d17d33b92d28a99d27419f53c22374491ca6",
|
||||
"sha256_gz": "d6c97508f805aff040d2b4ed4e06c0e7373db2ea6227c7d79ef24c8c79074115"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-19",
|
||||
"window": [
|
||||
"2026-08-19T00:00:00+02:00",
|
||||
"2026-08-20T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 989720,
|
||||
"attempts": 1,
|
||||
"samples": 14549,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-18T22:00:00+00:00",
|
||||
"last": "2026-08-19T21:59:56.556610+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-19T01:33:29.599224+00:00",
|
||||
"to": "2026-08-19T01:34:01.653383+00:00",
|
||||
"seconds": 32.054
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.6,
|
||||
"mean_cadence_s": 5.939,
|
||||
"file": "p1-net-2026-08-19.jsonl.gz",
|
||||
"bytes_gz": 126345,
|
||||
"bytes_raw": 800369,
|
||||
"sha256_raw": "0f11ce0cee43cbd7bdb08dc48b202e58c442157ac08b2cda5b64bbab9bd56300",
|
||||
"sha256_gz": "13d61b7da903877c8ae7813b833cb5836ec04a1cf1efd0e1fa87e916bef95fd5"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-20",
|
||||
"window": [
|
||||
"2026-08-20T00:00:00+02:00",
|
||||
"2026-08-21T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1068038,
|
||||
"attempts": 1,
|
||||
"samples": 15600,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-19T22:00:00+00:00",
|
||||
"last": "2026-08-20T21:59:56.742604+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-20T02:11:56.607373+00:00",
|
||||
"to": "2026-08-20T02:12:38.840458+00:00",
|
||||
"seconds": 42.233
|
||||
},
|
||||
{
|
||||
"from": "2026-08-20T20:42:04.623102+00:00",
|
||||
"to": "2026-08-20T20:43:41.666543+00:00",
|
||||
"seconds": 97.043
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.7,
|
||||
"mean_cadence_s": 5.539,
|
||||
"file": "p1-net-2026-08-20.jsonl.gz",
|
||||
"bytes_gz": 139693,
|
||||
"bytes_raw": 865024,
|
||||
"sha256_raw": "2612153963951dd13fe310ad7fbb5bec2e195db56fd99fd963134c8badae25ba",
|
||||
"sha256_gz": "362f2ee3d9d7f23f9051ba78993c5af7443dabb3b5f16339efc673444720c892"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-21",
|
||||
"window": [
|
||||
"2026-08-21T00:00:00+02:00",
|
||||
"2026-08-22T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1118002,
|
||||
"attempts": 1,
|
||||
"samples": 16383,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-20T22:00:00+00:00",
|
||||
"last": "2026-08-21T21:59:57.771588+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.8,
|
||||
"mean_cadence_s": 5.274,
|
||||
"file": "p1-net-2026-08-21.jsonl.gz",
|
||||
"bytes_gz": 139105,
|
||||
"bytes_raw": 904809,
|
||||
"sha256_raw": "933d9a6899f59c128d0d9b84d6944cea63b1ad58dc57ac52e3e476f9c616f07b",
|
||||
"sha256_gz": "a563581a9a3338af173abb99a200db32881ddaf893d2458f5d8de59b5ec6041f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-22",
|
||||
"window": [
|
||||
"2026-08-22T00:00:00+02:00",
|
||||
"2026-08-23T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1129094,
|
||||
"attempts": 1,
|
||||
"samples": 16520,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-21T22:00:00+00:00",
|
||||
"last": "2026-08-22T21:59:56.913786+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-21T23:05:20.035692+00:00",
|
||||
"to": "2026-08-21T23:07:51.147719+00:00",
|
||||
"seconds": 151.112
|
||||
},
|
||||
{
|
||||
"from": "2026-08-21T23:07:56.966604+00:00",
|
||||
"to": "2026-08-21T23:08:27.695101+00:00",
|
||||
"seconds": 30.728
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.23,
|
||||
"file": "p1-net-2026-08-22.jsonl.gz",
|
||||
"bytes_gz": 141564,
|
||||
"bytes_raw": 914120,
|
||||
"sha256_raw": "39ccdf67c806344490f791648822cd2cd14d73ea7cc0b6e8572a1e15e8f74641",
|
||||
"sha256_gz": "5c46f1879a723f9986d521124200cab839af2a4722b76bc97c6f6f59d0df65a2"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-23",
|
||||
"window": [
|
||||
"2026-08-23T00:00:00+02:00",
|
||||
"2026-08-24T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1107164,
|
||||
"attempts": 1,
|
||||
"samples": 16351,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-22T22:00:00+00:00",
|
||||
"last": "2026-08-23T21:59:56.921115+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.284,
|
||||
"file": "p1-net-2026-08-23.jsonl.gz",
|
||||
"bytes_gz": 136277,
|
||||
"bytes_raw": 894387,
|
||||
"sha256_raw": "ffb15367bf3c98eff8a5600e4858ac99778b597c03d0a4f6154e6971b8f98a44",
|
||||
"sha256_gz": "3fa76981758a468ba0aa68b50fe68d5bdb1ef38cd7faec2b6ffdb61c9ddb2db2"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-24",
|
||||
"window": [
|
||||
"2026-08-24T00:00:00+02:00",
|
||||
"2026-08-25T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1122175,
|
||||
"attempts": 1,
|
||||
"samples": 16584,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-23T22:00:00+00:00",
|
||||
"last": "2026-08-24T21:59:57.940474+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.9,
|
||||
"mean_cadence_s": 5.21,
|
||||
"file": "p1-net-2026-08-24.jsonl.gz",
|
||||
"bytes_gz": 136834,
|
||||
"bytes_raw": 906369,
|
||||
"sha256_raw": "30465d5ff51220201b4a79341e5da7bfc3d88290465b1c9d886ba94eee8dd234",
|
||||
"sha256_gz": "5103088e047bfcbd9b3f55521f431b01cd27de10c2642e00281a13faa762bdd9"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-25",
|
||||
"window": [
|
||||
"2026-08-25T00:00:00+02:00",
|
||||
"2026-08-26T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1110567,
|
||||
"attempts": 1,
|
||||
"samples": 16451,
|
||||
"non_numeric_states": [
|
||||
{
|
||||
"t": "2026-08-25T01:52:02.104225+00:00",
|
||||
"state": "unavailable"
|
||||
}
|
||||
],
|
||||
"first": "2026-08-24T22:00:00+00:00",
|
||||
"last": "2026-08-25T21:59:56.883744+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-25T01:51:25.942719+00:00",
|
||||
"to": "2026-08-25T01:52:53.203963+00:00",
|
||||
"seconds": 87.261
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.252,
|
||||
"file": "p1-net-2026-08-25.jsonl.gz",
|
||||
"bytes_gz": 135890,
|
||||
"bytes_raw": 896416,
|
||||
"sha256_raw": "0e7c86b2b74f2108bc7f5f5d2de2627caaafe473ca4ae18515534dff10526e26",
|
||||
"sha256_gz": "2a5411c10d8d3b271f133c62c04bd722279b48b08d7376084fbad6cc6dd66d76"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-26",
|
||||
"window": [
|
||||
"2026-08-26T00:00:00+02:00",
|
||||
"2026-08-27T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1123610,
|
||||
"attempts": 1,
|
||||
"samples": 16542,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-25T22:00:00+00:00",
|
||||
"last": "2026-08-26T21:59:58.915457+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.9,
|
||||
"mean_cadence_s": 5.223,
|
||||
"file": "p1-net-2026-08-26.jsonl.gz",
|
||||
"bytes_gz": 139301,
|
||||
"bytes_raw": 908350,
|
||||
"sha256_raw": "88d72c1b9f51707d62e744f40402de4260a04617875b85c65d2df6079c6c84ca",
|
||||
"sha256_gz": "0661588fbc0c6c42825b67d483c8818c070405c1002424b3f1d61d8fdadb6370"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-27",
|
||||
"window": [
|
||||
"2026-08-27T00:00:00+02:00",
|
||||
"2026-08-28T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1122433,
|
||||
"attempts": 1,
|
||||
"samples": 16573,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-26T22:00:00+00:00",
|
||||
"last": "2026-08-27T21:59:58.538446+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-27T18:50:29.030586+00:00",
|
||||
"to": "2026-08-27T18:52:22.938470+00:00",
|
||||
"seconds": 113.908
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.214,
|
||||
"file": "p1-net-2026-08-27.jsonl.gz",
|
||||
"bytes_gz": 138009,
|
||||
"bytes_raw": 906770,
|
||||
"sha256_raw": "528ab8ee10a46275d2e91d56ef2ed847e5932fae76a1752e6f2e9828dbfa9865",
|
||||
"sha256_gz": "5cacc8c5788f43d6e6e2f569445fa82d9edc5a63b1f082f9a1d6a3bb0b404260"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-28",
|
||||
"window": [
|
||||
"2026-08-28T00:00:00+02:00",
|
||||
"2026-08-29T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1114796,
|
||||
"attempts": 1,
|
||||
"samples": 16456,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-27T22:00:00+00:00",
|
||||
"last": "2026-08-28T21:59:58.515555+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.251,
|
||||
"file": "p1-net-2026-08-28.jsonl.gz",
|
||||
"bytes_gz": 135917,
|
||||
"bytes_raw": 900654,
|
||||
"sha256_raw": "f89838bc9f61af074c17817e4a6eab037318f38fa3c65dad3edfa2106abbc9fb",
|
||||
"sha256_gz": "97090b605f67dcc1f84dea2c51d4832dfe1c2af497a4f4cfe50ecb51024d9b4c"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-29",
|
||||
"window": [
|
||||
"2026-08-29T00:00:00+02:00",
|
||||
"2026-08-30T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1092983,
|
||||
"attempts": 1,
|
||||
"samples": 16132,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-28T22:00:00+00:00",
|
||||
"last": "2026-08-29T21:59:57.574524+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-29T00:03:20.539844+00:00",
|
||||
"to": "2026-08-29T00:03:50.569581+00:00",
|
||||
"seconds": 30.03
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86397.6,
|
||||
"mean_cadence_s": 5.356,
|
||||
"file": "p1-net-2026-08-29.jsonl.gz",
|
||||
"bytes_gz": 134210,
|
||||
"bytes_raw": 883053,
|
||||
"sha256_raw": "896d4c9d660be4faabc5a088323e246e257877d5ff6bd0ec084b785480d76e12",
|
||||
"sha256_gz": "b6f264cd3afba63209123a27490e7e1c3ae70a86731d7ea05ae67f88917f136f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-30",
|
||||
"window": [
|
||||
"2026-08-30T00:00:00+02:00",
|
||||
"2026-08-31T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1118895,
|
||||
"attempts": 1,
|
||||
"samples": 16490,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-29T22:00:00+00:00",
|
||||
"last": "2026-08-30T21:59:58.533323+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.24,
|
||||
"file": "p1-net-2026-08-30.jsonl.gz",
|
||||
"bytes_gz": 138429,
|
||||
"bytes_raw": 904311,
|
||||
"sha256_raw": "e704818efb58bb6c505791bc9eab2e9476bf48ff8911d0e79b401fd35b529fa0",
|
||||
"sha256_gz": "9052fead2418bff3083bec3ae8ce64d6b9f6224b668f4bb510c79775f66c5b79"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"days_requested": 30,
|
||||
"days_read": 30,
|
||||
"days_failed": 0,
|
||||
"samples": 485841,
|
||||
"bytes_gz": 4126003,
|
||||
"empty_days": []
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -212,6 +212,14 @@ def shipped(key):
|
||||
if in_schema or not line.startswith(f" {key}:"):
|
||||
continue
|
||||
raw = line.split(":", 1)[1].split("#", 1)[0].strip().strip('"').strip("'")
|
||||
# OPT-04 added the first BOOLEAN price option, and without this the
|
||||
# mirror check would compare the string "false" against `False` and go
|
||||
# red for a file that is perfectly in step. Parsing a bool as a bool is
|
||||
# what the yaml module this image does not carry would have done; it
|
||||
# widens nothing, because a key whose shipped value disagrees with the
|
||||
# mirror still drifts.
|
||||
if raw in ("true", "false"):
|
||||
return raw == "true"
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
|
||||
@@ -71,6 +71,7 @@ from app.price_forecast import (
|
||||
MAINTENANCE_CHARGE, MAINTENANCE_DRAIN, Gate, Irreversible, committed_hours,
|
||||
fill_from_yesterday, price_permits, begin_phase,
|
||||
)
|
||||
from app.arbitrage import ArbitrageDecision
|
||||
from app.static_tariff import Arbitrage, TariffPlan
|
||||
|
||||
UTC = timezone.utc
|
||||
@@ -582,7 +583,13 @@ _ctors = {"Gate.permit": Gate.permit, "Gate.block": Gate.block,
|
||||
"Arbitrage.allow": Arbitrage.allow, "Arbitrage.refuse": Arbitrage.refuse,
|
||||
"Arbitrage.match": Arbitrage.match,
|
||||
"_Allowed.__init__": _st._Allowed.__init__,
|
||||
"_Refused.__init__": _st._Refused.__init__}
|
||||
"_Refused.__init__": _st._Refused.__init__,
|
||||
# ⚠️ OPT-04's outcome type is scanned HERE, in the same dict, rather
|
||||
# than by a hand-written case in test_arbitrage.py - which is TEST-14's
|
||||
# actual instruction. A new outcome type extends the SCAN; a
|
||||
# hand-written case is the thing that gets forgotten, and the whole
|
||||
# reason this dict exists instead of five separate checks.
|
||||
"ArbitrageDecision.__init__": ArbitrageDecision.__init__}
|
||||
_with_defaults = derived(lambda: {n: d for n, d in
|
||||
((n, _defaulted(f)) for n, f in _ctors.items())
|
||||
if d})
|
||||
@@ -835,13 +842,28 @@ print("7. CFG-03 and validate.py rule 7 - what this ticket did NOT widen")
|
||||
_defaults = derived(lambda: sorted(_price.PRICE_OPTION_DEFAULTS))
|
||||
_cfg = derived(lambda: open(os.path.join(HERE, "config.yaml"),
|
||||
encoding="utf-8").read())
|
||||
check(f"FCST-03 added ZERO option keys: the in-code mirror still holds exactly "
|
||||
f"TARIFF-02's and TARIFF-03's {_defaults}",
|
||||
# ⚠️ THIS LIST IS A DRIFT PIN, NOT A BAN ON EVER ADDING AN OPTION. FCST-03 added
|
||||
# none, and said so here. OPT-04 added FOUR - and added them to this ONE dict
|
||||
# and to config.yaml, which is precisely the behaviour CFG-03 asks for: the
|
||||
# alternative is a fourth freehand copy in main.py's `o.get(k, X)` fallbacks.
|
||||
# The pin is what makes such an addition a decision somebody signs rather than a
|
||||
# thing that appears; whoever adds the fifth comes here and says which card and
|
||||
# why, as this line does.
|
||||
_OPT04_KEYS = ["arbitrage_enabled", "cycle_cost_threshold_eur",
|
||||
"arbitrage_soc_floor", "arbitrage_soc_ceiling"]
|
||||
check(f"the in-code mirror holds exactly TARIFF-02's, TARIFF-03's and OPT-04's "
|
||||
f"keys - FCST-03 still added none of its own {_defaults}",
|
||||
_defaults == sorted(["price_provider", "price_markup_eur_kwh",
|
||||
"price_taxes_eur_kwh", "export_price_eur_kwh",
|
||||
"price_csv_path", "price_feed_url",
|
||||
"price_horizon_h", "price_timeout_s",
|
||||
"static_profile"]))
|
||||
"static_profile"] + _OPT04_KEYS))
|
||||
# ...and the four are genuinely NEW, not a rename of something that was already
|
||||
# there, or the line above would pass while the mirror silently lost a key.
|
||||
check("...and OPT-04's four keys are additions rather than replacements: the "
|
||||
"nine TARIFF-02/03 keys are all still present",
|
||||
isinstance(_defaults, list)
|
||||
and len(set(_defaults) - set(_OPT04_KEYS)) == 9)
|
||||
check("...and every one of them is still in config.yaml, which stays the "
|
||||
"source of truth",
|
||||
isinstance(_cfg, str)
|
||||
|
||||
@@ -35,16 +35,39 @@ NOT allowed to be tested at:
|
||||
10. THE WIRING, through the real `Controller` and its real status page.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
⚠️ AC 3 IS NOT MET, AND PART 9 DOES NOT PRETEND OTHERWISE.
|
||||
⚠️ PART 9 IS A TWO-DAY MEASUREMENT AND SAYS SO. THE 30-DAY ANSWER IS ELSEWHERE
|
||||
(DATA-01), AND IT DOES NOT LAND WHERE ANYONE EXPECTED.
|
||||
|
||||
AC 3 asks for "the monthly peak held under the soft limit in >= 95 % of
|
||||
intervals over a 30-day simulation replay". THERE IS NO 30-DAY DATASET. The
|
||||
infra repo's `sim/scenarios/` holds exactly two days of this house's real meter
|
||||
- 2026-08-20 (the Illusmart failure day) and 2026-08-23 (the healthy reference)
|
||||
- exported on 2026-08-25 because they were about to age out of HA's recorder,
|
||||
and nothing longer was ever captured. `sim/scenarios/README.md` lists the whole
|
||||
inventory; the rest is RS485 bus frames, OTA witness logs and a 578-row rollback
|
||||
poll, none of which carries grid power over calendar time.
|
||||
intervals over a 30-day simulation replay". When this file was written there
|
||||
were two days of this house's real meter - 2026-08-20 (the Illusmart failure
|
||||
day) and 2026-08-23 (the healthy reference) - exported on 2026-08-25 because
|
||||
they were about to age out of HA's recorder.
|
||||
|
||||
⚠️ AND THE SENTENCE THAT USED TO STAND HERE, "nothing longer was ever
|
||||
captured", WAS TRUE ABOUT THE PAST AND WAS READ AS A CLAIM ABOUT THE FUTURE.
|
||||
It was not one. HA's recorder still held the whole month; DATA-01 probed it on
|
||||
2026-08-31 (230 samples in a 20-minute window at 30 days back, 0 at 45) and
|
||||
exported thirty days the same day, into `scenarios/p1-net-30d-2026-08/` with
|
||||
`tools/export_p1_history.py` - which is committed now precisely because the
|
||||
2026-08-25 one-off was not, and its absence is what let the belief survive.
|
||||
|
||||
Over those thirty days the criterion measures 98.89 % UNCONTROLLED and 98.99 %
|
||||
guarded, so it is met - and that is a finding about the CRITERION as much as
|
||||
about the guard, because a month with no controller at all passes it. Read the
|
||||
other way - the monthly PEAK itself under the soft limit - it is not met: the
|
||||
guard takes August's peak from 5029.8 W to 4107.3 W, still over 3500 W. Both
|
||||
numbers, and why they part company, are in
|
||||
`scenarios/p1-net-30d-2026-08/README.md`.
|
||||
|
||||
PART 9 IS DELIBERATELY LEFT GRADING THE TWO COMMITTED DAYS. It is a per-PR
|
||||
regression check and the thirty days cost about fifty seconds of replay; they
|
||||
also live one directory down precisely so this file's glob does not pick them
|
||||
up. `tools/replay_ac3.py` is the thing that runs the month, and it is checked
|
||||
against part 6 of test_peakguard.py rather than trusted:
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
So part 9 replays the two days that exist, reports the percentage over them, and
|
||||
states the sample size in days next to it. It is a MEASUREMENT, not a verdict:
|
||||
@@ -60,8 +83,8 @@ export) for the reason `scenarios/p1-age-2026-08-26.jsonl.gz` already is: the
|
||||
infra repo is private, CI's token is minted for this repository only, and a
|
||||
regression fixture that CI cannot see is not a regression fixture.
|
||||
|
||||
Closing AC 3 needs 30 days of capture. That is TEST-01's replay harness, and it
|
||||
cannot start before somebody starts recording.
|
||||
The recorder's window rolls: 2026-08-01 leaves it around 2026-09-15, which is
|
||||
why the committed copy is the artefact and not the API call.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -748,9 +771,11 @@ print()
|
||||
print(f" AC 3 MEASURED: {under}/{total_q} quarters = {PCT:.1f} % under the "
|
||||
f"{SHIPPED_CAP_W:.0f} W soft limit, over {replay_days:.1f} DAYS of "
|
||||
f"replay, UNCONTROLLED (no peak shaving was running on either day).")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS. {replay_days:.1f} days exist. "
|
||||
f"AC 3 IS NOT MET and this run does not claim it is - see the module "
|
||||
f"docstring. The figure above is the BASELINE SAFETY-07 has to improve.")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS and this run is {replay_days:.1f}, so it "
|
||||
f"does not settle it either way. The 30 days DO now exist - "
|
||||
f"scenarios/p1-net-30d-2026-08/, DATA-01 - and "
|
||||
f"tools/replay_ac3.py is what runs them; the figure above is the "
|
||||
f"BASELINE SAFETY-07 has to improve, on the two days CI replays.")
|
||||
check("the replay produced a measurable number over a stated sample size, "
|
||||
"rather than a criterion quietly redefined to fit the data",
|
||||
total_q > 100 and replay_days < 30)
|
||||
|
||||
@@ -0,0 +1,455 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export a P1 power signal out of Home Assistant's recorder into the committed
|
||||
scenario format - one gzipped JSON object per line, one file per LOCAL day.
|
||||
|
||||
WHY THIS FILE EXISTS AT ALL
|
||||
===========================
|
||||
`scenarios/p1-net-2026-08-20.jsonl.gz` and `p1-net-2026-08-23.jsonl.gz` were
|
||||
produced once, by hand, and the script was thrown away. The consequence was not
|
||||
theoretical: TARIFF-01's and SAFETY-07's AC 3 ("monthly peak held under the soft
|
||||
limit in >= 95 % of intervals over a 30-day simulation replay") were both merged
|
||||
UNMET on the belief that thirty days could not be produced retroactively - a
|
||||
belief that was simply wrong, and that survived because nobody could re-run the
|
||||
export to find out. So the export is a committed, runnable artefact now, and the
|
||||
next person does not have to re-derive it under time pressure while a retention
|
||||
window rolls.
|
||||
|
||||
python3 tools/export_p1_history.py --days 30 --out-dir /some/where
|
||||
|
||||
THE TWO RULES THIS FILE IS BUILT AROUND
|
||||
=======================================
|
||||
1. A FAILED READ IS NOT AN OBSERVATION. An HTTP error, a timeout, a truncated
|
||||
body, a non-JSON body - none of those are "the meter reported nothing that
|
||||
hour". They are "we do not know what the meter reported that hour", and the
|
||||
two must never end up looking the same in the output. So:
|
||||
|
||||
* every request is retried with backoff;
|
||||
* a day whose read ultimately fails writes NO FILE and is recorded in the
|
||||
manifest as `"read": "FAILED"` with the error text, and the process exits
|
||||
non-zero;
|
||||
* a day that reads back successfully but EMPTY writes an empty file and is
|
||||
recorded as `"read": "OK", "samples": 0`. Empty-and-known and
|
||||
unknown are therefore distinguishable both in the manifest and by the
|
||||
presence or absence of the file;
|
||||
* before any day is fetched, the entity is proved to exist through
|
||||
/api/states/<entity>. Without that check a typo in --entity would return
|
||||
200 + `[]` for all thirty days and produce a beautifully consistent
|
||||
dataset of nothing.
|
||||
|
||||
2. GAPS ARE VISIBLE, NEVER INTERPOLATED. Nothing here carries a value forward,
|
||||
fills, resamples or averages. A recorder outage or an HA restart shows up as
|
||||
what it is: consecutive `t` values further apart than the meter's cadence.
|
||||
The controller's own ingest already reads that correctly - `meter_max_age_s`
|
||||
(30 s shipped) is exactly the age past which a held reading stops counting as
|
||||
coverage - so an hour with no telegrams replays as an hour of NO COVERAGE
|
||||
rather than as an hour of whatever the last sample said. The manifest lists
|
||||
every gap over --gap-s so a reader does not have to diff timestamps to find
|
||||
them.
|
||||
|
||||
DAY AT A TIME, AND WHY NOT SMALLER
|
||||
==================================
|
||||
One request per local day. That is gentle on a live HA (~1 MB of JSON per day)
|
||||
and it is also the only chunking that reproduces the committed files: the
|
||||
history API prepends the state as it stood AT the period start
|
||||
(`include_start_time_state`), stamped with the start instant. Splitting a day
|
||||
into six-hour chunks would therefore inject three synthetic boundary samples per
|
||||
day that no meter ever sent. Days are the natural boundary because the committed
|
||||
files are local days.
|
||||
|
||||
VERIFYING THE EXPORTER
|
||||
======================
|
||||
python3 tools/export_p1_history.py --verify scenarios/p1-net-2026-08-20.jsonl.gz
|
||||
|
||||
re-fetches that day by this exact route and compares the DECOMPRESSED bytes with
|
||||
the committed file. Byte equality of the .gz containers is not expected and not
|
||||
required: a gzip header carries an mtime, and this file writes mtime 0 so its
|
||||
own output is reproducible. Content equality is the claim that matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
try: # py3.9+; the add-on image has it
|
||||
from zoneinfo import ZoneInfo
|
||||
except ImportError: # pragma: no cover
|
||||
ZoneInfo = None # type: ignore
|
||||
|
||||
DEFAULT_URL = "http://192.168.2.158:8123"
|
||||
DEFAULT_ENTITY = "sensor.p1_meter_active_power"
|
||||
DEFAULT_TZ = "Europe/Brussels"
|
||||
# The shipped meter_max_age_s. A hole longer than this is a hole the controller
|
||||
# itself would refuse to see through, which makes it the honest gap threshold.
|
||||
DEFAULT_GAP_S = 30.0
|
||||
|
||||
|
||||
class ReadFailed(RuntimeError):
|
||||
"""The read did not happen. NOT the same thing as an empty result."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# transport
|
||||
|
||||
|
||||
def _get(url: str, token: str, timeout: float):
|
||||
req = urllib.request.Request(url, headers={
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json",
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
if resp.status != 200:
|
||||
raise ReadFailed(f"HTTP {resp.status}")
|
||||
return resp.read()
|
||||
|
||||
|
||||
def fetch_json(url: str, token: str, *, timeout: float, retries: int,
|
||||
backoff: float, log=print):
|
||||
"""GET and parse, or raise ReadFailed. Never returns a partial answer."""
|
||||
last = None
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
raw = _get(url, token, timeout)
|
||||
except urllib.error.HTTPError as exc:
|
||||
last = f"HTTP {exc.code} {exc.reason}"
|
||||
except Exception as exc: # URLError, timeout, ...
|
||||
last = f"{type(exc).__name__}: {exc}"
|
||||
else:
|
||||
try:
|
||||
return json.loads(raw.decode("utf-8")), len(raw), attempt
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
# A truncated body looks exactly like this. It is a failure.
|
||||
last = f"unparseable body ({len(raw)} bytes): {exc}"
|
||||
if attempt < retries:
|
||||
wait = backoff * attempt
|
||||
log(f" read failed ({last}); retry {attempt + 1}/{retries} "
|
||||
f"in {wait:.0f} s")
|
||||
time.sleep(wait)
|
||||
raise ReadFailed(last or "unknown")
|
||||
|
||||
|
||||
def assert_entity_exists(base: str, entity: str, token: str, *, timeout: float,
|
||||
retries: int, backoff: float, log=print):
|
||||
"""Prove the entity is real BEFORE trusting any empty day.
|
||||
|
||||
Without this, `--entity sensor.typo` answers 200 + [] for every day and the
|
||||
exporter cheerfully writes thirty empty files.
|
||||
"""
|
||||
url = f"{base}/api/states/{urllib.parse.quote(entity)}"
|
||||
doc, _, _ = fetch_json(url, token, timeout=timeout, retries=retries,
|
||||
backoff=backoff, log=log)
|
||||
if not isinstance(doc, dict) or doc.get("entity_id") != entity:
|
||||
raise ReadFailed(f"{entity} is not a known entity")
|
||||
return doc
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# one day
|
||||
|
||||
|
||||
def day_bounds(day: datetime, tz):
|
||||
start = datetime(day.year, day.month, day.day, tzinfo=tz)
|
||||
return start, start + timedelta(days=1)
|
||||
|
||||
|
||||
def history_url(base: str, entity: str, start: datetime, end: datetime) -> str:
|
||||
q = urllib.parse.urlencode({
|
||||
"filter_entity_id": entity,
|
||||
"end_time": end.isoformat(),
|
||||
"minimal_response": "",
|
||||
})
|
||||
return (f"{base}/api/history/period/"
|
||||
f"{urllib.parse.quote(start.isoformat())}?{q}")
|
||||
|
||||
|
||||
def rows_from(doc, entity: str):
|
||||
"""HA's shape -> [(iso_t, state_str)], plus the states that are not numbers.
|
||||
|
||||
`minimal_response` returns the first record in full and the rest as
|
||||
{state, last_changed}. An entity that went unavailable reports the string
|
||||
"unavailable" (or "unknown") where a number belongs. Those are dropped from
|
||||
the numeric series - there is no float to write and the committed schema has
|
||||
no slot for a non-number - but every one is returned here so it lands in the
|
||||
manifest. Dropping them is what MAKES the outage visible: the series simply
|
||||
has no samples across it, which is the same thing the meter falling silent
|
||||
looks like, and is what the controller's freshness rule already handles.
|
||||
"""
|
||||
if not isinstance(doc, list):
|
||||
raise ReadFailed(f"expected a list, got {type(doc).__name__}")
|
||||
if not doc:
|
||||
return [], []
|
||||
series = doc[0]
|
||||
if not isinstance(series, list):
|
||||
raise ReadFailed("history series is not a list")
|
||||
good, bad = [], []
|
||||
for rec in series:
|
||||
eid = rec.get("entity_id")
|
||||
if eid is not None and eid != entity:
|
||||
raise ReadFailed(f"unexpected entity {eid!r} in the response")
|
||||
t = rec.get("last_changed") or rec.get("last_updated")
|
||||
s = rec.get("state")
|
||||
if t is None:
|
||||
raise ReadFailed("a record carries no timestamp")
|
||||
try:
|
||||
w = float(s)
|
||||
except (TypeError, ValueError):
|
||||
bad.append({"t": t, "state": s})
|
||||
continue
|
||||
good.append((t, w))
|
||||
return good, bad
|
||||
|
||||
|
||||
def render(rows, *, newline: str) -> bytes:
|
||||
"""The committed format, exactly: `{"t": <iso>, "w": <float>}` per line.
|
||||
|
||||
json.dumps' default separators (", " / ": ") and repr of a float are what
|
||||
the two committed files contain; so is the CRLF, an artefact of the original
|
||||
one-off having been run on Windows in text mode. It is reproduced rather
|
||||
than tidied because reproducing the committed bytes is this exporter's
|
||||
verification, and gzip.open(..., 'rt') strips it either way.
|
||||
"""
|
||||
out = bytearray()
|
||||
for t, w in rows:
|
||||
out += json.dumps({"t": t, "w": w}).encode("utf-8")
|
||||
out += newline.encode("ascii")
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def gz(payload: bytes) -> bytes:
|
||||
"""gzip level 9, mtime 0 - so the same input always gives the same bytes."""
|
||||
import io
|
||||
buf = io.BytesIO()
|
||||
with gzip.GzipFile(fileobj=buf, mode="wb", compresslevel=9, mtime=0) as f:
|
||||
f.write(payload)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def gaps_in(rows, gap_s: float):
|
||||
out = []
|
||||
prev = None
|
||||
for t, _ in rows:
|
||||
cur = datetime.fromisoformat(t)
|
||||
if prev is not None:
|
||||
d = (cur - prev).total_seconds()
|
||||
if d > gap_s:
|
||||
out.append({"from": prev.isoformat(), "to": cur.isoformat(),
|
||||
"seconds": round(d, 3)})
|
||||
prev = cur
|
||||
return out
|
||||
|
||||
|
||||
def export_day(base, entity, token, day, tz, *, timeout, retries, backoff,
|
||||
gap_s, newline, log=print):
|
||||
"""Read one local day. Raises ReadFailed rather than returning a guess."""
|
||||
start, end = day_bounds(day, tz)
|
||||
doc, nbytes, attempts = fetch_json(
|
||||
history_url(base, entity, start, end), token,
|
||||
timeout=timeout, retries=retries, backoff=backoff, log=log)
|
||||
rows, bad = rows_from(doc, entity)
|
||||
entry = {
|
||||
"date": day.date().isoformat(),
|
||||
"window": [start.isoformat(), end.isoformat()],
|
||||
"read": "OK",
|
||||
"http_bytes": nbytes,
|
||||
"attempts": attempts,
|
||||
"samples": len(rows),
|
||||
"non_numeric_states": bad,
|
||||
"first": rows[0][0] if rows else None,
|
||||
"last": rows[-1][0] if rows else None,
|
||||
"gaps": gaps_in(rows, gap_s),
|
||||
}
|
||||
if rows:
|
||||
span = (datetime.fromisoformat(rows[-1][0])
|
||||
- datetime.fromisoformat(rows[0][0])).total_seconds()
|
||||
entry["covered_span_s"] = round(span, 1)
|
||||
entry["mean_cadence_s"] = round(span / max(1, len(rows) - 1), 3)
|
||||
return rows, entry
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# modes
|
||||
|
||||
|
||||
def do_export(args, token, tz, log=print):
|
||||
base = args.url.rstrip("/")
|
||||
out_dir = Path(args.out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
newline = "\r\n" if args.newline == "crlf" else "\n"
|
||||
|
||||
log(f" proving {args.entity} exists before trusting any empty day")
|
||||
st = assert_entity_exists(base, args.entity, token, timeout=args.timeout,
|
||||
retries=args.retries, backoff=args.backoff,
|
||||
log=log)
|
||||
log(f" {args.entity} = {st.get('state')} "
|
||||
f"{st.get('attributes', {}).get('unit_of_measurement', '')}")
|
||||
|
||||
if args.end_date:
|
||||
last = datetime.fromisoformat(args.end_date).replace(tzinfo=tz)
|
||||
else: # yesterday: today is still being logged
|
||||
last = datetime.now(tz).replace(hour=0, minute=0, second=0,
|
||||
microsecond=0) - timedelta(days=1)
|
||||
days = [last - timedelta(days=i) for i in range(args.days - 1, -1, -1)]
|
||||
|
||||
manifest = {
|
||||
"generated": datetime.now(timezone.utc).isoformat(),
|
||||
"generator": "tools/export_p1_history.py",
|
||||
"source": f"{base}/api/history/period",
|
||||
"entity": args.entity,
|
||||
"timezone": args.tz,
|
||||
"gap_threshold_s": args.gap_s,
|
||||
"format": '{"t": <ISO-8601 UTC>, "w": <float watts>} per line, gzip',
|
||||
"days": [],
|
||||
}
|
||||
failures = []
|
||||
for day in days:
|
||||
name = f"{args.prefix}{day.date().isoformat()}.jsonl.gz"
|
||||
log(f" {name}")
|
||||
try:
|
||||
rows, entry = export_day(
|
||||
base, args.entity, token, day, tz, timeout=args.timeout,
|
||||
retries=args.retries, backoff=args.backoff, gap_s=args.gap_s,
|
||||
newline=newline, log=log)
|
||||
except ReadFailed as exc:
|
||||
# NO FILE. A missing file is the honest record of a read that did
|
||||
# not happen; an empty one would claim the meter was silent.
|
||||
entry = {"date": day.date().isoformat(), "read": "FAILED",
|
||||
"error": str(exc), "file": None}
|
||||
manifest["days"].append(entry)
|
||||
failures.append(entry)
|
||||
log(f" READ FAILED: {exc} - no file written")
|
||||
continue
|
||||
payload = render(rows, newline=newline)
|
||||
blob = gz(payload)
|
||||
(out_dir / name).write_bytes(blob)
|
||||
entry["file"] = name
|
||||
entry["bytes_gz"] = len(blob)
|
||||
entry["bytes_raw"] = len(payload)
|
||||
entry["sha256_raw"] = hashlib.sha256(payload).hexdigest()
|
||||
entry["sha256_gz"] = hashlib.sha256(blob).hexdigest()
|
||||
manifest["days"].append(entry)
|
||||
ng = len(entry.get("gaps", []))
|
||||
log(f" {entry['samples']} samples, {len(blob)} B gz, "
|
||||
f"{ng} gap(s) over {args.gap_s:g} s"
|
||||
+ (f", {len(entry['non_numeric_states'])} non-numeric state(s)"
|
||||
if entry["non_numeric_states"] else ""))
|
||||
if day is not days[-1]:
|
||||
time.sleep(args.delay)
|
||||
|
||||
ok = [d for d in manifest["days"] if d["read"] == "OK"]
|
||||
manifest["summary"] = {
|
||||
"days_requested": len(days),
|
||||
"days_read": len(ok),
|
||||
"days_failed": len(failures),
|
||||
"samples": sum(d["samples"] for d in ok),
|
||||
"bytes_gz": sum(d["bytes_gz"] for d in ok),
|
||||
"empty_days": [d["date"] for d in ok if d["samples"] == 0],
|
||||
}
|
||||
# newline="\n" explicitly: write_text() on Windows emits CRLF, and
|
||||
# the manifest would then differ from a Linux run of the same export
|
||||
# for no reason visible in a diff.
|
||||
with open(args.manifest or (out_dir / "manifest.json"), "w",
|
||||
encoding="utf-8", newline="\n") as fh:
|
||||
fh.write(json.dumps(manifest, indent=2) + "\n")
|
||||
log(f" {manifest['summary']['days_read']}/{len(days)} days, "
|
||||
f"{manifest['summary']['samples']} samples, "
|
||||
f"{manifest['summary']['bytes_gz'] / 1e6:.2f} MB gz")
|
||||
if failures:
|
||||
log(f" {len(failures)} DAY(S) NOT READ - the dataset is incomplete "
|
||||
f"and says so: " + ", ".join(d["date"] for d in failures))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def do_verify(args, token, tz, log=print):
|
||||
"""Re-export a committed day by this route and diff it against the file."""
|
||||
base = args.url.rstrip("/")
|
||||
newline = "\r\n" if args.newline == "crlf" else "\n"
|
||||
rc = 0
|
||||
for target in args.verify:
|
||||
path = Path(target)
|
||||
stamp = path.name.split(".")[0].rsplit("-", 3)[-3:]
|
||||
day = datetime(int(stamp[0]), int(stamp[1]), int(stamp[2]), tzinfo=tz)
|
||||
log(f" {path.name} (local day {day.date().isoformat()})")
|
||||
committed = gzip.decompress(path.read_bytes())
|
||||
rows, entry = export_day(base, args.entity, token, day, tz,
|
||||
timeout=args.timeout, retries=args.retries,
|
||||
backoff=args.backoff, gap_s=args.gap_s,
|
||||
newline=newline, log=log)
|
||||
fresh = render(rows, newline=newline)
|
||||
if fresh == committed:
|
||||
log(f" IDENTICAL: {len(fresh)} bytes, {entry['samples']} "
|
||||
f"samples, sha256 {hashlib.sha256(fresh).hexdigest()[:16]}")
|
||||
continue
|
||||
rc = 1
|
||||
cl = committed.split(newline.encode())
|
||||
fl = fresh.split(newline.encode())
|
||||
log(f" DIFFERS: committed {len(committed)} B / {len(cl)} lines, "
|
||||
f"re-export {len(fresh)} B / {len(fl)} lines")
|
||||
shown = 0
|
||||
for i in range(max(len(cl), len(fl))):
|
||||
a = cl[i] if i < len(cl) else b"<absent>"
|
||||
b = fl[i] if i < len(fl) else b"<absent>"
|
||||
if a != b:
|
||||
log(f" line {i}: committed {a!r}")
|
||||
log(f" re-export {b!r}")
|
||||
shown += 1
|
||||
if shown >= args.max_diff:
|
||||
log(f" ... stopping after {shown} differing lines")
|
||||
break
|
||||
return rc
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
p.add_argument("--url", default=DEFAULT_URL)
|
||||
p.add_argument("--entity", default=DEFAULT_ENTITY)
|
||||
p.add_argument("--token", help="bearer token; else $HASS_TOKEN")
|
||||
p.add_argument("--tz", default=DEFAULT_TZ,
|
||||
help="local timezone that defines a day boundary")
|
||||
p.add_argument("--days", type=int, default=30)
|
||||
p.add_argument("--end-date", help="last local day to export (YYYY-MM-DD); "
|
||||
"default yesterday")
|
||||
p.add_argument("--out-dir", default=".")
|
||||
p.add_argument("--prefix", default="p1-net-")
|
||||
p.add_argument("--manifest", help="path for the manifest JSON")
|
||||
p.add_argument("--gap-s", type=float, default=DEFAULT_GAP_S)
|
||||
p.add_argument("--newline", choices=("crlf", "lf"), default="crlf",
|
||||
help="line terminator; crlf reproduces the committed files")
|
||||
p.add_argument("--timeout", type=float, default=120.0)
|
||||
p.add_argument("--retries", type=int, default=4)
|
||||
p.add_argument("--backoff", type=float, default=5.0)
|
||||
p.add_argument("--delay", type=float, default=2.0,
|
||||
help="pause between days - this is a live HA")
|
||||
p.add_argument("--verify", nargs="+", metavar="FILE",
|
||||
help="re-export these committed days and diff them")
|
||||
p.add_argument("--max-diff", type=int, default=20)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
token = args.token or __import__("os").environ.get("HASS_TOKEN")
|
||||
if not token:
|
||||
print("no token: pass --token or set HASS_TOKEN", file=sys.stderr)
|
||||
return 2
|
||||
if ZoneInfo is None:
|
||||
print("zoneinfo unavailable", file=sys.stderr)
|
||||
return 2
|
||||
tz = ZoneInfo(args.tz)
|
||||
try:
|
||||
if args.verify:
|
||||
return do_verify(args, token, tz)
|
||||
return do_export(args, token, tz)
|
||||
except ReadFailed as exc:
|
||||
print(f"read failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run TARIFF-01's and SAFETY-07's AC 3 over a directory of `p1-net-*.jsonl.gz`.
|
||||
|
||||
AC 3, both cards, same words: *"monthly peak held under the soft limit in
|
||||
>= 95 % of intervals over a 30-day simulation replay."* Both cards merged with
|
||||
it UNMET because two days of capture existed. DATA-01 exported thirty. This is
|
||||
the thing that turns those thirty days into the number.
|
||||
|
||||
cd goodwe_controller
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
WHAT IT MEASURES
|
||||
================
|
||||
Two runs over the same telegrams, one file at a time in date order into a
|
||||
SHARED peak store, so the 15-minute blocks accumulate into a real MONTH rather
|
||||
than into 30 disconnected days:
|
||||
|
||||
UNCONTROLLED the recorded meter as it happened. Nothing was shaving peaks on
|
||||
this house during August 2026, so this is not a simulation of
|
||||
anything - it is what the month did. It is TARIFF-01's AC 3.
|
||||
GUARDED the same month with `peakguard.decide()` in the loop and the
|
||||
battery it asks for modelled, closed-loop: the meter the guard
|
||||
sees is the recorded net MINUS the discharge it obtained. That
|
||||
is SAFETY-07's AC 3.
|
||||
|
||||
Both numbers are `quarters <= peak_cap_w` over `quarters closed`, with the cap
|
||||
read out of config.yaml rather than typed. Nothing here exits non-zero on 95 %:
|
||||
a measurement below the criterion is a finding about the house and the guard,
|
||||
and this tool's job is to report it, not to be graded by it.
|
||||
|
||||
FAITHFULNESS TO THE SUITE
|
||||
=========================
|
||||
The replay loop is `test_peakguard.py` part 6's, with `pack=None` (SoC pinned
|
||||
above the floor, so SAFETY-24's energy floor is not the thing deciding the
|
||||
answer). It is not imported from there because that file runs its whole suite on
|
||||
import. The duplication is checked instead of asserted: pointed at `scenarios/`
|
||||
- the two committed days - this tool must reproduce part 6's published totals,
|
||||
and `--expect` makes that a hard check.
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
|
||||
GAPS
|
||||
====
|
||||
Nothing is interpolated. A hole in the export is a hole here: the shipped
|
||||
`meter_max_age_s` is what decides whether a held reading still counts as
|
||||
coverage, so an hour with no telegrams closes no quarters and contributes to
|
||||
neither numerator nor denominator. `--report-gaps` lists them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
from app.p1 import P1Ingest, make_sample, split_signed # noqa: E402
|
||||
from app.peakguard import decide # noqa: E402
|
||||
from app.tariff import PeakStore, PeakTracker, month_key # noqa: E402
|
||||
|
||||
|
||||
def read_option(key: str):
|
||||
"""The shipped default for `key`, out of config.yaml.
|
||||
|
||||
⚠️ `peak_cap_w:` appears TWICE in config.yaml - once under `options:` with
|
||||
the default, once under `schema:` as `int(500,15000)`. The FIRST match is
|
||||
the options block, which is the number an install actually lands on; taking
|
||||
the last would try to float() the schema line. Same rule as the suites.
|
||||
"""
|
||||
for line in (HERE / "config.yaml").read_text(encoding="utf-8").splitlines():
|
||||
if line.strip().startswith(f"{key}:"):
|
||||
try:
|
||||
return float(line.split(":", 1)[1].strip())
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def read_schema_max(key: str):
|
||||
for line in (HERE / "config.yaml").read_text(encoding="utf-8").splitlines():
|
||||
s = line.strip()
|
||||
if s.startswith(f"{key}:") and "(" in s and ")" in s:
|
||||
try:
|
||||
return float(s[s.index("(") + 1:s.index(")")].split(",")[-1])
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def sample(t: datetime, net_w: float):
|
||||
"""One telegram through the SHIPPED sign derivation - the suites' helper."""
|
||||
imp, exp = split_signed(net_w)
|
||||
return make_sample("ha_signed", imp, exp, phases=1,
|
||||
ingest_ts=t, ingest_mono=t.timestamp())
|
||||
|
||||
|
||||
def rows(path):
|
||||
"""Every sample in one file, in order, with the gaps left as gaps."""
|
||||
with gzip.open(path, "rt", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
row = json.loads(line)
|
||||
yield datetime.fromisoformat(row["t"]), float(row["w"])
|
||||
|
||||
|
||||
def replay(paths, *, guarded: bool, cap_w, max_w, max_hold_s, cycle_s,
|
||||
soc, soc_floor, tmp: Path, gap_s: float):
|
||||
"""test_peakguard.py part 6's loop, over an arbitrary number of days.
|
||||
|
||||
⚠️ ONE STORE, AND AN INGEST THAT IS RESET ONLY AT A REAL DISCONTINUITY.
|
||||
Neither half of that is a detail.
|
||||
|
||||
The STORE is shared across the files because "monthly peak" is what AC 3 is
|
||||
about and a per-file store cannot express it.
|
||||
|
||||
The INGEST is carried across a file boundary when the next file's first
|
||||
telegram follows the previous file's last one within `gap_s`, and rebuilt
|
||||
when it does not. Both failure modes are real and were measured here:
|
||||
|
||||
* carried across a genuine hole, one averager manufactures quarters out of
|
||||
nothing - streamed straight through the three-day gap between the two
|
||||
committed days this reported 285/286 where the suite reports 189/190;
|
||||
* rebuilt at every file, each day's LAST quarter (21:45-22:00 UTC) never
|
||||
closes, because it closes only when a telegram past the boundary
|
||||
arrives and that telegram is in the next file. That is why the suite's
|
||||
two days give 95 quarters each rather than 96, and it silently discards
|
||||
one quarter per day - 30 of them over a month.
|
||||
|
||||
So contiguous days are joined and non-contiguous ones are not, which gives
|
||||
189/190 on the committed pair and a full 96 quarters a day on a run of
|
||||
consecutive days.
|
||||
"""
|
||||
store = PeakStore(str(tmp / f"ac3-{'g' if guarded else 'u'}.db"))
|
||||
tracker = PeakTracker(store, cap_w=cap_w)
|
||||
engaged = blind = saturated = cycles = n = joins = 0
|
||||
wh = 0.0
|
||||
gaps = []
|
||||
months = set()
|
||||
ing = None
|
||||
batt = 0.0 # delivered discharge, W (positive)
|
||||
next_cycle = None
|
||||
prev_t = None
|
||||
for path in paths:
|
||||
for i, (t, w) in enumerate(rows(path)):
|
||||
if i == 0:
|
||||
contiguous = (prev_t is not None
|
||||
and 0 <= (t - prev_t).total_seconds() <= gap_s)
|
||||
if contiguous:
|
||||
joins += 1
|
||||
else:
|
||||
if ing is not None:
|
||||
months |= {month_key(b.start) for b in ing.blocks}
|
||||
ing = P1Ingest(phases=1, max_age_s=max_hold_s)
|
||||
batt = 0.0
|
||||
next_cycle = None
|
||||
prev_t = None
|
||||
if prev_t is not None:
|
||||
dt = (t - prev_t).total_seconds()
|
||||
if dt > gap_s:
|
||||
gaps.append((prev_t, t, dt))
|
||||
wh += batt * dt / 3600.0
|
||||
prev_t = t
|
||||
n += 1
|
||||
ing.submit(sample(t, w - batt))
|
||||
tracker.sync(ing)
|
||||
if next_cycle is not None and t < next_cycle:
|
||||
continue
|
||||
next_cycle = t + timedelta(seconds=cycle_s)
|
||||
cycles += 1
|
||||
d = decide(PeakTracker.project(ing.averager, t), batt_w=batt,
|
||||
cap_w=cap_w, reading=tracker.peak(t),
|
||||
achievable_w=max_w, max_hold_s=max_hold_s, soc=soc,
|
||||
soc_floor=soc_floor, frozen=False)
|
||||
if d.engaged:
|
||||
blind += 1 if d.blind else 0
|
||||
engaged += 0 if d.blind else 1
|
||||
saturated += 1 if d.saturated else 0
|
||||
if not guarded:
|
||||
continue
|
||||
engaged_floor = d.engaged and d.floor_w is not None
|
||||
batt = max(0.0, d.floor_w) if engaged_floor else 0.0
|
||||
if ing is not None:
|
||||
months |= {month_key(b.start) for b in ing.blocks}
|
||||
out = {m: dict(store.blocks_in(m)) for m in sorted(months)}
|
||||
return {"months": out, "samples": n, "cycles": cycles, "joins": joins,
|
||||
"engaged": engaged, "blind": blind, "saturated": saturated,
|
||||
"wh": wh, "gaps": gaps}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
p.add_argument("directory", help="a directory of p1-net-*.jsonl.gz")
|
||||
p.add_argument("--glob", default="p1-net-*.jsonl.gz")
|
||||
p.add_argument("--expect", help="UNDER/TOTAL the uncontrolled run must "
|
||||
"reproduce, e.g. 189/190")
|
||||
p.add_argument("--report-gaps", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
paths = sorted(Path(args.directory).glob(args.glob))
|
||||
if not paths:
|
||||
print(f"no {args.glob} under {args.directory}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
CAP_W = read_option("peak_cap_w")
|
||||
MAX_HOLD_S = read_option("meter_max_age_s")
|
||||
CYCLE_S = read_option("cycle_floor_s")
|
||||
SOC_FLOOR = read_option("peak_soc_floor")
|
||||
LIVE_MAX_W = read_schema_max("max_w")
|
||||
if None in (CAP_W, MAX_HOLD_S, CYCLE_S, SOC_FLOOR, LIVE_MAX_W):
|
||||
print("could not read the shipped options out of config.yaml",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
# The suites' fixture SoC: comfortably above the floor, so SAFETY-24's
|
||||
# energy floor is not what decides an AC 3 that is not about it.
|
||||
SOC_OK = SOC_FLOOR + 30.0
|
||||
|
||||
days = len(paths)
|
||||
span_first = span_last = None
|
||||
print(f"AC 3 - {days} file(s) from {args.directory}")
|
||||
print(f" shipped: peak_cap_w {CAP_W:.0f} W, meter_max_age_s "
|
||||
f"{MAX_HOLD_S:.0f} s, cycle_floor_s {CYCLE_S:.0f} s, "
|
||||
f"max_w schema ceiling {LIVE_MAX_W:.0f} W")
|
||||
|
||||
tmp = Path(tempfile.mkdtemp(prefix="goodwe-ac3-"))
|
||||
kw = dict(cap_w=CAP_W, max_w=LIVE_MAX_W, max_hold_s=MAX_HOLD_S,
|
||||
cycle_s=CYCLE_S, soc=SOC_OK, soc_floor=SOC_FLOOR, tmp=tmp,
|
||||
gap_s=MAX_HOLD_S)
|
||||
u = replay(paths, guarded=False, **kw)
|
||||
g = replay(paths, guarded=True, **kw)
|
||||
|
||||
def flat(run):
|
||||
return [w for m in run["months"].values() for w in m.values()]
|
||||
|
||||
uq, gq = flat(u), flat(g)
|
||||
u_under = sum(1 for w in uq if w <= CAP_W)
|
||||
g_under = sum(1 for w in gq if w <= CAP_W)
|
||||
u_pct = 100.0 * u_under / max(len(uq), 1)
|
||||
g_pct = 100.0 * g_under / max(len(gq), 1)
|
||||
|
||||
print(f" {u['samples']} samples, {len(uq)} closed quarters "
|
||||
f"({len(uq) / 96.0:.1f} days' worth), {days - u['joins']} contiguous "
|
||||
f"run(s) of days")
|
||||
for m in sorted(u["months"]):
|
||||
mu = u["months"][m]
|
||||
mg = g["months"].get(m, {})
|
||||
print(f" month {m}: {len(mu)} quarters, peak "
|
||||
f"{max(mu.values()):.1f} W uncontrolled -> "
|
||||
f"{max(mg.values()) if mg else 0.0:.1f} W guarded")
|
||||
print()
|
||||
print(f" TARIFF-01 AC 3 (UNCONTROLLED, what the month did): "
|
||||
f"{u_under}/{len(uq)} = {u_pct:.2f} % of quarters at or under "
|
||||
f"{CAP_W:.0f} W")
|
||||
print(f" SAFETY-07 AC 3 (GUARDED, peakguard.decide in the loop): "
|
||||
f"{g_under}/{len(gq)} = {g_pct:.2f} %")
|
||||
print(f" guard cycles {g['cycles']}, engaged on the measurement "
|
||||
f"{g['engaged']}, blind {g['blind']}, saturated {g['saturated']}, "
|
||||
f"modelled discharge {g['wh']:.0f} Wh")
|
||||
# AC 3'S WORDS SUPPORT TWO READINGS AND BOTH ARE REPORTED, because they
|
||||
# disagree on this data and picking the flattering one is the failure mode
|
||||
# this whole card exists to correct.
|
||||
u_peak = max(uq) if uq else 0.0
|
||||
g_peak = max(gq) if gq else 0.0
|
||||
print(f" reading A - PERCENTAGE OF INTERVALS under the cap (the reading "
|
||||
f"test_tariff.py part 9 already uses): uncontrolled {u_pct:.2f} %, "
|
||||
f"guarded {g_pct:.2f} %")
|
||||
print(f" reading B - THE MONTHLY PEAK ITSELF under the cap: "
|
||||
f"{u_peak:.1f} W uncontrolled -> {g_peak:.1f} W guarded, against a "
|
||||
f"{CAP_W:.0f} W soft limit "
|
||||
f"({'under' if g_peak <= CAP_W else 'STILL OVER'})")
|
||||
print(f" verdict at the 95 % criterion: uncontrolled "
|
||||
f"{'MEETS' if u_pct >= 95.0 else 'DOES NOT MEET'}, guarded "
|
||||
f"{'MEETS' if g_pct >= 95.0 else 'DOES NOT MEET'} "
|
||||
f"({len(uq) / 96.0:.1f} days of replay)")
|
||||
|
||||
if u["gaps"]:
|
||||
tot = sum(d for _, _, d in u["gaps"])
|
||||
print(f" gaps over {MAX_HOLD_S:.0f} s: {len(u['gaps'])}, "
|
||||
f"{tot:.0f} s total - NOT interpolated; the quarters they fall in "
|
||||
f"simply carry less coverage")
|
||||
if args.report_gaps:
|
||||
for a, b, d in u["gaps"]:
|
||||
print(f" {a.isoformat()} -> {b.isoformat()} {d:.0f} s")
|
||||
|
||||
if args.expect:
|
||||
want_u, want_t = (int(x) for x in args.expect.split("/"))
|
||||
ok = (u_under, len(uq)) == (want_u, want_t)
|
||||
print(f" reproduces test_peakguard.py part 6's published "
|
||||
f"{want_u}/{want_t}: {'yes' if ok else 'NO'} "
|
||||
f"(got {u_under}/{len(uq)})")
|
||||
return 0 if ok else 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user