diff --git a/goodwe_controller/DECISIONS/PRICE-01-own-price-server.md b/goodwe_controller/DECISIONS/PRICE-01-own-price-server.md new file mode 100644 index 0000000..acad88f --- /dev/null +++ b/goodwe_controller/DECISIONS/PRICE-01-own-price-server.md @@ -0,0 +1,95 @@ +# PRICE-01 - the owner's price server as the price source, with fallbacks + +**Status: for review.** Tests: `test_price_service.py`. One option added, +`price_service_token` (password). Follows [[TARIFF-10]], whose section 4 this +card answers for one provider. Design: the owner's price service note of +2026-10-05. Owner go 2026-10-05 and 2026-10-14. + +## What it is + +`price_provider: price_service` reads `GET https://prices.kammenstraatha.duckdns.org/v1/dayahead/BE` +with `Authorization: Bearer `. The answer is the energy-charts layout +plus `covers`, so the existing columnar parser reads it. The price is wholesale +energy only; the framework adds markup and taxes, as for `generic_dayahead`. + +## Rules + +1. **The host is a constant in the plugin**, declared as its only egress host, + HTTPS only. The operator's `price_feed_url` is not read by this provider. + *Cost:* every client talks to the one host, and moving the server is a new + add-on version. *What replaces the per-site setting:* the egress allow-list, + TLS, and a token per client that the server can revoke. +2. **The chain is server, then energy-charts, then the saved prices** + (`app/price_chain.py`). A step counts only if it has a price for *now*: a + server that answers with old prices is skipped like one that is down. A step + down and a step up are each one log line, written when the step changes. The + operator's `price_feed_url` is cleared for the energy-charts step, which is + asked for its own feed. +3. **The saved prices are `/data/price_history.json`**, the file `PriceHistory` + already keeps (the last three Brussels days of what was published). They are + the all-in prices as stored; a change of markup shows from the next fetch + that works. A read from the store is not filed again. +4. **No price for now** means no published price covers the present moment. Not + "none for the coming hours": before about 13:00 tomorrow's prices are not + published on any healthy day, and carried hours are already marked estimated. + It sets one status row (*Prices*), one Home plan line (English and Dutch), + and one notification (*No prices*, English and Dutch), taken down when a + price returns. A start-up before the first refresh finished is not a gap. + The notification is not re-asserted after a Home Assistant core restart + (SAFETY-34 does that for latched safety alarms); the status row stays. + *Open for the owner:* if "coming hours" should mean a look-ahead of N hours, + that is a change to `gate_for()` and `PriceWatch`, not to the chain. +5. **The maintenance charge gate** (owner's decision, 2026-10-14): the charge may + run when the *current 15-minute price is at or below today's average*, today + being the Brussels day, computed from the provider's prices. This is the rule + the Home Assistant pair gave (`price_service.yaml`), so a site that moves from + the pair to the provider does not change when the maintenance charge runs. + It is not `TariffPlan.cheap_now()` (strictly below the mean of the firm + hours of the whole plan). This replaces, for this provider only, + TARIFF-10 section 4: its three reasons are answered as follows. (1) Two + writers: the pair beside a provider stays refused; `static_profile` beside + `price_service` is accepted and no longer writes `cheap`. (2) An expired or + absent plan shuts the gate (the charge waits), as an unavailable HA sensor + did. (3) The basis is the one above, not `cheap_now()`. +6. **The token is a secret.** `password?` in the schema; hidden on the settings + page (`uilayout.HIDDEN`), which can keep `price_service` but not choose it; + never put in a message by the plugin; taken out of any reason text by + `price.scrub()`; the real transport reports only a status or an exception's + name. validate.py refuses `price_service` with no token, and a token with a + space or line break. + +## Where the gate equals the Home Assistant pair, and where it does not + +Equal: `test_price_service.py` section 8 feeds the same day both ways, to the +rules in `price_service.yaml` (price at the quarter, average of today's +quarters rounded to 0.00001 EUR/kWh, `<=`), over 6 days including both +clock-change days, at three instants in every period, and through a real +`Controller` for a whole day. Every answer is the same. + +Not equal, by design or by arithmetic: + +- The comparison is made on all-in prices. With a flat taxes figure and a + markup and taxes of at most five decimals the offset cancels and the answer + is the same. With `price_taxes_profile` (an hour-varying levy) the all-in + prices differ by hour, so the answer can differ from the wholesale comparison + the HA sensors made. A site on that profile should check before moving. +- A quarter within 0.00001 EUR/kWh of the average can differ if markup or taxes + have more than five decimals. +- The plan is retired when no refresh has succeeded for 90 minutes + (`TARIFF_MAX_AGE_S`) and the gate shuts then. How long the HA pair kept an + answer after its REST sensor stopped updating was not measured here. +- The average is over the firm prices the provider holds for today. After a + restart in the middle of a day with only the saved part of today, it is the + average of that part. + +## Not covered + +- **No real answer of the server was seen.** No token was available. The + layout is the design's; `test_price_service.py` uses a mock and, for the + parser, the real energy-charts capture reshaped with the design's extra keys. + The first capture from the live server (rig test) should replace the mock. +- The 429 limit (about 60 calls an hour) is not approached: one fetch per + 30 minutes, and at most two requests when the server fails. +- Battery modes read the plan as for `generic_dayahead` and were not changed. +- The `price_service` REST and template sensors in Home Assistant are not + touched; they stop being read when the pair is cleared. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index 3736fde..db3102d 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -222,6 +222,7 @@ The label and description each setting shows on the Configuration tab, in tab or | Tariff | Your own injection formula | `price_export_formula` | Optional. The injection formula from your tariff card, in the same shape as your own tariff formula. Leave it empty to use the flat injection price. | | Tariff | Price CSV file | `price_csv_path` | For manual_csv only. Where in /share the add-on reads your price file. | | Tariff | Price feed URL override | `price_feed_url` | Leave it empty to use the source's own feed. An override is sent exactly as typed and often returns only one day. | +| Tariff | Price service token | `price_service_token` | For the price service only. The token you were given for it. Leave it empty with any other price source. | | Tariff | Price look-ahead (hours) | `price_horizon_h` | How many hours of prices to ask the source for. | | Tariff | Price request timeout (s) | `price_timeout_s` | How long one request to the price source may take before it is given up. | | Tariff | Current price entity (older) | `price_now_entity` | The older way to get a price, used only to charge in a cheap window during maintenance. Do not combine it with a dynamic price source. | @@ -290,6 +291,7 @@ When the add-on refuses to start, each reason names the setting by its label and - **Fixed or day/night tariff** (`static_profile`): the periods must cover every day type exactly once with no gap and no overlap, and a gap is refused rather than filled. A window 22:00-07:00 on mon-fri does not cover Saturday morning, so a weekend row is required. See [Static day/night and fully-fixed tariffs](#static-daynight-and-fully-fixed-tariffs-static_profile). - **Your own tariff and injection formula** (`price_formula`, `price_export_formula`): type the card's numbers in the card's units. The `card` block is required on the tariff formula: the add-on recomputes the card's example price and refuses to start if it does not come out. Injection is VAT-exempt in Belgium, so `vat_pct` there is normally 0. See [Your own tariff formula](#your-own-tariff-formula-custom_formula) and [`price_export_formula`, field by field](#price_export_formula-field-by-field). - **Price feed URL override** (`price_feed_url`): only a host the chosen source declares is accepted. Nothing is added to the URL, not even the date range, so an override often returns one day and the rest of the plan is carried over and excluded from trading. The log warns on every refresh. +- **Price service token** (`price_service_token`): a password field, shown masked on the Configuration tab and never drawn on the add-on page. It is needed with `price_provider: price_service`, and the add-on refuses to start without it. It never appears in the log, the status page, a refusal or a notification. See [`price_service`](#price_service--your-own-price-server-and-what-it-falls-back-to). - **Current and average price entities** (`price_now_entity`, `price_avg_entity`): kept because removing them would take the maintenance cycle's only price signal away from a live site. - **Trade on the price spread** (`arbitrage_enabled`): off as shipped, on purpose, because it moves energy against the meter by design. It needs a price source and a P1 meter, and the add-on refuses combinations that cannot work. - **Battery cycle cost** (`cycle_cost_threshold_eur`): wear plus conversion losses. It is not derived from a measurement: 0.10 is a product decision still open with the owner (OPT-08). At this value a fixed day/night site does not trade, since a realistic Belgian day/night spread is about 0.08 EUR/kWh. @@ -2198,20 +2200,24 @@ does, and only after you have read why. > `static_profile` below needs no network at all, so its cheap window is > computed on every cycle and reaches the maintenance cycle's grid top-up > whether or not arbitrage is armed. ⚠️ **And it is still the only source of -> that window.** A provider-backed plan now exists on every configured site, -> but *Cheap window* and the maintenance cycle's grid top-up read -> `static_profile` and nothing else — a deliberate limit, recorded with what it -> would take to change in `DECISIONS/TARIFF-10-fetching-is-not-trading.md`. +> that window, with one exception.** A provider-backed plan exists on every +> configured site, but *Cheap window* and the maintenance cycle's grid top-up +> read `static_profile` and nothing else — a deliberate limit, recorded with +> what it would take to change in +> `DECISIONS/TARIFF-10-fetching-is-not-trading.md`. The exception is +> `price_service` (PRICE-01): with it, the window comes from the provider's own +> prices, as described under [`price_service`](#price_service--your-own-price-server-and-what-it-falls-back-to). | option | meaning | |---|---| -| `price_provider` | `off` (default) or one of the three plugins below | +| `price_provider` | `off` (default) or one of the plugins below | | `price_markup_eur_kwh` | Your supplier's margin, EUR/kWh. **Added only when the chosen plugin's feed does not already contain it** | | `price_taxes_eur_kwh` | Distribution, levies and VAT, EUR/kWh. Same rule. Defaults to **0.11736** — a real Flemish figure, and **yours is probably different**: see *What `price_taxes_eur_kwh` is made of* below | | `export_price_eur_kwh` | The **flat injection** price, EUR/kWh. May be negative — see below | | `price_export_formula` | The **injection formula**, if your contract has one. Same shape as `price_formula`, its own coefficients, its own `vat_pct` — see below | | `price_csv_path` | Where `manual_csv` reads its file | | `price_feed_url` | Override the plugin's default feed. Used **exactly as written** — the add-on appends nothing to it, including the date range, so an override gets whatever horizon that URL returns. Only a host the plugin itself declares is accepted | +| `price_service_token` | The token for `price_service`. A password field. Empty with any other source | | `price_horizon_h` | How far ahead to ask for, hours | | `price_timeout_s` | Per-request timeout | | `static_profile` | A static day/night or fully-fixed tariff. Empty by default — see below | @@ -2224,6 +2230,7 @@ arbitrage decision wrong by that amount, in a direction nothing reports. | plugin | product | the published figure contains | the add-on adds | |---|---|---|---| | `generic_dayahead` | day-ahead | energy only (wholesale EPEX BE) | markup **and** taxes | +| `price_service` | day-ahead | energy only (wholesale, from your own price server) | markup **and** taxes | | `custom_formula` | day-ahead | energy **and** your contract's markup — your card's `Tariefformule` line | taxes | | `manual_csv` | day-ahead | everything, all-in | nothing | @@ -2253,6 +2260,76 @@ from. Any alias would have to invent the numbers your battery is priced against. Eneco's dynamic product is *Zon & Wind Dynamisch*, and it is indexed to Belpex/EPEX **day-ahead**, not to imbalance.) +#### `price_service` — your own price server, and what it falls back to + +*For installers: PRICE-01. The decision is in +`DECISIONS/PRICE-01-own-price-server.md`.* + +`price_provider: price_service` reads the day-ahead prices for Belgium from the +owner's price server, `https://prices.kammenstraatha.duckdns.org/v1/dayahead/BE`. +The host is fixed in the add-on and only HTTPS is used. The token goes in +`price_service_token` and is sent as `Authorization: Bearer `. The server +answers the energy-charts layout, so the same parser reads it. The price is the +wholesale energy price and nothing else, so the add-on adds your markup and +taxes exactly as for `generic_dayahead`. + +**Where the prices come from.** Each step is used only if it has a price for +now. A server that answers with last night's prices is treated as down. + +| Step | Source | When it is used | What the log says (once per change) | +|---|---|---|---| +| 1 | The price server | It answers, with a price for now | `The price service answers again.` on the way back. | +| 2 | energy-charts.info | Step 1 fails: down, timeout, refused token (401), error (5xx), a damaged answer, or no price for now | `The price service does not answer, so prices come from energy-charts.` Or `The price service refused the token, so prices come from energy-charts.` | +| 3 | The prices saved in `/data/price_history.json` | Steps 1 and 2 fail | `No price source answers, so the add-on uses the prices it saved.` | +| none | Nothing | No saved price covers now either | `No price source answers, and no saved price covers now.` | + +The saved prices are the last answers that worked, kept on disk, so a site +whose server and energy-charts are both down for a day goes on planning from +what it has. It keeps the days the history holds (today, tomorrow once it was +fetched, and a day back). They are the all-in prices as stored: a change of +markup or taxes shows from the next fetch that works. Going back up the chain +is logged once as well: `Prices come from energy-charts again.` and `The price +service answers again.` The refresh runs every 30 minutes. + +**When there is no price for now (step none).** One status line, `Prices: No +prices since 00:00, maintenance waits.` (on the add-on page; the Home screen says +the same in English and Dutch), and one notification, *No prices*. Both go when a +source has a price for now again. "Since" is the moment the last price ran out. +Before the first refresh has finished there is no line, so a start-up is not a +gap. Control is not touched: the battery follows the meter as ever, battery +modes and trading read what prices there are, and only the maintenance charge +waits. + +**The maintenance charge and the cheap window.** With this provider the +maintenance charge gate reads the provider's prices and no Home Assistant +entity. It keeps the rule the Home Assistant pair gave: the charge may run +when the **current 15-minute price is at or below today's average**, today +being the Brussels day. Moving a site from the pair to `price_service` does not +change when the charge runs. Where the two can differ is listed in the decision +record: the Home Assistant average was rounded to 0.00001 EUR/kWh, and so is +this one before it is compared. With no price for now the gate is shut, as it +was when an entity was unavailable. *Cheap window* on the status page shows this +answer. This is not the *cheap hours* rule of battery modes and not +`cheap_now()`. + +**Moving a site from the Home Assistant price sensors to `price_service`.** +Do this on the Configuration tab, in one save, and restart once: + +1. Set **Dynamic price source** (`price_provider`) to `price_service`. +2. Paste the token into **Price service token** (`price_service_token`). +3. **Clear** **Current price entity** (`price_now_entity`) and **Average price + entity** (`price_avg_entity`). The add-on refuses to start with a price + source and these two together. +4. Keep **Supplier markup** and **Distribution, levies and VAT** as they are + for a wholesale price. If the Home Assistant sensors already included them, + the figures were never in these two options, so nothing else changes. +5. Keep **Fixed or day/night tariff** (`static_profile`) if you have one. It + stays the fallback for trading when no source answers. +6. Remove the `price_service` REST and template sensors from Home Assistant + when you are satisfied. They are no longer read. + +To go back: put the two entities back, set the source to `off`, and restart. + #### So what should a Belgian dynamic-tariff customer do today? **Honestly: `generic_dayahead` gets you close, and you should know exactly how @@ -2809,6 +2886,9 @@ uses the result as a cheap-window hint for the maintenance charge. They are — configure two of them and the add-on refuses to start, naming the conflict, because the add-on reads exactly one and the loser is silent. Use one. +To move a site off them onto the owner's price server, see [`price_service`](#price_service--your-own-price-server-and-what-it-falls-back-to): +it lists which options to set and which to clear. + ### The three buckets — which price components can move a decision *For installers: card TARIFF-17.* @@ -3282,6 +3362,15 @@ days before it tries again, so it does not empty the battery every night. To try sooner, start a cycle on the add-on page. If it keeps happening, check that the battery follows its commands and that the cheap hours leave time to charge. +### No prices + +No price source has a price for the present moment: not the price server, +not energy-charts, and nothing saved. The monthly battery cycle waits, because +it will not charge from the grid without a price to judge by. Control of the +battery goes on as before. Check that the add-on can reach the internet and that +the price service token is right. The notification goes by itself when a price +is back. + ### For installers: notifications This part is for installers. The texts live in `app/messages.py` and follow @@ -3302,6 +3391,7 @@ default `info` hides it. | *name* is not delivering | alarm | `goodwe_not_delivering_` | SAFETY-42 | | Meter reading may be stuck | alarm | `goodwe_meter_frozen` | SAFETY-18 | | Battery cycle did not finish | warning | `goodwe_maint_failed_` | MAINT-13 | +| No prices | warning | `goodwe_no_prices` | PRICE-01 | **The Details link.** Home Assistant 2026.2 and newer show add-on pages at `/config/app//documentation`; 2026.5 removed the old @@ -3499,6 +3589,7 @@ for a new install*. | Direction check, Site direction check | Whether each inverter's power goes the way the add-on asks. *Opposite for 12 of 60 seconds* counts toward a stop. When it stops, the inverter, or every inverter, is set to 0 W until you start control. | | Check for inverters working against each other, Inverters worked against each other | Only with several inverters. See *Inverters working against each other*. | | Delivery check, *name* is not delivering | Only with several inverters. See *An inverter is not delivering*. | +| Prices | Only while no price source has a price for now: *No prices since 00:00, maintenance waits.* See *No prices* under Notifications. | | Solar production | Your solar production. Sources are numbered in the order of “Solar production entities, per inverter”. A strongly negative reading by day means that entity reads the wrong way round: add it to “Reversed-sign solar entities”. | ### For installers: status page diff --git a/goodwe_controller/app/homeview.py b/goodwe_controller/app/homeview.py index 0e2ebfa..b8fa8ae 100644 --- a/goodwe_controller/app/homeview.py +++ b/goodwe_controller/app/homeview.py @@ -582,6 +582,8 @@ def build_plan(c, now, shadow, running, held=False, settled=True): cyc, waits = maintenance_runs(c, day0, day1, now, shadow) runs += cyc lines += waits + if getattr(c, "price_gap", False): + lines.append(line_no_prices(c.price_gap_since, now)) runs += peak_run(c, day0, day1, now, shadow, limit_w) if c.p1_enabled: # The guard's ceiling: the cap, or this month's peak once above it. @@ -1268,6 +1270,13 @@ def line_cycle_waits(name) -> str: return _t("cycle_waits", name=name) +def line_no_prices(since, now) -> str: + """PRICE-01: no price covers now. `since` is an instant or None.""" + if since is None: + return _t("no_prices_yet") + return _t("no_prices_since", at=messages.since_text(brussels(since), brussels(now))) + + def line_peak_limit(limit_w, shadow) -> str: return _w(shadow, "step_in", kw=_kw(limit_w)) @@ -1387,6 +1396,12 @@ def _samples() -> list: ("plan", run_house_why()), ("plan", run_house_why(True)), ("plan", run_house_amount()), ("plan", line_cycle_waits("Battery B")), + ("plan", line_no_prices(None, datetime(2026, 10, 14, 9, 0, + tzinfo=timezone.utc))), + ("plan", line_no_prices(datetime(2026, 10, 13, 22, 0, + tzinfo=timezone.utc), + datetime(2026, 10, 14, 9, 0, + tzinfo=timezone.utc))), ("plan", line_prices(0.16, {"eur_kwh": 0.14, "label": "13:00"}, {"eur_kwh": 0.36, "label": "19:00"})), ("plan", line_prices(None, None, None)), diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index cb71db6..4a897ae 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -90,6 +90,7 @@ from .p1 import P1Ingest, build_source, is_enabled from .price import Absence, Uncertainty, brussels, price_option, provider_configured from .price import required_spread as price_required_spread from .price import uncertainty_of as price_uncertainty_of +from .price_chain import PriceWatch, gate_for, uses_chain from .price_history import PriceHistory from .static_tariff import resolve_tariff, static_gated, static_status from .tariff import PeakStore, PeakTracker, next_quarter @@ -1202,6 +1203,16 @@ class Controller: self.peak_fc = None self.cheap = False self._price_gated = False # MAINT-13: is a price gate set up + # PRICE-01. Which step of the price chain the prices come from (logged + # once per change), the last series that was published (so a gap can say + # since when), and the gap itself: True while a refresh has finished and + # no published price covers now. The notification is raised once per + # gap and taken down when it ends. + self.price_watch = PriceWatch() + self._last_published = None + self.price_gap = False + self.price_gap_since = None + self._price_gap_announced = False self._gated_key = None # MAINT-13: (options, hour) of the cache # TARIFF-03. One line of prose for the status page saying WHICH tariff # window we are in and why it is or is not cheap - so "Cheap window: @@ -3435,6 +3446,12 @@ class Controller: self.cheap = (now_p is not None and avg_p is not None and now_p <= avg_p) self._price_gated = True self.tariff_words = "Prices from Home Assistant entities" + elif uses_chain(o): + # PRICE-01. A provider with fallbacks gives the maintenance gate + # its own prices: the current 15-minute price against today's + # average, as the Home Assistant pair did. Still no I/O here - the + # series is the one the background refresh stored. + self._provider_gate(datetime.now(timezone.utc)) else: # TARIFF-03. §9.2 makes a static day/night tariff a first-class # product, and its required EMS handling is TIME-WINDOW SHIFTING. @@ -3998,6 +4015,35 @@ class Controller: inv.refusal_note())] if n] + def _provider_gate(self, now: datetime) -> None: + """PRICE-01: `cheap`, the gate and the gap from the standing plan. + + The gate exists whenever the provider does: no price for now is a + closed gate, so the charge waits and does not run unpriced. The gap + flag stays False until the first refresh has finished, so a start-up + is not a gap. + """ + plan = self.fresh_tariff_plan(now) + published = None if plan is None else plan.published + if published is not None: + self._last_published = published + gate = gate_for(published, now) + self.cheap, self._price_gated = gate.cheap, True + self.tariff_words = gate.words + self.price_gap = (not gate.covered) and self.tariff_at is not None + last = self._last_published + if gate.covered or last is None: + self.price_gap_since = None + elif last.covers(now): + # The last series still covers now, but no refresh has made it + # current for TARIFF_MAX_AGE_S: the refresh task has stopped. The + # gap began when the plan was retired. + self.price_gap_since = ( + None if self.tariff_at is None + else self.tariff_at + timedelta(seconds=TARIFF_MAX_AGE_S)) + else: + self.price_gap_since = last.last_end_before(now) + def _static_gated(self, o: dict, now: datetime) -> bool: """`static_gated()`, worked out once an hour: it builds the same month-long plan as `static_status()` (about 20 ms with a profile), and @@ -5536,6 +5582,11 @@ class Controller: # is not, and test_freeze.py part 5 fails on exactly that. await self.check_meter_frozen() + # --- PRICE-01: no price for now, said once and taken down once -------- + # Not a safety check, so it is one of the cheap ones: an attribute test + # and, on a change only, one Home Assistant call. + await self.check_price_gap() + # --- SAFETY-31's alarm, and its boundary is the LATCH, not a number --- # Unconstrained where the two above are not: it grades # `self.circ_wd.tripped`, which was advanced near the top of this method @@ -6152,6 +6203,33 @@ class Controller: await self.hass.call("persistent_notification", "create", messages.not_delivering(inv.name, slug).data()) + async def check_price_gap(self) -> None: + """PRICE-01. One notification when no price covers now, gone when one does. + + `price_gap` is worked out on the tick from the stored plan. A gap + raises the notification once: the flag is set BEFORE the await, for + `check_circulating()`'s reason (a Home Assistant call can raise or + hang, and a retry every cycle would be a storm). When the gap ends the + same notification is taken down. Control is not touched either way: the + maintenance charge waits because its gate is shut (`_provider_gate`), + and nothing here decides that. + """ + if self.price_gap and not self._price_gap_announced: + self._price_gap_announced = True + await self.hass.call("persistent_notification", "create", + messages.no_prices(self._gap_text()).data()) + elif not self.price_gap and self._price_gap_announced: + self._price_gap_announced = False + await self.hass.call("persistent_notification", "dismiss", + messages.prices_back().data()) + + def _gap_text(self) -> str: + """`00:00` for a gap that began today, `13 Oct at 21:00` before that.""" + if self.price_gap_since is None: + return "" + return messages.since_text(brussels(self.price_gap_since), + brussels(datetime.now(timezone.utc))) + async def check_meter_frozen(self) -> None: """SAFETY-18. Alarm on a meter that answers forever without changing. @@ -6286,7 +6364,7 @@ class Controller: try: plan = await resolve_tariff( self.o, now, now + timedelta(hours=horizon), now=now, - history=self.price_history) + history=self.price_history, watch=self.price_watch) standing = self.fresh_tariff_plan(now) # Three terms, and each of them is the only false one in some # check: the feed answered (replace), nothing current stands @@ -7363,6 +7441,9 @@ class Controller: ("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")), ("Cheap window", "yes" if self.cheap else "no"), ("Tariff", self.tariff_words), + # PRICE-01: only while no price covers now. + *((("Prices", messages.row_no_prices(self._gap_text())),) + if self.price_gap else ()), # FCST-04. WHAT §8.3 HAS TO CARRY FORWARD, which is the difference # between a 48-hour plan and a 12-hour one on a weekday morning. # It is its own row rather than a clause of "Tariff" because it diff --git a/goodwe_controller/app/messages.py b/goodwe_controller/app/messages.py index 25523b7..3a986d1 100644 --- a/goodwe_controller/app/messages.py +++ b/goodwe_controller/app/messages.py @@ -129,6 +129,8 @@ S_NOT_DELIVERING = "An inverter is not delivering" T_METER_FROZEN = "Meter reading may be stuck" T_MAINT_FAILED = "Battery cycle did not finish" T_MAINT_FAILED_NL = "Batterijcyclus niet afgerond" +T_NO_PRICES = "No prices" +T_NO_PRICES_NL = "Geen prijzen" ID_LOOP_BOUND = "goodwe_integrator_bound" ID_OUTPUT_CLAMP = "goodwe_output_clamp" @@ -138,6 +140,7 @@ ID_CIRCULATING = "goodwe_circulating_power" ID_NOT_DELIVERING = "goodwe_not_delivering_{slug}" ID_METER_FROZEN = "goodwe_meter_frozen" ID_MAINT_FAILED = "goodwe_maint_failed_{slug}" +ID_NO_PRICES = "goodwe_no_prices" # Which limit cut a command (clamp.ClampResult.bound, plus the loop's own). # The setting is named by its label, "Inverter rating (W)". @@ -314,6 +317,51 @@ def maint_failed(kind: str, days: int, unit: str = "", slug: str = "") -> Notice T_MAINT_FAILED, en, body_nl=f"{T_MAINT_FAILED_NL}. {nl}") +def since_text(since, now) -> str: + """PRICE-01: `00:00` for a moment today, `13 Oct at 21:00` for an earlier + day. Both arguments are already in the site's zone. Empty for no moment.""" + if since is None: + return "" + return f"{since:%H:%M}" if since.date() == now.date() else when(since) + + +def no_prices(since_text: str = "") -> Notice: + """PRICE-01: no source has a price for now. Said once, gone with the gap.""" + if since_text: + en = (f"No prices since {since_text}, so the monthly battery cycle " + f"waits until they return.") + nl = (f"Geen prijzen sinds {since_text}, dus de maandelijkse " + f"batterijcyclus wacht tot ze terug zijn.") + else: + en = ("No prices have come in yet, so the monthly battery cycle waits " + "until they do.") + nl = ("Er zijn nog geen prijzen binnen, dus de maandelijkse " + "batterijcyclus wacht tot ze er zijn.") + return Notice(ID_NO_PRICES, WARNING, T_NO_PRICES, en, + body_nl=f"{T_NO_PRICES_NL}. {nl}") + + +@dataclass(frozen=True) +class Dismiss: + """The payload that takes one notification down again.""" + id: str + + def data(self) -> dict: + return {"notification_id": self.id} + + +def prices_back() -> Dismiss: + """PRICE-01: the 'No prices' notification, taken down when a price is back.""" + return Dismiss(ID_NO_PRICES) + + +def row_no_prices(since_text: str = "") -> str: + """PRICE-01: the status row while no price covers now.""" + if since_text: + return f"No prices since {since_text}, maintenance waits." + return "No prices yet, maintenance waits." + + def row_maint_waits(until_text: str) -> str: """MAINT-13: the status row while a failed cycle waits.""" return f"Waits until {until_text} after a cycle that did not finish." @@ -1186,6 +1234,9 @@ def status_samples() -> list: True)), ("row_circulating_check", GREEN, row_circulating_check(0, 0, 60)), ("row_maint_waits", AMBER, row_maint_waits("2026-10-13")), + ("row_no_prices", AMBER, row_no_prices("00:00")), + ("row_no_prices", AMBER, row_no_prices("13 Oct at 21:00")), + ("row_no_prices", AMBER, row_no_prices()), ] @@ -1218,6 +1269,9 @@ def samples() -> list: meter_frozen(90), maint_failed("charge", 7), maint_failed("drain", 7, "inverter-b", "inverter_b"), + no_prices("00:00"), + no_prices("13 Oct at 21:00"), + no_prices(), ] @@ -1250,6 +1304,7 @@ LABELS = { "price_formula": "Your own tariff formula", "price_export_formula": "Your own injection formula", "price_csv_path": "Price CSV file", + "price_service_token": "Price service token", "price_now_entity": "Current price entity", "price_avg_entity": "Average price entity", "arbitrage_enabled": "Trade on the price spread", diff --git a/goodwe_controller/app/price.py b/goodwe_controller/app/price.py index b1c1ca6..65fd8c3 100644 --- a/goodwe_controller/app/price.py +++ b/goodwe_controller/app/price.py @@ -750,6 +750,18 @@ class _Series(PriceRead): except Absence: return False + def last_end_before(self, instant: datetime): + """When the last price that has run out by `instant` stopped covering, + or None if no price has (PRICE-01: "no prices since 00:00"). + + Each point's end is the one `_cover_table()` worked out, so a hole in + the series is reported from where the hole starts and not from the end + of the day. + """ + u = instant.astimezone(timezone.utc) + ends = [end for _start, end, _p in self._cover if end <= u] + return max(ends) if ends else None + def window(self, since: datetime, until: datetime) -> tuple: """Points whose START falls in [since, until). May legitimately be empty. @@ -906,6 +918,10 @@ PRICE_OPTION_DEFAULTS = { "export_price_eur_kwh": 0.04, # §14 tariffs.export_price_eur_kwh (AC4) "price_csv_path": "/share/goodwe_prices.csv", "price_feed_url": "", + # PRICE-01. The bearer token of the owner's price server. A secret: it is + # in SECRET_OPTIONS below, so no text built from an error reaches a log, + # the status page or a notification with it in. + "price_service_token": "", "price_horizon_h": 48, "price_timeout_s": 10, # TARIFF-03. §14 `tariffs.static_profile: null`. A JSON object describing a @@ -992,6 +1008,26 @@ FRAMEWORK_KEYS = frozenset({ }) +# PRICE-01. Options whose VALUE must never be printed. `scrub()` removes them +# from any text the price code builds out of an error. +SECRET_OPTIONS = frozenset({"price_service_token"}) + + +def scrub(text, opts) -> str: + """`text` with every secret option's value replaced by `***`. + + The price code builds reasons out of library errors, and a library may put + a request header into its error text. Taking the value out here is the + last line of defence; the plugin also never puts it in a message itself. + """ + out = str(text) + for key in SECRET_OPTIONS: + secret = str((opts or {}).get(key) or "").strip() + if len(secret) >= 4: + out = out.replace(secret, "***") + return out + + class PluginOptions: """The options dict, minus the keys only the framework may read. @@ -1216,6 +1252,13 @@ class PriceProvider(ABC): id = "?" + # PRICE-01. The ids to try, in order, when this source cannot answer, and + # whether the last answer is kept for a day without any source. Empty and + # False for every plugin that existed before, which is what keeps them + # exactly as they were: `price_chain.py` is not used for them at all. + FALLBACKS: tuple = () + KEEPS_LAST_GOOD: bool = False + @classmethod @abstractmethod def capabilities(cls) -> ProviderCapabilities: @@ -1533,19 +1576,54 @@ def assemble(caps: ProviderCapabilities, raw, markup: float, # Transport - one place, HTTPS only, host checked against the declaration # --------------------------------------------------------------------------- # -async def https_get(url: str, timeout_s: float) -> bytes: +async def https_get(url: str, timeout_s: float, headers=None) -> bytes: """The default transport. §16: outbound is allow-listed and TLS'd. Imported lazily so `manual_csv` - the offline fallback - never needs aiohttp to be importable at all. AC3 says it works fully offline; that includes a machine where the HTTP stack is missing. + + PRICE-01. `headers` is None for every plugin but `price_service`, and with + None this is the code it has always been, byte for byte. With headers it + answers in the SAFE words of `status_words()` and never lets a library + error through: an error from a request that carried a token is the one + place a token could be printed by somebody else's code. """ import aiohttp t = aiohttp.ClientTimeout(total=float(timeout_s)) - async with aiohttp.ClientSession(timeout=t) as s: - async with s.get(url) as r: - r.raise_for_status() - return await r.read() + if headers is None: + async with aiohttp.ClientSession(timeout=t) as s: + async with s.get(url) as r: + r.raise_for_status() + return await r.read() + try: + async with aiohttp.ClientSession(timeout=t) as s: + async with s.get(url, headers=headers) as r: + if r.status != 200: + raise SourceUnavailable(status_words(r.status)) + return await r.read() + except SourceUnavailable: + raise + except Exception as e: # noqa: BLE001 + raise SourceUnavailable( + f"the price service did not answer ({type(e).__name__})") + + +# PRICE-01. The one phrase a refused token is reported with. The fallback chain +# reads it to tell "the owner must act" from "the server is down", so it is a +# constant both sides import. +AUTH_REFUSED = "the price service refused the token" + + +def status_words(status: int) -> str: + """What an HTTP status from the price service is called. No header text.""" + if status in (401, 403): + return f"{AUTH_REFUSED} ({status})" + if status == 429: + return "the price service says too many requests (429)" + if status >= 500: + return f"the price service has an error ({status})" + return f"the price service answered {status}" def check_egress(url: str, caps: ProviderCapabilities) -> str: @@ -1870,6 +1948,91 @@ class GenericDayAhead(PriceProvider): provisional_default=False) +@register +class PriceService(PriceProvider): + """The owner's own price server (PRICE-01), COLUMNAR, in EUR/MWh. + + SOURCES: energy + The framework ADDS: markup, taxes + + The server answers the energy-charts layout (`unix_seconds`, `price`, + `unit`) plus `covers`, the complete Brussels days it holds. The price is + the wholesale energy price and nothing else, so markup and taxes are added + by the framework exactly as for `generic_dayahead`. + + The host is fixed here, HTTPS only, and a bearer token (the option + `price_service_token`) goes in the `Authorization` header. The token is not + put into any message this class builds. + + `FALLBACKS` names the next source down. `app/price_chain.py` walks it and + then the stored prices; this class only fetches. + """ + + id = "price_service" + + HOST = "prices.kammenstraatha.duckdns.org" + URL = "https://prices.kammenstraatha.duckdns.org/v1/dayahead/BE" + FALLBACKS = ("generic_dayahead",) + KEEPS_LAST_GOOD = True + + @classmethod + def capabilities(cls): + return ProviderCapabilities( + product=PriceProduct.DAY_AHEAD, + sources=frozenset({Component.ENERGY}), + egress_hosts=(cls.HOST,)) + + async def fetch(self, opts, since, until, transport): + token = str(opts.get("price_service_token") or "").strip() + if not token: + raise SourceUnavailable("no token is set for the price service") + if any(c.isspace() or not c.isprintable() for c in token): + raise SourceUnavailable( + "the price service token has a space or a line break in it") + check_egress(self.URL, self.capabilities()) + try: + body = await transport( + self.URL, float(opts.get("price_timeout_s")), + headers={"Authorization": "Bearer " + token}) + except SourceUnavailable: + raise + except Exception as e: # noqa: BLE001 + raise SourceUnavailable( + f"the price service did not answer ({type(e).__name__})") + points = _points_from_columnar_json(body, expect_unit="EUR / MWh", + scale=0.001, + provisional_default=False) + covers = _covered_days(body) + if covers is None: + return points + whole = [p for p in points + if brussels(p.start).date().isoformat() in covers] + if not whole: + raise SourceUnavailable( + "the price service sent no complete day") + return whole + + +def _covered_days(body): + """The `covers` list of a price service answer as a set, or None if absent. + + A day the server does not list as complete is not used: an hour missing + inside a day is a hole, and a hole is not a free hour. + """ + try: + text = body.decode("utf-8") if isinstance(body, (bytes, bytearray)) \ + else str(body) + covers = json.loads(text).get("covers") + except (ValueError, UnicodeDecodeError, AttributeError): + return None + if covers is None: + return None + if not isinstance(covers, list) or \ + not all(isinstance(d, str) for d in covers): + raise SourceUnavailable("the price service sent a bad list of days") + return set(covers) + + @register class ManualCsv(PriceProvider): """A file on disk. AC3: the documented fallback, fully offline. @@ -2169,9 +2332,10 @@ async def load_prices(opts: dict, since: datetime, until: datetime, except FrameworkOption: raise # a plugin bug, not a site condition except SourceUnavailable as e: - return PriceRead.no_data(f"{pid}: {e}") + return PriceRead.no_data(scrub(f"{pid}: {e}", opts)) except Exception as e: # noqa: BLE001 - see the docstring - return PriceRead.no_data(f"{pid}: {type(e).__name__}: {e}") + return PriceRead.no_data( + scrub(f"{pid}: {type(e).__name__}: {e}", opts)) _emit_startup_preview(plugin, pid, points) return PriceRead.series(points, caps.product, pid, diff --git a/goodwe_controller/app/price_chain.py b/goodwe_controller/app/price_chain.py new file mode 100644 index 0000000..ce89325 --- /dev/null +++ b/goodwe_controller/app/price_chain.py @@ -0,0 +1,217 @@ +"""PRICE-01 - where the prices come from when the first source does not answer. + +A provider that names `FALLBACKS` (app/price.py) is read through this module +instead of straight from `load_prices()`. The order is fixed: + + 1. the provider itself (the owner's price server) + 2. each id in its `FALLBACKS` (energy-charts.info) + 3. the prices the history store kept from the last fetch that worked + +A step counts only if its answer holds a price for NOW. A server that answers +200 with yesterday's prices is as useless to a battery as one that is down, so +it is skipped, with the same log line. + +Every other provider never comes here (`uses_chain()` is False for it), so +`generic_dayahead`, `manual_csv`, `custom_formula` and `off` take the road they +always took. + +THE GATE. `gate_for()` is the maintenance charge rule for a site on such a +provider: the current 15-minute price is at or below today's average. It is the +rule the Home Assistant pair (`price_now_entity`, `price_avg_entity`) gave, read +from the provider's own prices, so moving a site from the pair to the provider +does not change WHEN the charge runs. DECISIONS/PRICE-01-own-price-server.md +says how equal, and where it is not. +""" + +import logging +from dataclasses import dataclass +from datetime import datetime + +from .price import (AUTH_REFUSED, PROVIDERS, PriceRead, brussels, + provider_id, scrub) + +_LOG = logging.getLogger("goodwe.price.chain") + +# The four places a plan can come from, best first. A higher number is further +# down the chain. +STEP_OWN, STEP_PUBLIC, STEP_STORED, STEP_NONE = 0, 1, 2, 3 + +# What a step is called in a status line. +SOURCE_WORDS = { + "price_service": "the price service", + "generic_dayahead": "energy-charts", + "price_history": "saved prices", +} + + +def uses_chain(opts: dict) -> bool: + """Does the selected provider ask for the fallback chain?""" + cls = PROVIDERS.get(provider_id(opts)) + return bool(cls is not None and (cls.FALLBACKS or cls.KEEPS_LAST_GOOD)) + + +@dataclass(frozen=True) +class Resolved: + """What the chain found. `published` is the series BEFORE the yesterday + fill and the window, which is what the day's average has to be read from: + the plan itself starts at the current hour.""" + + read: PriceRead + step: int + reasons: tuple = () + + @property + def published(self): + return self.read.match(series=lambda r: r, no_data=lambda _r: None) + + +class PriceWatch: + """Which step the prices come from, and one log line per change of it. + + The line is written when the step CHANGES and not before. Nothing repeats + while a source stays down, and nothing is written for a refresh that + changes nothing. The reasons (a status code, a timeout) go on a DEBUG line + beside it, with the token taken out. + """ + + def __init__(self): + self.step = None + self.auth_refused = False + + def note(self, step: int, reasons=(), opts=None) -> None: + old, self.step = self.step, step + self.auth_refused = any(AUTH_REFUSED in r for r in reasons) + if old == step: + return + if reasons: + _LOG.debug("Price sources said: %s.", + scrub("; ".join(reasons), opts)) + if step == STEP_OWN: + if old is None: + _LOG.info("Prices come from the price service.") + else: + _LOG.info("The price service answers again.") + elif step == STEP_PUBLIC: + if old is not None and old > step: + _LOG.info("Prices come from energy-charts again.") + elif self.auth_refused: + _LOG.warning("The price service refused the token, so prices " + "come from energy-charts.") + else: + _LOG.warning("The price service does not answer, so prices " + "come from energy-charts.") + elif step == STEP_STORED: + _LOG.warning("No price source answers, so the add-on uses the " + "prices it saved.") + else: + _LOG.warning("No price source answers, and no saved price covers " + "now.") + + +async def resolve(opts: dict, since: datetime, until: datetime, transport, + history, stamp: datetime, watch: PriceWatch, + load) -> Resolved: + """Walk the chain and return the first step that has a price for `stamp`. + + `load` is the caller's `load_prices`, passed in so that `static_tariff.py` + stays the one module that fetches prices. Never raises + for a source condition: a read that gets nowhere is a `no_data` whose + reason names every step, so the §11.4 row applies exactly as for a single + provider that failed. + """ + pid = provider_id(opts) + cls = PROVIDERS[pid] + reasons = [] + for step, sid in enumerate((pid,) + tuple(cls.FALLBACKS)): + # A fallback is asked for ITS OWN feed. The operator's feed override is + # for the first provider and would send energy-charts' request to a + # URL it was never meant for. + o = opts if step == 0 else dict(opts, price_provider=sid, + price_feed_url="") + read = await load(o, since, until, transport) + if read.match(series=lambda r: r.covers(stamp), no_data=lambda _r: False): + # One fallback is shipped. Any further one would also be "public". + where = STEP_OWN if step == 0 else STEP_PUBLIC + watch.note(where, reasons, opts) + return Resolved(read, where, tuple(reasons)) + reasons.append(read.match( + series=lambda r: f"{sid}: it has no price for now", + no_data=lambda r: r.reason)) + if cls.KEEPS_LAST_GOOD: + kept = history.stored(stamp) + if kept.match(series=lambda r: r.covers(stamp), + no_data=lambda _r: False): + watch.note(STEP_STORED, reasons, opts) + return Resolved(kept, STEP_STORED, tuple(reasons)) + reasons.append(kept.match( + series=lambda r: "price_history: it has no price for now", + no_data=lambda r: r.reason)) + watch.note(STEP_NONE, reasons, opts) + # The reason is short on purpose: it is printed on the status page and in + # a WARNING every refresh. What each step said is on the DEBUG line. + return Resolved(PriceRead.no_data( + "no price source has a price for now"), STEP_NONE, + tuple(scrub(r, opts) for r in reasons)) + + +# --------------------------------------------------------------------------- # +# The maintenance charge gate, read from the provider's prices +# --------------------------------------------------------------------------- # + +@dataclass(frozen=True) +class Gate: + """What the 30 s tick takes from the standing plan. + + `covered` is whether a published price covers now. When it is False, + `cheap` is False and the charge waits: no price is not a cheap price. + `since` is when the cover ended, for the status line, or None when nothing + was ever published. + """ + + cheap: bool + covered: bool + since: datetime | None + words: str + + +NO_PLAN = Gate(False, False, None, "No prices yet") + + +def _round5(value: float) -> float: + # The Home Assistant template rounded the average to 5 decimals, and a + # price to 5 as well. Both sides are compared as it did, so a quarter that + # sits within 0.00001 EUR/kWh of the average gets the same answer. + return round(float(value), 5) + + +def gate_for(published, now: datetime) -> Gate: + """The charge gate for `now`, from the series the chain published. + + THE RULE (owner's decision, PRICE-01): the current 15-minute price is at or + below the average of today's prices, today being the Brussels day. This is + what `now_p <= avg_p` was on the Home Assistant pair. It is NOT + `TariffPlan.cheap_now()`, which is strictly below the mean of the firm + hours of the whole plan, and it is not meant to be. + + `published` is a series or None. With no price for now there is no gate + answer: the charge waits. + """ + if published is None: + return NO_PLAN + if not published.covers(now): + return Gate(False, False, published.last_end_before(now), + "No price for now") + today = brussels(now).date() + todays = [p.eur_per_kwh for p in published.points + if not p.provisional and brussels(p.start).date() == today] + here = published.at(now) + if here.provisional or not todays: + return Gate(False, False, published.last_end_before(now), + "No price for now") + avg = sum(todays) / len(todays) + cheap = _round5(here.eur_per_kwh) <= _round5(avg) + source = SOURCE_WORDS.get(published.provider, published.provider) + return Gate(cheap, True, None, + f"Prices from {source}, {here.eur_per_kwh:.4f} EUR/kWh now, " + f"{'cheap' if cheap else 'not cheap'} against today's average " + f"of {avg:.4f}") diff --git a/goodwe_controller/app/price_history.py b/goodwe_controller/app/price_history.py index e6576db..804f48d 100644 --- a/goodwe_controller/app/price_history.py +++ b/goodwe_controller/app/price_history.py @@ -581,7 +581,7 @@ class PriceHistory: # -- the one public entry point ----------------------------------------- def carry(self, read: PriceRead, *, since: datetime, until: datetime, - stamp: datetime) -> PriceRead: + stamp: datetime, remember: bool = True) -> PriceRead: """Remember what this read published, then §8.3-fill the missing hours. THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is @@ -599,7 +599,11 @@ class PriceHistory: source can price is left out (§11.4 for that hour), and `no_data` comes back when that is every hour. """ - self._remember(read, stamp) + if remember: + # PRICE-01: False only for a read that CAME FROM this store + # (`stored()`), which is already filed; filing it again would + # rewrite the same file for nothing, and nothing else may pass it. + self._remember(read, stamp) return fill_from_yesterday(read, yesterday=self.yesterday(stamp), since=since, until=until, stamp=stamp) @@ -798,6 +802,37 @@ class PriceHistory: str(rec.get("provider") or "price_history"), stamp) + def stored(self, stamp: datetime) -> PriceRead: + """PRICE-01: every price this store holds, as one series, or no_data. + + This is the LAST step of the price server's fallback (price_chain.py): + the answer of the last fetch that worked, kept on disk, so a site can + plan for as long as the days it holds last. It is a read of what was + PUBLISHED and filed, in the order they happened, and it is never the + carried fill (rule 1 holds: `carry(..., remember=False)` is how this + read is filled, and it is not filed again). + + The prices are the all-in ones stored when they were fetched. They are + not worked out again, so a change of markup or taxes shows from the + next fetch that works, not in these. + """ + if stamp.tzinfo is None or stamp.utcoffset() is None: + raise ValueError(f"stored({stamp!r}): a naive datetime is not an " + f"instant") + points = [] + for day in sorted(self._days): + rec = self._days[day] + got, _bad = _parse_points(rec.get("points") if isinstance( + rec, dict) else None) + points.extend(got) + if not points: + why = self.last_error or "it holds no prices" + return PriceRead.no_data( + f"the price history at {self.path} gives nothing: {why}") + points.sort(key=lambda p: p.start.astimezone(timezone.utc)) + return PriceRead.series(points, PriceProduct.DAY_AHEAD, + "price_history", stamp) + # -- observability ------------------------------------------------------- @property def words(self) -> str: @@ -873,9 +908,14 @@ class NoHistory: last_error = "no history is kept by this caller" words = "not kept" - def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead: + def carry(self, read: PriceRead, *, since, until, stamp, + remember: bool = True) -> PriceRead: return read + def stored(self, stamp: datetime) -> PriceRead: + return PriceRead.no_data("this caller keeps no price history, so " + "there are no stored prices") + def yesterday(self, stamp: datetime) -> PriceRead: return PriceRead.no_data( "this caller keeps no price history, so there are no prices from " diff --git a/goodwe_controller/app/static_tariff.py b/goodwe_controller/app/static_tariff.py index aa838c2..3cb7a37 100644 --- a/goodwe_controller/app/static_tariff.py +++ b/goodwe_controller/app/static_tariff.py @@ -204,12 +204,13 @@ import math from dataclasses import dataclass from datetime import datetime, timedelta, timezone -from . import messages +from . import messages, price_chain from .price import ( MAX_PLAUSIBLE_EUR_KWH, Absence, PricePoint, PriceProduct, PriceRead, Uncertainty, _Outcome, brussels, load_prices, price_option, provider_configured, required_spread, uncertainty_of, widen_factor, ) +from .price_chain import uses_chain _LOG = logging.getLogger("goodwe.tariff.static") @@ -793,6 +794,13 @@ class TariffPlan: self.read = read self.profile = profile self.origin = origin + # PRICE-01. The series the price chain found, BEFORE the yesterday fill + # cut it to the hours ahead: the maintenance gate needs the whole of + # today to know its average. None for every plan that did not come + # through the chain, which is every plan of every other provider. + self.published = None + self.step = None + self.reasons = () # -- derived facts ------------------------------------------------------- @property @@ -1274,7 +1282,8 @@ def static_status(opts: dict, now: datetime): async def resolve_tariff(opts: dict, since: datetime, until: datetime, - transport=None, now=None, *, history) -> TariffPlan: + transport=None, now=None, *, history, + watch=None) -> TariffPlan: """The full §9.2 + §11.4 resolution. MAY PERFORM I/O - see `static_plan`. ⚠️ NOT ON THE 30 s TICK, and that is a deliberate boundary rather than an @@ -1330,13 +1339,30 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, if not provider_configured(opts): return static_plan(opts, since, until, provisional=False, now=stamp) + if uses_chain(opts): + # PRICE-01. The provider names fallbacks, so its prices come from the + # first step of the chain that has one for now. Nothing else does. + found = await price_chain.resolve( + opts, since, until, transport, history, stamp, + watch if watch is not None else price_chain.PriceWatch(), + load=load_prices) + plan = found.read.match( + series=lambda r: _provider_plan( + opts, r, since, until, stamp, history, + remember=found.step != price_chain.STEP_STORED), + no_data=lambda r: _degrade(opts, r, since, until, stamp)) + plan.published, plan.step = found.published, found.step + plan.reasons = found.reasons + return plan + read = await load_prices(opts, since, until, transport) return read.match( series=lambda r: _provider_plan(opts, r, since, until, stamp, history), no_data=lambda r: _degrade(opts, r, since, until, stamp)) -def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan: +def _provider_plan(opts, read, since, until, stamp, history, *, + remember=True) -> TariffPlan: """§8.3: record what the feed published, carry yesterday into the gaps. The fill can itself return `no_data` - a feed whose points all fall outside @@ -1344,7 +1370,12 @@ def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan: §11.4 row arrived at one step later, so it degrades exactly as a failed fetch does rather than being special-cased here. """ - filled = history.carry(read, since=since, until=until, stamp=stamp) + if remember: + filled = history.carry(read, since=since, until=until, stamp=stamp) + else: + # PRICE-01: a read that came out of the store is not filed again. + filled = history.carry(read, since=since, until=until, stamp=stamp, + remember=False) return filled.match( series=lambda r: TariffPlan(r, _quiet_profile(opts), "provider"), no_data=lambda r: _degrade(opts, r, since, until, stamp)) diff --git a/goodwe_controller/app/ui_text/en.json b/goodwe_controller/app/ui_text/en.json index b5f07ee..a9e496c 100644 --- a/goodwe_controller/app/ui_text/en.json +++ b/goodwe_controller/app/ui_text/en.json @@ -35,7 +35,7 @@ "meter_source": {"off": "One grid power entity", "ha_dsmr": "Import and export entities", "mqtt_p1": "Meter over MQTT", "ha_signed": "One net power entity", "homewizard_local": "HomeWizard meter on the network"}, "meter_phases": {"1": "1 phase", "3": "3 phases"}, "shadow_mode": {"auto": "Automatic", "on": "Always on", "off": "Off"}, - "price_provider": {"generic_dayahead": "Day-ahead market price", "custom_formula": "Your own price formula", "manual_csv": "A price file"}, + "price_provider": {"generic_dayahead": "Day-ahead market price", "custom_formula": "Your own price formula", "manual_csv": "A price file", "price_service": "Your own price service"}, "log_level": {"trace": "Everything, for support", "debug": "Detailed, to find a problem", "info": "Normal", "warning": "Warnings and errors", "error": "Errors only"}, "period_days": {"mon-sun": "Every day", "mon-fri": "Monday to Friday", "sat,sun": "Saturday and Sunday", "mon": "Monday", "tue": "Tuesday", "wed": "Wednesday", "thu": "Thursday", "fri": "Friday", "sat": "Saturday", "sun": "Sunday"} }, @@ -410,6 +410,8 @@ "give_would": "Tomorrow: would give power at {at}, {eur} per kWh." }, "cycle_waits": "{name} waits its turn for the monthly cycle.", + "no_prices_yet": "No prices yet, maintenance waits.", + "no_prices_since": "No prices since {at}, maintenance waits.", "step_in": "The batteries step in above {kw}.", "step_in_would": "The batteries would step in above {kw}.", "period": "This 15-minute period is heading for {kw}.", diff --git a/goodwe_controller/app/ui_text/nl.json b/goodwe_controller/app/ui_text/nl.json index dfddd81..a3be7a3 100644 --- a/goodwe_controller/app/ui_text/nl.json +++ b/goodwe_controller/app/ui_text/nl.json @@ -35,7 +35,7 @@ "meter_source": {"off": "Eén entiteit voor netvermogen", "ha_dsmr": "Entiteiten voor afname en injectie", "mqtt_p1": "Meter via MQTT", "ha_signed": "Eén entiteit voor nettovermogen", "homewizard_local": "HomeWizard-meter op het netwerk"}, "meter_phases": {"1": "1 fase", "3": "3 fasen"}, "shadow_mode": {"auto": "Automatisch", "on": "Altijd aan", "off": "Uit"}, - "price_provider": {"generic_dayahead": "Day-aheadprijs van de markt", "custom_formula": "Uw eigen prijsformule", "manual_csv": "Een prijsbestand"}, + "price_provider": {"generic_dayahead": "Day-aheadprijs van de markt", "custom_formula": "Uw eigen prijsformule", "manual_csv": "Een prijsbestand", "price_service": "Uw eigen prijsdienst"}, "log_level": {"trace": "Alles, voor support", "debug": "Uitgebreid, om een probleem te vinden", "info": "Normaal", "warning": "Waarschuwingen en fouten", "error": "Alleen fouten"}, "period_days": {"mon-sun": "Elke dag", "mon-fri": "Maandag tot vrijdag", "sat,sun": "Zaterdag en zondag", "mon": "maandag", "tue": "dinsdag", "wed": "woensdag", "thu": "donderdag", "fri": "vrijdag", "sat": "zaterdag", "sun": "zondag"} }, @@ -345,6 +345,8 @@ }, "tomorrow_trade": {"grid": "Morgen: laadt om {at}, {eur} per kWh.", "grid_would": "Morgen: zou laden om {at}, {eur} per kWh.", "give": "Morgen: levert stroom om {at}, {eur} per kWh.", "give_would": "Morgen: zou stroom leveren om {at}, {eur} per kWh."}, "cycle_waits": "{name} wacht zijn beurt af voor de maandelijkse cyclus.", + "no_prices_yet": "Nog geen prijzen, onderhoud wacht.", + "no_prices_since": "Geen prijzen sinds {at}, onderhoud wacht.", "step_in": "De batterijen springen bij boven {kw}.", "step_in_would": "De batterijen zouden bijspringen boven {kw}.", "period": "Dit kwartier gaat richting {kw}.", diff --git a/goodwe_controller/app/uilayout.py b/goodwe_controller/app/uilayout.py index 4317993..279563a 100644 --- a/goodwe_controller/app/uilayout.py +++ b/goodwe_controller/app/uilayout.py @@ -297,7 +297,9 @@ DERIVED = { "price_taxes_profile": "ui.region", # a preset writes it, "own" keeps it } # Option keys the page never shows and never writes: passed through as stored. -HIDDEN: dict = {} +# PRICE-01: the price service token is a secret. It is set in the Configuration +# tab (a password field there) and this page never draws it. +HIDDEN: dict = {"price_service_token": "a secret, set on the Configuration tab"} def fields(): diff --git a/goodwe_controller/app/uipage.py b/goodwe_controller/app/uipage.py index f6ecf1a..5c3a90c 100644 --- a/goodwe_controller/app/uipage.py +++ b/goodwe_controller/app/uipage.py @@ -18,8 +18,10 @@ from . import uimodel as U from .configui import help_of, label_of, problems_by_field, split_group from .validate import power_request_entities -# A choice the page cannot make, only keep. None since P5's period editor. -ONLY_IF_HELD = () +# A choice the page cannot make, only keep. `price_service` needs a token, and +# the token is a secret this page never draws (uilayout.HIDDEN), so it is set on +# the Configuration tab and the page only keeps it (PRICE-01). +ONLY_IF_HELD = ("price_service",) _TIERS = ("bad", "warn", "info") diff --git a/goodwe_controller/app/validate.py b/goodwe_controller/app/validate.py index d8a56bf..ab94cf3 100644 --- a/goodwe_controller/app/validate.py +++ b/goodwe_controller/app/validate.py @@ -1706,6 +1706,22 @@ def _price_problems(opts: dict, notes: list | None = None) -> list[str]: f"{S('price_csv_path')} is empty. Set the path to the CSV file, or " "choose 'off'.") + # PRICE-01. The price service answers only to its token. Without one every + # fetch is refused and the add-on lives on its fallbacks, which the + # operator would not see: a source they chose, overridden by a layer they + # cannot see. A token the server cannot read as a header is the same case. + if pid == "price_service": + token = str(opts.get("price_service_token", "") or "").strip() + if not token: + problems.append( + f"{S('price_provider')} is 'price_service', but " + f"{S('price_service_token')} is empty. Set the token, or " + "choose another source.") + elif any(c.isspace() or not c.isprintable() for c in token): + problems.append( + f"{S('price_service_token')} has a space or a line break in " + "it. Paste the token again, with nothing around it.") + return problems diff --git a/goodwe_controller/config.yaml b/goodwe_controller/config.yaml index 85919d6..a7953c7 100644 --- a/goodwe_controller/config.yaml +++ b/goodwe_controller/config.yaml @@ -556,6 +556,10 @@ options: # naming that consequence on every fetch that uses an override; clearing this # option is what restores the full horizon. price_feed_url: "" + # PRICE-01. The bearer token for the owner's price server + # (`price_provider: price_service`). A password field. Empty with any other + # source. Never printed: not in the log, the status page or a notification. + price_service_token: "" price_horizon_h: 48 price_timeout_s: 10 @@ -1085,7 +1089,7 @@ schema: # this line offers exactly PROVIDERS + RETIRED_PROVIDERS + 'off' - derived, not # typed - and register() refuses to re-register a retired id, so the two # cannot both claim one. - price_provider: list(off|generic_dayahead|custom_formula|engie_dynamic|totalenergies_dynamic|eneco_eddy|manual_csv) + price_provider: list(off|generic_dayahead|price_service|custom_formula|engie_dynamic|totalenergies_dynamic|eneco_eddy|manual_csv) price_markup_eur_kwh: float(0,1) price_taxes_eur_kwh: float(0,1) @@ -1119,6 +1123,7 @@ schema: price_formula: str? price_csv_path: str? price_feed_url: str? + price_service_token: password? price_horizon_h: int(1,72) price_timeout_s: int(2,60) price_now_entity: str? diff --git a/goodwe_controller/test_arbitrage.py b/goodwe_controller/test_arbitrage.py index 5faf5c6..724cc00 100644 --- a/goodwe_controller/test_arbitrage.py +++ b/goodwe_controller/test_arbitrage.py @@ -81,6 +81,7 @@ from datetime import datetime, timedelta, timezone from app import arbitrage as _arb from named_absence import first from app import main as _main +from app.price_chain import PriceWatch from app import peakguard as _pg from app.arbiter import ( P_ARBITRAGE, P_LOOP, P_MAINT_SHAPE, P_MAINTENANCE, P_PEAK, P_SAFETY, Claim, @@ -2311,6 +2312,7 @@ class _T12Controller: self.tariff_plan = None self.tariff_at = None self.price_history = NoHistory() + self.price_watch = PriceWatch() # PRICE-01: the refresh passes it on self.stop_event = asyncio.Event() self.stop_event.set() # so the inter-refresh wait returns self._ticks = 0 diff --git a/goodwe_controller/test_price.py b/goodwe_controller/test_price.py index 7a53af1..f8449be 100644 --- a/goodwe_controller/test_price.py +++ b/goodwe_controller/test_price.py @@ -534,11 +534,15 @@ FIXTURES = { [v / 1000 for v in EPEX_EUR_MWH]), "custom_formula": (columnar_feed(EPEX_EUR_MWH), FORMULA_EUR_KWH), "manual_csv": (None, [0.1834, -0.0210, 0.3115]), + # PRICE-01: the owner's price server answers the same columnar layout. + "price_service": (columnar_feed(EPEX_EUR_MWH), + [v / 1000 for v in EPEX_EUR_MWH]), } check(f"the plugins that ship after TARIFF-07 and TARIFF-15 are registered " f"{sorted(PROVIDERS)}", - set(PROVIDERS) == {"generic_dayahead", "custom_formula", "manual_csv"}) + set(PROVIDERS) == {"generic_dayahead", "custom_formula", "manual_csv", + "price_service"}) check(f"...and the three withdrawn ids resolve to NOTHING " f"{sorted(RETIRED_PROVIDERS)}", set(RETIRED_PROVIDERS) == {"engie_dynamic", "totalenergies_dynamic", @@ -571,7 +575,7 @@ def canned(body): """A transport that serves recorded bytes and records that it was used.""" calls = [] - async def t(url, timeout_s): + async def t(url, timeout_s, headers=None): calls.append((url, timeout_s)) return body t.calls = calls @@ -603,6 +607,9 @@ def load_any(pid): # is exactly what section 3 grades. return load(pid, {"price_formula": json.dumps(ENECO_CARD)}, transport=canned(body)) + if pid == "price_service": + return load(pid, {"price_service_token": "test-token-1234"}, + transport=canned(body)) return load(pid, transport=canned(body)) @@ -898,7 +905,16 @@ _networked = sorted(pid for pid, cls in PROVIDERS.items() # SAME CAPTURED BYTES (section 13c), because it reads the same endpoint with # the same parser. A networked plugin with no capture is what this line exists # to forbid. -_replayed = ["custom_formula", "generic_dayahead"] +# +# ⚠️ PRICE-01 PUT A THIRD ON IT, AND ITS CAPTURE IS A RESHAPE, SAID PLAINLY. The +# price server answers only to a token this session does not hold, so no real +# answer of its own exists to replay. Its answer is the energy-charts layout +# (design, API section), so test_price_service.py replays the REAL energy-charts +# two-day capture with the three extra keys the design lists (`covers`, +# `source`, `fetched_at`). The prices and instants are real; the wrapper is the +# design's. The first answer captured from the live server on the rig test +# replaces it - that is the open item, and it is in the PRICE-01 report. +_replayed = ["custom_formula", "generic_dayahead", "price_service"] check(f"the networked plugins are {_networked}, and EVERY one of them is " f"replayed from a captured real body ({_replayed}) - no shipped plugin " f"reaches a network this file has never seen answer (TARIFF-07)", @@ -1252,7 +1268,7 @@ check(f"a negative day-ahead price passes through, all-in {_neg}", # (TARIFF-15): it reads the SAME columnar document, which has no per-point # flag, and applies arithmetic to it - arithmetic cannot invent a provenance # the document never carried. -_SETTLED_FEED = {"generic_dayahead", "custom_formula"} +_SETTLED_FEED = {"generic_dayahead", "custom_formula", "price_service"} for pid in sorted(set(READS) - _SETTLED_FEED): read = READS[pid] flags = derived(lambda r=read: [p.provisional for p in r.points]) diff --git a/goodwe_controller/test_price_forecast.py b/goodwe_controller/test_price_forecast.py index 7bb640e..65d5459 100644 --- a/goodwe_controller/test_price_forecast.py +++ b/goodwe_controller/test_price_forecast.py @@ -1133,6 +1133,11 @@ _T17_KEYS = ["price_taxes_profile"] # `export_price_eur_kwh` is untouched, every site on the scalar is unaffected, # and it lives in this dict and in config.yaml and nowhere else. _T16_KEYS = ["price_export_formula"] +# PRICE-01 added the eighth: `price_service_token`, the bearer token of the +# owner's price server. An ADDITION, like the three above: nothing was renamed, +# every other provider ignores it, and it lives in this dict and in config.yaml +# (as a password field) and nowhere else. +_P01_KEYS = ["price_service_token"] check(f"the in-code mirror holds exactly TARIFF-02's, TARIFF-03's, OPT-04's, " f"TARIFF-15's, TARIFF-17's and TARIFF-16's keys - FCST-03 still added " f"none of its own {_defaults}", @@ -1142,7 +1147,7 @@ check(f"the in-code mirror holds exactly TARIFF-02's, TARIFF-03's, OPT-04's, " "price_horizon_h", "price_timeout_s", "static_profile"] + _OPT04_KEYS + _T15_KEYS + _T17_KEYS - + _T16_KEYS)) + + _T16_KEYS + _P01_KEYS)) # ...and the later keys are genuinely NEW, not a rename of something that was # already there, or the line above would pass while the mirror silently lost a # key. @@ -1151,7 +1156,7 @@ check("...and OPT-04's four, TARIFF-15's one, TARIFF-17's one and TARIFF-16's " "are all still present", isinstance(_defaults, list) and len(set(_defaults) - set(_OPT04_KEYS) - set(_T15_KEYS) - - set(_T17_KEYS) - set(_T16_KEYS)) == 9) + - set(_T17_KEYS) - set(_T16_KEYS) - set(_P01_KEYS)) == 9) # ⚠️ AND THE ONE TARIFF-17 IS THE HOUR-VARYING TWIN OF IS STILL THERE, BY NAME. # The line above counts nine, and a rename of `price_taxes_eur_kwh` to # `price_taxes_profile` would also count nine. The scalar path being unchanged diff --git a/goodwe_controller/test_price_service.py b/goodwe_controller/test_price_service.py new file mode 100644 index 0000000..27f0545 --- /dev/null +++ b/goodwe_controller/test_price_service.py @@ -0,0 +1,1290 @@ +"""PRICE-01 - the owner's price server as a price source, and its fallbacks. +`python3 test_price_service.py` + +WHAT THIS FILE DEFENDS, in the order the card names it: + + 1. `price_service` fetches from ONE fixed HTTPS host with a bearer token, and + the price it returns is wholesale energy, so markup and taxes are added + once, exactly as for `generic_dayahead`. + 2. The chain own server -> energy-charts -> stored prices: every way the + server can fail steps down, every step down and back up is logged once, + and a day with every source down still plans from what was kept. + 3. No price for now: one status line and one notification, once, gone when a + price returns. English and Dutch. + 4. The token is in no log, no status row, no notification, no error text. + 5. The maintenance charge gate keeps the Home Assistant rule (current price + at or below today's average), read from the provider. The same day is fed + both ways and every 15-minute period must give the same answer. + 6. Modes plan on these prices. + +THE SERVER IS A MOCK. This session holds no token, so it cannot ask the real +one for a price. `Server` below answers the layout the design's API section +gives (`unix_seconds`, `price`, `unit`, `zone`, `covers`, `source`, +`fetched_at`), over a transport with the same contract as `price.https_get`. +Section 12 drives the REAL `https_get` against a local aiohttp server, for the +parts a mock transport cannot show: the header on the wire, the status words, +the timeout, a refused connection. + +THE PRICES ARE NOT A FLAT LINE. They are deterministic and uneven, with a +negative midday, so "at or below the average" separates quarters on both sides +and a rule that always says yes or always says no cannot pass. + +⚠️ MUTATING A MODULE THIS FILE GRADES? Purge `__pycache__` first, or use +`tools/mutation_run.sh`. See MUTATION.md. +""" + +import asyncio +import io +import json +import logging +import os +import shutil +import sys +import tempfile +from datetime import date, datetime, timedelta, timezone + +from named_absence import nth +from app import main as _main +from app import messages, price as P, price_chain as C +from app.price import ( + PROVIDERS, PriceProduct, SourceUnavailable, brussels, brussels_offset, + load_prices, scrub, status_words) +from app.price_history import PriceHistory +from app.static_tariff import resolve_tariff +from app.validate import validate_startup_config + +fails = [] + + +def check(name, cond): + print(f" {'ok ' if cond else 'FAIL'} {name}") + if not cond: + fails.append(name) + + +def run(coro): + return asyncio.run(coro) + + +UTC = timezone.utc +HERE = os.path.dirname(os.path.abspath(__file__)) +TOKEN = "tk-7Hq2-secret-9xZ4-PRICE01" +HOST = P.PriceService.HOST +EC_HOST = "api.energy-charts.info" +MARKUP, TAXES = 0.031, 0.057 +_TMP = [] + + +def tmpdir(prefix="price01-"): + d = tempfile.mkdtemp(prefix=prefix) + _TMP.append(d) + return d + + +# --------------------------------------------------------------------------- # +# The mock server, and the day it serves +# --------------------------------------------------------------------------- # + +def local_midnight(d: date) -> datetime: + """Brussels midnight of `d` as a UTC instant. Midnight is never inside the + 02:00-03:00 change, so one of the two offsets is always right.""" + for off in (P.CET, P.CEST): + t = datetime(d.year, d.month, d.day, tzinfo=off) + if brussels_offset(t) == off: + return t.astimezone(UTC) + raise AssertionError(d) + + +def day_starts(d: date) -> list: + """The UTC instants of every 15-minute period of Brussels day `d`: 96, or + 92 and 100 on the two clock-change days.""" + a, b = local_midnight(d), local_midnight(d + timedelta(days=1)) + out, t = [], a + while t < b: + out.append(t) + t += timedelta(seconds=900) + return out + + +def ws_price(d: date, i: int) -> float: + """EUR/MWh for period `i` of day `d`. Uneven, two decimals, negative at + midday on some days. Not a function of anything the code under test sees.""" + base = 40 + 0.9 * ((d.toordinal() * 131 + i * i * 7) % 120) + if 44 <= i <= 58 and d.toordinal() % 2 == 0: + base -= 80 + return round(base, 2) + + +def day_rows(d: date) -> list: + return [(int(t.timestamp()), ws_price(d, i)) + for i, t in enumerate(day_starts(d))] + + +class Clock: + def __init__(self, now): + self.now = now + + def __call__(self): + return self.now + + +class Server: + """The price server, as the design's API section describes it. + + `published(now)` is the days it holds: today always, tomorrow from 13:00 + Brussels time if `tomorrow` is on. `mode` picks what it does to a request; + every mode that is not "up" is a way a real server or the road to it can + fail. + """ + + def __init__(self, clock, token=TOKEN): + self.clock, self.token = clock, token + self.mode = "up" + self.tomorrow = True + self.requests = [] + + def days(self): + today = brussels(self.clock()).date() + return [today] + ([today + timedelta(days=1)] if self.tomorrow else []) + + def doc(self, days=None): + days = self.days() if days is None else days + rows = [r for d in days for r in day_rows(d)] + return {"unix_seconds": [r[0] for r in rows], + "price": [r[1] for r in rows], + "unit": "EUR / MWh", "zone": "BE", + "covers": [d.isoformat() for d in days], + "source": {d.isoformat(): "entsoe" for d in days}, + "fetched_at": self.clock().strftime("%Y-%m-%dT%H:%M:%SZ")} + + def answer(self, headers): + self.requests.append(dict(headers or {})) + m = self.mode + if (headers or {}).get("Authorization") != "Bearer " + self.token \ + and m == "up": + raise SourceUnavailable(status_words(401)) + if m == "refused": + # A hostile library text: it carries the header it was sent. + raise ConnectionRefusedError( + f"Cannot connect to host {HOST}:443 ssl:default " + f"[Connect call failed] Authorization: Bearer {self.token}") + if m == "timeout": + raise asyncio.TimeoutError() + if m in ("401", "403", "429", "500", "503"): + raise SourceUnavailable(status_words(int(m))) + if m == "bad_json": + return b"Bad gateway" + d = self.doc() + if m == "no_arrays": + del d["price"] + elif m == "unequal": + d["price"] = d["price"][:-3] + elif m == "wrong_unit": + d["unit"] = "EUR / kWh" + elif m == "bad_covers": + d["covers"] = "2026-10-14" + elif m == "empty": + d["unix_seconds"], d["price"] = [], [] + elif m == "stale": + d = self.doc([brussels(self.clock()).date() - timedelta(days=3)]) + elif m == "uncovered": + d["covers"] = [] + elif m == "partial_tomorrow": + # Tomorrow's rows are there but the server does not call the day + # complete, so it must not be used. + d["covers"] = d["covers"][:1] + return json.dumps(d).encode() + + +class EnergyCharts: + """api.energy-charts.info: the columnar layout, for the dates asked for.""" + + def __init__(self, clock): + self.clock = clock + self.mode = "up" + self.tomorrow = True + self.requests = [] + + def answer(self, url): + self.requests.append(url) + if self.mode == "down": + raise SourceUnavailable("energy-charts is not answering") + if self.mode == "503": + raise ConnectionError("503 Service Unavailable") + q = dict(p.split("=") for p in url.split("?", 1)[1].split("&")) + a, b = date.fromisoformat(q["start"]), date.fromisoformat(q["end"]) + today = brussels(self.clock()).date() + out, d = [], a + while d <= b: + if d <= today or (d == today + timedelta(days=1) + and self.tomorrow): + out += day_rows(d) + d += timedelta(days=1) + return json.dumps({"unix_seconds": [r[0] for r in out], + "price": [r[1] for r in out], + "unit": "EUR / MWh", "license_info": "x", + "deprecated": False}).encode() + + +class Net: + """The transport. Same contract as `price.https_get`.""" + + def __init__(self, clock): + self.clock = clock + self.server = Server(clock) + self.ec = EnergyCharts(clock) + self.calls = [] + + async def __call__(self, url, timeout_s, headers=None): + self.calls.append((url, dict(headers or {}))) + host = url.split("/")[2] + if host == HOST: + return self.server.answer(headers) + if host == EC_HOST: + return self.ec.answer(url) + raise AssertionError(f"a request left for an undeclared host: {url}") + + def down(self): + self.server.mode, self.ec.mode = "refused", "down" + + def up(self): + self.server.mode, self.ec.mode = "up", "up" + + +# --------------------------------------------------------------------------- # +# A place to read logs, notifications and statuses from +# --------------------------------------------------------------------------- # + +class Tap(logging.Handler): + def __init__(self): + super().__init__(logging.DEBUG) + self.records = [] + + def emit(self, record): + self.records.append(record) + + def lines(self, minimum=logging.INFO): + return [r.getMessage() for r in self.records + if r.levelno >= minimum] + + def text(self): + return "\n".join(f"{r.levelname} {r.name} {r.getMessage()}" + for r in self.records) + + +TAP = Tap() +_root = logging.getLogger() +_root.addHandler(TAP) +_root.setLevel(logging.DEBUG) + + +def mark(): + return len(TAP.records) + + +def since_mark(n, minimum=logging.INFO): + return [r.getMessage() for r in TAP.records[n:] if r.levelno >= minimum] + + +def opts(**over): + o = {"price_provider": "price_service", "price_service_token": TOKEN, + "price_markup_eur_kwh": MARKUP, "price_taxes_eur_kwh": TAXES, + "price_timeout_s": 5} + o.update(over) + return o + + +NOW = datetime(2026, 10, 14, 15, 7, tzinfo=UTC) # 17:07 Brussels (CEST) + + +def make_ctx(now=NOW, **over): + clock = Clock(now) + net = Net(clock) + hist = PriceHistory(os.path.join(tmpdir(), "price_history.json")) + return type("Ctx", (), {"clock": clock, "net": net, "hist": hist, + "watch": C.PriceWatch(), "o": opts(**over)})() + + +def refresh(ctx, now=None): + """One background refresh, exactly as `run_tariff_refresh` calls it.""" + if now is not None: + ctx.clock.now = now + now = ctx.clock.now + return run(resolve_tariff(ctx.o, now, now + timedelta(hours=48), + transport=ctx.net, now=now, history=ctx.hist, + watch=ctx.watch)) + + +def covers_now(plan, now): + return plan.published is not None and plan.published.covers(now) + + +def all_in(ws, markup=MARKUP, taxes=TAXES): + return round(ws / 1000 + markup + taxes, 6) + + +# =========================================================================== # +print("0. the mock is the server the design describes, and the day is uneven") +_ctx = make_ctx() +_d = _ctx.net.server.doc() +check("the answer has the design's keys", + {"unix_seconds", "price", "unit", "zone", "covers", "source", + "fetched_at"} <= set(_d)) +check("...with one price per start, 900 s apart, today and tomorrow", + len(_d["unix_seconds"]) == len(_d["price"]) == 192 + and {b - a for a, b in zip(_d["unix_seconds"], + _d["unix_seconds"][1:])} == {900} + and _d["covers"] == ["2026-10-14", "2026-10-15"]) +check("the fixture is uneven and has a negative price, so 'at or below the " + "average' separates periods on both sides", + min(_d["price"]) < 0 < max(_d["price"]) + and 20 < sum(1 for p in _d["price"][:96] + if p <= sum(_d["price"][:96]) / 96) < 76) +check("clock-change days have 92 and 100 periods, not 96", + (len(day_starts(date(2026, 3, 29))), len(day_starts(date(2026, 10, 25)))) + == (92, 100)) + +# =========================================================================== # +print("\n1. the provider: one fixed host, a bearer token, wholesale energy") +_ctx = make_ctx() +_n = mark() +_plan = refresh(_ctx) +check("the first step answers: the plan came from the price service", + _plan.step == C.STEP_OWN and _plan.published.provider == "price_service" + and _plan.from_feed) +_url, _hdr = _ctx.net.calls[0] +check("the request went to the one fixed HTTPS URL, and only to it", + _url == "https://prices.kammenstraatha.duckdns.org/v1/dayahead/BE" + and [u for u, _h in _ctx.net.calls if HOST not in u] == []) +check("...with the token as a bearer header", _hdr == {"Authorization": + "Bearer " + TOKEN}) +check("...and the token is not in the URL", TOKEN not in _url) +check("the host is declared as the plugin's one egress host, so the framework " + "refuses any other", + P.PriceService.capabilities().egress_hosts == (HOST,)) +def _refused(fn): + try: + fn() + except SourceUnavailable: + return True + return False + + +check("...an http:// URL on the right host is refused", + _refused(lambda: P.check_egress("http://" + HOST + "/v1/dayahead/BE", + P.PriceService.capabilities()))) +check("...and so is another host over https", + _refused(lambda: P.check_egress("https://example.org/v1/dayahead/BE", + P.PriceService.capabilities()))) +_ctx2 = make_ctx(price_feed_url="http://evil.example/x") +refresh(_ctx2) +check("the operator's feed override is ignored by this provider: the request " + "still went to the fixed URL", + _ctx2.net.calls[0][0] == P.PriceService.URL) + +# markup and taxes are added once, as for generic_dayahead +_ps = _plan.published +_gd = run(load_prices(opts(price_provider="generic_dayahead"), NOW, + NOW + timedelta(hours=48), transport=_ctx.net)) +_ps_prices = [p.eur_per_kwh for p in _ps.points] +_want = [all_in(r[1]) for d in _ctx.net.server.days() for r in day_rows(d)] +check("the price is the wholesale figure plus markup and taxes, once: every " + "point equals ws/1000 + markup + taxes", + _ps_prices == _want) +check("...and a price_service series equals the generic_dayahead one for the " + "same days, point for point (start and price)", + [(p.start, p.eur_per_kwh) for p in _ps.points] + == [(p.start, p.eur_per_kwh) for p in _gd.points]) +check("the provider declares energy only, so the framework adds markup and " + "taxes, and the product is day-ahead", + P.PriceService.capabilities().adds + == P.GenericDayAhead.capabilities().adds + and P.PriceService.capabilities().product is PriceProduct.DAY_AHEAD) +check("the plugin cannot reach a framework key (register() would refuse it)", + P.code_leaks(P.PriceService) == []) + +_ctx3 = make_ctx(price_service_token="") +_p3 = refresh(_ctx3) +check("an empty token sends NO request to the server and falls to energy-charts", + not any(HOST in u for u, _h in _ctx3.net.calls) + and _p3.step == C.STEP_PUBLIC) +check("a token with a line break in it is never sent", + not any(HOST in u for u, _h in + (lambda c: (refresh(c), c.net.calls)[1])( + make_ctx(price_service_token="abc\r\nX-Evil: 1")))) + +# =========================================================================== # +print("\n2. every way the server can fail steps down to energy-charts") +_FAILURES = { + "refused": "does not answer", + "timeout": "does not answer", + "401": "refused the token", + "403": "refused the token", + "429": "does not answer", + "500": "does not answer", + "503": "does not answer", + "bad_json": "does not answer", + "no_arrays": "does not answer", + "unequal": "does not answer", + "wrong_unit": "does not answer", + "bad_covers": "does not answer", + "empty": "does not answer", + "stale": "does not answer", + "uncovered": "does not answer", +} +for _mode, _words in _FAILURES.items(): + _c = make_ctx() + _c.net.server.mode = _mode + _n = mark() + _p = refresh(_c) + _warn = since_mark(_n, logging.WARNING) + check(f"server {_mode}: prices come from energy-charts, firm, covering now", + _p.step == C.STEP_PUBLIC and _p.from_feed + and _p.published.provider == "generic_dayahead" + and covers_now(_p, _c.clock.now)) + check(f"...one WARNING says why, plainly ({_warn})", + len(_warn) == 1 and _words in _warn[0] + and "energy-charts" in _warn[0]) + check(f"...and the energy-charts request never carried the token", + all(TOKEN not in json.dumps(h) + for u, h in _c.net.calls if EC_HOST in u)) + _n2 = mark() + refresh(_c, _c.clock.now + timedelta(minutes=30)) + check(f"...and the next refresh of the same failure logs nothing again", + since_mark(_n2, logging.INFO) == []) + +_c = make_ctx(price_feed_url="https://api.energy-charts.info/price?bzn=BE") +_c.net.server.mode = "refused" +_p = refresh(_c) +_ec_urls = [u for u, _h in _c.net.calls if EC_HOST in u] +check("the operator's feed override is not used for the energy-charts step: it " + "is asked for its own feed with the date range, and answers", + _p.step == C.STEP_PUBLIC and len(_ec_urls) == 1 + and "start=" in _ec_urls[0] and "end=" in _ec_urls[0]) + +# partial day: tomorrow not published yet (before ~13:00 the server holds today) +_c = make_ctx(now=datetime(2026, 10, 14, 6, 0, tzinfo=UTC)) +_c.net.server.tomorrow = False +_n = mark() +_p = refresh(_c) +check("tomorrow missing: the server is still the source, no warning", + _p.step == C.STEP_OWN and since_mark(_n, logging.WARNING) == []) +_days = {brussels(x.start).date() for x in _p.published.points} +check("...the series holds today only, and the plan carries the rest from " + "yesterday's saved day as estimated, not firm", + _days == {date(2026, 10, 14)}) +_c = make_ctx() +_c.net.server.mode = "partial_tomorrow" +_p = refresh(_c) +check("a day the server does not list as complete is not used, even if its " + "rows are there", + _p.step == C.STEP_OWN + and {brussels(x.start).date() for x in _p.published.points} + == {date(2026, 10, 14)}) + +# =========================================================================== # +print("\n3. both sources down: the prices the history kept") +_c = make_ctx() +refresh(_c) # a fetch that worked +_snap = open(_c.hist.path, "rb").read() +_st0 = os.stat(_c.hist.path) +_c.net.down() +_n = mark() +_p = refresh(_c, NOW + timedelta(minutes=30)) +_warn = since_mark(_n, logging.WARNING) +check("the plan comes from the saved prices, is firm, and covers now", + _p.step == C.STEP_STORED and _p.from_feed + and _p.published.provider == "price_history" + and covers_now(_p, _c.clock.now)) +check(f"...and one WARNING says so plainly ({_warn})", + _warn == ["No price source answers, so the add-on uses the prices it " + "saved."]) +check("...with the reasons of each step on a DEBUG line, token-free", + any("Price sources said" in r.getMessage() and r.levelno == logging.DEBUG + for r in TAP.records[_n:]) + and all(TOKEN not in r.getMessage() for r in TAP.records[_n:])) +_st1 = os.stat(_c.hist.path) +check("...and the saved file was not written again from a read of itself (the " + "file is replaced on every save, so the same inode and time mean no save)", + open(_c.hist.path, "rb").read() == _snap + and (_st1.st_ino, _st1.st_mtime_ns) == (_st0.st_ino, _st0.st_mtime_ns)) +_stored_prices = [p.eur_per_kwh for p in _p.published.points] +check("the saved prices are the ones fetched, not carried guesses", + all(not p.provisional for p in _p.published.points) + and _stored_prices == _want) +_n = mark() +refresh(_c, NOW + timedelta(minutes=60)) +check("a second refresh in the same state logs nothing", + since_mark(_n, logging.INFO) == []) + +# nothing stored, everything down: no price +_c = make_ctx() +_c.net.down() +_n = mark() +_p = refresh(_c) +check("with nothing saved and every source down there is no plan from a feed", + _p.step == C.STEP_NONE and not _p.from_feed and _p.published is None) +check("...the chain says so in one WARNING", + [r.getMessage() for r in TAP.records[_n:] + if r.name == "goodwe.price.chain" and r.levelno >= logging.WARNING] + == ["No price source answers, and no saved price covers now."]) +check("...and what each step said is kept for DEBUG, without the token", + len(_p.reasons) == 3 and all(TOKEN not in r for r in _p.reasons) + and "price_service" in _p.reasons[0] + and "generic_dayahead" in _p.reasons[1] + and "price_history" in _p.reasons[2]) + +# =========================================================================== # +print("\n4. back up the chain, one line per step") +_c = make_ctx() +_c.net.down() +refresh(_c) +_n = mark() +_c.net.ec.mode = "up" +refresh(_c, NOW + timedelta(minutes=30)) +check("energy-charts back: INFO says so once", + since_mark(_n, logging.INFO) + == ["Prices come from energy-charts again."]) +_n = mark() +_c.net.server.mode = "up" +refresh(_c, NOW + timedelta(minutes=60)) +check("the server back: INFO says so once", + since_mark(_n, logging.INFO) == ["The price service answers again."]) +_n = mark() +refresh(_c, NOW + timedelta(minutes=90)) +check("and nothing more while it stays up", since_mark(_n, logging.INFO) == []) +_c = make_ctx() +refresh(_c) +_c.net.down() +refresh(_c, NOW + timedelta(minutes=30)) +_n = mark() +_c.net.server.mode = "up" +refresh(_c, NOW + timedelta(minutes=60)) +check("stored -> server directly is one 'answers again' line", + since_mark(_n, logging.INFO) == ["The price service answers again."]) +_c = make_ctx() +_n = mark() +_c.net.server.mode = "401" +refresh(_c) +check("a start-up with the server refusing is one WARNING, not an INFO then a " + "WARNING", + since_mark(_n, logging.INFO) + == ["The price service refused the token, so prices come from " + "energy-charts."]) + +# =========================================================================== # +print("\n5. a full day with every source down") +# A good fetch at 13:30 Brussels with tomorrow in it; then nothing answers for +# 24 hours. Every 30-minute refresh must still produce a firm plan that covers +# now, until the saved day itself runs out at midnight after tomorrow. +_start = datetime(2026, 10, 14, 11, 30, tzinfo=UTC) # 13:30 CEST +_c = make_ctx(now=_start) +refresh(_c) +_c.net.down() +_bad = [] +_t = _start +_gate_nones = 0 +while _t < _start + timedelta(hours=24): + _t += timedelta(minutes=30) + _p = refresh(_c, _t) + if not (_p.from_feed and _p.step == C.STEP_STORED + and covers_now(_p, _t)): + _bad.append(_t.isoformat()) + if not _p.firm_points(): + _bad.append("no firm point at " + _t.isoformat()) +check(f"48 refreshes over 24 h, all from the saved day, all firm, all covering " + f"now ({_bad[:3]})", not _bad) +_g = C.gate_for(_p.published, _t) +check("...and the charge gate still gives an answer 24 h later", + _g.covered and _g.words.startswith("Prices from saved prices")) +_end = local = local_midnight(date(2026, 10, 16)) +_p = refresh(_c, _end - timedelta(minutes=15)) +check("the last saved period still covers", covers_now(_p, _end + - timedelta(minutes=15))) +_p = refresh(_c, _end) +check("and at the first moment after the saved days there is no price, and " + "the series says since when", + _p.step == C.STEP_NONE and C.gate_for( + _c.hist.stored(_end).match(series=lambda r: r, no_data=None), _end + ).since == _end) + +# =========================================================================== # +print("\n6. no price for now: one status line, one notification, cleared") +OPTS_C = {"setpoint_entity": "number.sp", "max_w": 5000, + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "meter_entity": "sensor.grid", + "maintenance_charge_w": 2000, "maintenance_discharge_w": 2000} + + +class FakeHass: + def __init__(self): + self.calls = [] + self.values = {"sensor.grid": 0.0, "sensor.soc": 55.0, + "sensor.batt": 0.0} + + async def limits(self, entity): + return (-5000.0, 5000.0, 1.0) + + async def number(self, entity, invert=False): + return self.values.get(entity) + + async def call(self, domain, service, data): + self.calls.append((domain, service, data)) + return True + + +class FakeStore: + data = {} + + def set(self, *a): + pass + + def get_time(self, key): + return None + + +class FakePub: + enabled = False + + def publish(self, *a, **k): + pass + + def close(self): + pass + + +class FakeDT(datetime): + """`datetime` with a clock the test moves.""" + _clock = None + + @classmethod + def now(cls, tz=None): + t = cls._clock.now + return t.astimezone(tz) if tz else t.replace(tzinfo=None) + + +def controller(clock, o, tmp=None): + tmp = tmp or tmpdir() + stop = asyncio.Event() + ctl = _main.Controller(dict(OPTS_C, **o), FakeHass(), FakeStore(), + FakePub(), stop=stop, + peak_db_path=os.path.join(tmp, "tariff.db"), + price_history_path=os.path.join(tmp, "ph.json")) + return ctl + + +_REAL_DT, _REAL_GET = _main.datetime, P.https_get +_real_resolve = _main.resolve_tariff + + +class Wired: + """A controller whose clock and network are the test's.""" + + def __init__(self, now, **over): + self.clock = Clock(now) + FakeDT._clock = self.clock + self.net = Net(self.clock) + self.ctl = controller(self.clock, opts(**over)) + self.refreshes = 0 + + def __enter__(self): + _main.datetime = FakeDT + P.https_get = self.net + W = self + + async def counting(*a, **k): + r = await _real_resolve(*a, **k) + W.refreshes += 1 + return r + _main.resolve_tariff = counting + return self + + def __exit__(self, *exc): + _main.datetime, P.https_get = _REAL_DT, _REAL_GET + _main.resolve_tariff = _real_resolve + + async def refresh(self, now=None): + if now is not None: + self.clock.now = now + c = self.ctl + n = self.refreshes + c.stopping = False + c.stop_event.clear() + task = asyncio.create_task(c.run_tariff_refresh()) + for _ in range(500): + if self.refreshes > n: + break + await asyncio.sleep(0.01) + await asyncio.sleep(0.02) + c.stopping = True + c.stop_event.set() + await asyncio.wait_for(task, timeout=5) + + async def tick(self, now=None): + """One 30 s tick: read inputs, then the price-gap check.""" + if now is not None: + self.clock.now = now + await self.ctl.read_inputs() + await self.ctl.check_price_gap() + + def notes(self): + return [(s, d) for dom, s, d in self.ctl.hass.calls + if dom == "persistent_notification"] + + +async def _scenario_gap(): + out = {} + with Wired(NOW) as W: + c = W.ctl + out["rows_before"] = dict(c.status_payload()["rows"]) + await W.tick() + out["before_first_refresh"] = (c.price_gap, list(W.notes())) + W.net.down() + await W.refresh() + for k in range(6): # six ticks inside the gap + await W.tick(NOW + timedelta(seconds=30 * k)) + out["gap"] = (c.price_gap, c.price_gap_since, list(W.notes())) + out["rows_gap"] = dict(c.status_payload()["rows"]) + out["cheap_gap"] = (c.cheap, c._price_gated, + c._charge_gate_open(c.inverters[0])) + c.inverters[0].batt = -2000.0 # a pack really taking charge + out["open_while_charging"] = c._charge_gate_open(c.inverters[0]) + c.inverters[0].batt = None + # the prices come back + W.net.up() + await W.refresh(NOW + timedelta(minutes=30)) + for k in range(4): + await W.tick(NOW + timedelta(minutes=30, seconds=30 * k)) + out["back"] = (c.price_gap, list(W.notes())) + out["rows_back"] = dict(c.status_payload()["rows"]) + out["words_back"] = c.tariff_words + return out + + +_S = run(_scenario_gap()) +check("before the first refresh has finished there is no gap and no " + "notification, so a start-up is not an alarm", + _S["before_first_refresh"] == (False, []) and + "Prices" not in _S["rows_before"]) +_gap, _since, _notes = _S["gap"] +check("every source down with nothing saved: a gap, since nothing ever " + "published", _gap is True and _since is None) +check("...exactly ONE notification over six ticks, with the id the docs name", + len(_notes) == 1 and _notes[0][0] == "create" + and _notes[0][1]["notification_id"] == "goodwe_no_prices") +check("...in English and Dutch, plain and short", + "No prices have come in yet" in _notes[0][1]["message"] + and "Geen prijzen. Er zijn nog geen prijzen binnen" + in _notes[0][1]["message"] + and _notes[0][1]["title"] == "No prices") +check("...the status row says it once, in the card's words", + _S["rows_gap"].get("Prices") == "No prices yet, maintenance waits.") +check("...Cheap window is no, and the charge gate is shut", + _S["cheap_gap"] == (False, True, False)) +check("...but a pack that is really taking charge is not stopped by it", + _S["open_while_charging"] is True) +_gap2, _notes2 = _S["back"] +check("a price returns: the gap ends, the same notification is taken down " + "once, and no second one was raised", + _gap2 is False and len(_notes2) == 2 and _notes2[1][0] == "dismiss" + and _notes2[1][1] == {"notification_id": "goodwe_no_prices"}) +check("...and the row goes with it", "Prices" not in _S["rows_back"]) +check("...and the Tariff row names the source and the rule", + _S["words_back"].startswith("Prices from the price service,")) + +# the refresh task stops: after 90 minutes the plan is retired and the gate +# shuts, with the moment it was retired as 'since' +async def _scenario_dead_task(): + start = datetime(2026, 10, 14, 11, 30, tzinfo=UTC) + with Wired(start) as W: + c = W.ctl + await W.refresh() + await W.tick(start + timedelta(minutes=80)) + before = (c.price_gap, c.cheap is not None, len(W.notes())) + await W.tick(start + timedelta(minutes=95)) + await W.tick(start + timedelta(minutes=96)) + mid = (c.price_gap, c.price_gap_since, list(W.notes()), + c._charge_gate_open(c.inverters[0])) + await W.refresh(start + timedelta(minutes=100)) + await W.tick(start + timedelta(minutes=101)) + return before, mid, (c.price_gap, list(W.notes())) +_b, _m, _a = run(_scenario_dead_task()) +check("a plan 80 minutes old is still current: no gap, no notification", + _b == (False, True, 0)) +check("at 95 minutes with no refresh the plan is retired: a gap, since the " + "moment it was retired (90 minutes after the fetch), one notification, " + "gate shut", + _m[0] is True and _m[1] == datetime(2026, 10, 14, 11, 30, tzinfo=UTC) + + timedelta(seconds=_main.TARIFF_MAX_AGE_S) + and len(_m[2]) == 1 and _m[3] is False) +check("...and the next refresh that works takes it down again", + _a[0] is False and [n[0] for n in _a[1]] == ["create", "dismiss"]) + +# a gap that begins mid-day, with a price history that ends: 'since' is shown +async def _scenario_since(): + start = datetime(2026, 10, 14, 11, 30, tzinfo=UTC) + with Wired(start) as W: + c = W.ctl + await W.refresh() # a good fetch + W.net.down() + end = local_midnight(date(2026, 10, 16)) # the saved days run out + for k in range(3): + await W.refresh(end - timedelta(minutes=30 * (2 - k))) + await W.tick(end + timedelta(minutes=5)) + gap = (c.price_gap, c.price_gap_since) + row = dict(c.status_payload()["rows"]) + notes = list(W.notes()) + # the Home screen + from app import homeview + plan = homeview.build_plan(c, W.clock.now, False, True) + return gap, row, notes, plan["lines"], end +_gap, _row, _notes, _lines, _end = run(_scenario_since()) +check("when the saved days run out, the gap begins at that moment", + _gap == (True, _end)) +check("...the row reads 'No prices since 00:00, maintenance waits.' (Brussels " + "time, the card's words)", + _row.get("Prices") == "No prices since 00:00, maintenance waits.") +check("...the one notification carries the same moment, in English and Dutch", + len(_notes) == 1 + and "No prices since 00:00, so the monthly battery cycle waits" + in _notes[0][1]["message"] + and "Geen prijzen sinds 00:00, dus de maandelijkse batterijcyclus wacht" + in _notes[0][1]["message"]) +check("...and the Home screen says it once (English)", + _lines.count("No prices since 00:00, maintenance waits.") == 1) +from app import uitext as _uitext # noqa: E402 +_nl = _uitext.words("nl")["home_words"] +check("...with the Dutch words 'Geen prijzen sinds {at}, onderhoud wacht.'", + _nl["no_prices_since"] == "Geen prijzen sinds {at}, onderhoud wacht." + and _nl["no_prices_yet"] == "Nog geen prijzen, onderhoud wacht.") +check("an earlier day is spelled with its date, not only a time", + messages.since_text(brussels(datetime(2026, 10, 13, 19, 0, tzinfo=UTC)), + brussels(datetime(2026, 10, 14, 9, 0, tzinfo=UTC))) + == "13 Oct at 21:00") + +# =========================================================================== # +print("\n7. the token is in no log, status row, notification or error text") +_everything = [TAP.text(), + json.dumps(_S, default=str), json.dumps( + [_row, _notes, _lines], default=str)] +check("across every scenario above the token never appeared in any log record " + "at any level", + all(TOKEN not in r.getMessage() and TOKEN not in str(r.args) + and TOKEN not in (r.exc_text or "") for r in TAP.records)) +check("...nor in a status row, a notification or a home line", + all(TOKEN not in blob for blob in _everything[1:])) +check("...nor in the reason a failed read carries, even when the library's own " + "error text holds the header it was sent", + TOKEN not in (lambda c: (c.net.server.__setattr__("mode", "refused"), + c.net.ec.__setattr__("mode", "down"), + refresh(c).read.reason)[2])(make_ctx())) +_o = opts() +check("scrub() takes the value out of any text, and leaves other text alone", + scrub(f"x {TOKEN} y", _o) == "x *** y" + and scrub("no secret here", _o) == "no secret here" + and scrub(f"x {TOKEN}", {"price_service_token": ""}) == f"x {TOKEN}") + + +async def _hostile(url, timeout_s, headers=None): + # A transport that fails with the full request in its text, by any means. + raise RuntimeError(f"boom {headers} {url} {TOKEN}") + + +_r = run(load_prices(opts(), NOW, NOW + timedelta(hours=48), + transport=_hostile)) +check("an exception from the transport that carries the token reaches no " + "reason text", _r.kind == "no_data" and TOKEN not in _r.reason) + + +def _fetch_error(transport): + """What the PLUGIN itself raises, before the framework's scrub sees it.""" + try: + run(P.PriceService().fetch(P.PluginOptions(opts()), NOW, + NOW + timedelta(hours=48), transport)) + except SourceUnavailable as e: + return str(e) + return "" + + +_own = _fetch_error(_hostile) +check("the plugin's own error text has no token in it even before the " + "framework scrubs (defence in depth: two layers, each graded alone)", + _own.startswith("the price service did not answer (RuntimeError)") + and TOKEN not in _own) + + +async def _hostile2(url, timeout_s, headers=None): + raise SourceUnavailable(f"bad: {headers}") + + +check("a SourceUnavailable built from the headers is scrubbed too", + TOKEN not in run(load_prices(opts(), NOW, NOW + timedelta(hours=48), + transport=_hostile2)).reason) +from app import uilayout as _L, uipage as _UP # noqa: E402 +check("the settings page never draws the token (it is a hidden, secret key), " + "and it can keep the provider but not choose it", + "price_service_token" in _L.HIDDEN + and _L.field("price_service_token") is None + and "price_service" in _UP.ONLY_IF_HELD) + +# =========================================================================== # +print("\n8. the maintenance charge gate: the Home Assistant rule, from the " + "provider's prices") + + +def ha_state_now(rows, t): + """sensor.price_service_now, as price_service.yaml computes it.""" + u = [r[0] for r in rows] + p = [r[1] for r in rows] + step = (u[1] - u[0]) if len(u) > 1 else 900 + v = None + for i in range(len(u)): + if u[i] <= t < u[i] + step: + v = p[i] + if not (len(u) > 0 and u[0] <= t < u[-1] + 900): + return None + return None if v is None else round(v / 1000, 5) + + +def ha_state_avg(rows, day_start, day_end): + """sensor.price_service_avg, as price_service.yaml computes it.""" + s = n = 0 + for ts, pr in rows: + if day_start <= ts < day_end: + s += pr + n += 1 + return round(s / n / 1000, 5) if n else None + + +def ha_cheap(rows, t, today): + """The add-on's old rule: `now_p <= avg_p`, None means not cheap.""" + now_p = ha_state_now(rows, t) + a = int(local_midnight(today).timestamp()) + b = int(local_midnight(today + timedelta(days=1)).timestamp()) + avg_p = ha_state_avg(rows, a, b) + return now_p is not None and avg_p is not None and now_p <= avg_p + + +def published_series(day_list, markup=MARKUP, taxes=TAXES): + """The all-in series the provider would publish for these days, built by + the real framework from the mock's rows.""" + c = make_ctx(now=local_midnight(day_list[0]) + timedelta(hours=10), + price_markup_eur_kwh=markup, price_taxes_eur_kwh=taxes) + c.net.server.days = lambda: day_list + p = refresh(c) + assert p.step == C.STEP_OWN, p.step + return p.published, [r for d in day_list for r in day_rows(d)] + + +_days = [date(2026, 10, 14), date(2026, 10, 15), date(2026, 3, 29), + date(2026, 10, 25), date(2026, 7, 1), date(2026, 1, 2)] +_diffs, _checked, _cheap_yes = [], 0, 0 +for _d in _days: + _ser, _rows = published_series([_d, _d + timedelta(days=1)]) + for _t in day_starts(_d): + _want = ha_cheap(_rows, int(_t.timestamp()), _d) + for dt in (0, 450, 899): # first, middle and last second + _tt = _t + timedelta(seconds=dt) + _got = C.gate_for(_ser, _tt).cheap + _checked += 1 + if _got != _want: + _diffs.append((_d.isoformat(), _tt.isoformat(), _want, _got)) + _cheap_yes += _want +check(f"{_checked} instants over {len(_days)} days (96-period days, the two " + f"clock-change days, winter, summer): the provider's gate equals " + f"now_p <= avg_p on the same prices, every time ({_diffs[:2]})", + not _diffs and _checked > 1000) +check("...and the days really separate: a fifth to four fifths of periods are " + "cheap, so neither 'always' nor 'never' passes", + 0.2 < _cheap_yes / (_checked / 3) < 0.8) +_diffs = [] +for _mk, _tx in ((0.0, 0.0), (0.03, 0.11736), (0.04321, 0.09876), + (0.0, 0.2)): + _ser, _rows = published_series([date(2026, 10, 14), date(2026, 10, 15)], + _mk, _tx) + for _t in day_starts(date(2026, 10, 14)): + if C.gate_for(_ser, _t).cheap != ha_cheap(_rows, int(_t.timestamp()), + date(2026, 10, 14)): + _diffs.append((_mk, _tx, _t.isoformat())) +check(f"...for markup and taxes of up to five decimals, with the wholesale " + f"sensor unchanged: the same answer ({_diffs[:2]})", not _diffs) + +# the same day through the REAL controller, both ways +async def _both_ways(day): + rows = [r for d in (day, day + timedelta(days=1)) for r in day_rows(d)] + out = [] + t0 = local_midnight(day) + with Wired(t0 + timedelta(hours=10), price_markup_eur_kwh=MARKUP, + price_taxes_eur_kwh=TAXES) as W: + W.net.server.days = lambda: [day, day + timedelta(days=1)] + W.net.ec.tomorrow = True + W.clock.now = t0 + timedelta(hours=10) + await W.refresh() + ps = W.ctl + ha_hass = FakeHass() + FakeDT._clock = W.clock + ha = controller(W.clock, {"price_provider": "off", + "price_now_entity": "sensor.now", + "price_avg_entity": "sensor.avg"}) + ha.hass = ha_hass + a, b = (int(t0.timestamp()), + int(local_midnight(day + timedelta(days=1)).timestamp())) + last = None + for t in day_starts(day): + W.clock.now = t + timedelta(seconds=100) + # The background refresh runs every 30 minutes; a plan older than + # 90 minutes is retired, which is what keeps this gate shut when + # the refresh task has died. + if last is None or (t - last) >= timedelta(minutes=30): + await W.refresh(W.clock.now) + last = t + ha_hass.values["sensor.now"] = ha_state_now( + rows, int(W.clock.now.timestamp())) + ha_hass.values["sensor.avg"] = ha_state_avg(rows, a, b) + await ps.read_inputs() + await ha.read_inputs() + out.append((ha.cheap, ps.cheap, + ha._charge_gate_open(ha.inverters[0]), + ps._charge_gate_open(ps.inverters[0]), + ha._price_gated, ps._price_gated)) + return out + + +_rows_both = run(_both_ways(date(2026, 10, 14))) +check(f"the same day fed through the Home Assistant pair and through " + f"price_service: {len(_rows_both)} periods, the cheap flag and the " + f"charge gate agree in every one", + len(_rows_both) == 96 + and all(a == b and c == d and e == f + for a, b, c, d, e, f in _rows_both)) +check("...with both kinds of period present (cheap and not cheap)", + {r[0] for r in _rows_both} == {True, False}) +check("...and an open gate whenever it is cheap, a shut one when it is not " + "(no pack is charging)", + all(r[3] == r[1] for r in _rows_both)) + +# what the old rule did when the price was unavailable, and what this does +_ser, _rows = published_series([date(2026, 10, 14)]) +_late = local_midnight(date(2026, 10, 15)) + timedelta(minutes=1) +check("with no price for now both rules say not cheap (the HA sensor was " + "'unavailable'), and the gate stays shut", + C.gate_for(_ser, _late).cheap is False + and ha_cheap(_rows, int(_late.timestamp()), date(2026, 10, 15)) is False) +check("it is NOT TariffPlan.cheap_now(): on a price exactly at the average " + "this gate says cheap where cheap_now (strictly below) says no", + C.gate_for( + P.PriceRead.series( + [P.PricePoint(local_midnight(date(2026, 10, 14)) + + timedelta(minutes=15 * i), v, False) + for i, v in enumerate((0.10, 0.20, 0.30))], + PriceProduct.DAY_AHEAD, "price_service", NOW), + local_midnight(date(2026, 10, 14)) + timedelta(minutes=16)).cheap + is True) + +# The Home Assistant average was rounded to 5 decimals before it was compared, +# so a quarter that sits ABOVE the exact average but AT the rounded one was +# cheap. Real days almost never land on that, so it is built here: three +# periods at 0.20000 and five at 0.20001 average 0.20000625, which rounds to +# 0.20001, and a period at 0.20001 is therefore cheap by the old rule. +_t0 = local_midnight(date(2026, 10, 14)) +_vals = [0.20000] * 3 + [0.20001] * 5 +_toy = P.PriceRead.series( + [P.PricePoint(_t0 + timedelta(minutes=15 * i), v, False) + for i, v in enumerate(_vals)], PriceProduct.DAY_AHEAD, "price_service", + NOW) +_toy_rows = [(int((_t0 + timedelta(minutes=15 * i)).timestamp()), + round(v * 1000, 2)) for i, v in enumerate(_vals)] +_at = lambda i: _t0 + timedelta(minutes=15 * i + 1) # noqa: E731 +check("a period above the exact average but at the 5-decimal rounded one is " + "cheap, as the Home Assistant pair said: the gate and the pair agree on " + "all eight periods of a day built to sit on that edge, and the edge " + "period is cheap", + [C.gate_for(_toy, _at(i)).cheap for i in range(8)] + == [ha_cheap(_toy_rows, int(_at(i).timestamp()), date(2026, 10, 14)) + for i in range(8)] + and C.gate_for(_toy, _at(7)).cheap is True) + +# =========================================================================== # +print("\n9. battery modes plan on the provider's prices") +async def _modes(): + day = date(2026, 10, 14) + t = local_midnight(day) + timedelta(hours=8) + with Wired(t, battery_modes_enabled=True) as W: + await W.refresh() + c = W.ctl + plan = c.fresh_tariff_plan(t) + return (plan is not None and plan.has_prices, plan.origin, + plan.n_firm > 150, + [p.provider for p in [plan.read]][0], c.modes_on) + + +_m = run(_modes()) +check("the standing plan is a feed plan of firm price_service prices covering " + "the two days, and battery modes are on", + _m[0] and _m[1] == "provider" and _m[2] and _m[3] == "price_service" + and _m[4] is True) + +async def _modes_plan(): + day = date(2026, 10, 14) + t = local_midnight(day) + timedelta(hours=8) + with Wired(t, battery_modes_enabled=True, arbitrage_enabled=True) as W: + await W.refresh() + c = W.ctl + plan = c.fresh_tariff_plan(t) + verdict = plan.arbitrage( + peak_guard_active=False, min_spread_eur_kwh=0.05, + uncertainty=frozenset()) + return verdict.match(allowed=lambda v: ("allowed", v.spread_eur_kwh), + refused=lambda v: ("refused", v.reason)) + + +_v = run(_modes_plan()) +check(f"...and the arbitrage verdict on them is a real one over firm prices " + f"{_v}", _v[0] == "allowed" and _v[1] > 0.1) + +# =========================================================================== # +print("\n10. the options: validation, defaults, the old pair") +def problems(**over): + o = {"price_provider": "price_service", "price_service_token": TOKEN} + o.update(over) + base = dict(OPTS_C) + base.update(o) + return validate_startup_config(base, None, None) + + +check("price_service with a token starts", problems() == []) +_e = problems(price_service_token="") +check("...without a token it is refused, naming both settings by label and key " + f"({_e})", len(_e) == 1 and "“Dynamic price source” (price_provider)" + in _e[0] and "“Price service token” (price_service_token)" in _e[0] + and TOKEN not in _e[0]) +check("...a token with a space in it is refused, and the token is not echoed", + len(problems(price_service_token="ab cd")) == 1 + and "ab cd" not in problems(price_service_token="ab cd")[0]) +_e = problems(price_now_entity="sensor.a", price_avg_entity="sensor.b") +check(f"the existing conflict refusal stands: the Home Assistant pair beside " + f"this provider is refused ({len(_e)} problem)", + any("would win over it" in x for x in _e)) +check("a static_profile beside it is accepted (it is the fallback for " + "trading)", problems(static_profile=json.dumps({"periods": [ + {"name": "all", "days": "all", "from": "00:00", "to": "24:00", + "eur_per_kwh": 0.25}]})) == []) +check("the shipped default token is empty and the option is a password field", + P.PRICE_OPTION_DEFAULTS["price_service_token"] == "" + and "price_service_token: password?" in open( + os.path.join(HERE, "config.yaml")).read()) +check("config.yaml offers the new provider id", + "price_service" in open(os.path.join(HERE, "config.yaml")).read() + .split("price_provider: list(")[1].split(")")[0]) +check("only this provider asks for the chain; the others take the road they " + "always took", + [pid for pid in sorted(PROVIDERS) + if C.uses_chain({"price_provider": pid})] == ["price_service"] + and not C.uses_chain({"price_provider": "off"})) + +# =========================================================================== # +print("\n11. the real transport, against a local server") +async def _wire(): + from aiohttp import web + seen = [] + hold = {"mode": "ok"} + + async def handler(request): + seen.append(dict(request.headers)) + if hold["mode"] == "slow": + await asyncio.sleep(3) + if hold["mode"] in ("401", "403", "429", "503", "418"): + return web.Response(status=int(hold["mode"]), text="nope " + TOKEN) + if request.headers.get("Authorization") != "Bearer " + TOKEN: + return web.Response(status=401, text="bad token") + return web.Response(body=b'{"ok": 1}') + app = web.Application() + app.router.add_get("/v1/dayahead/BE", handler) + runner = web.AppRunner(app) + await runner.setup() + site = web.TCPSite(runner, "127.0.0.1", 0) + await site.start() + port = site._server.sockets[0].getsockname()[1] + url = f"http://127.0.0.1:{port}/v1/dayahead/BE" + res = {} + hdr = {"Authorization": "Bearer " + TOKEN} + res["ok"] = await P.https_get(url, 3, headers=hdr) + res["seen"] = nth(seen, -1) + for st in ("401", "403", "429", "503", "418"): + hold["mode"] = st + try: + await P.https_get(url, 3, headers=hdr) + res[st] = "no error" + except SourceUnavailable as e: + res[st] = str(e) + except Exception as e: # noqa: BLE001 + res[st] = "LEAK " + repr(e) + hold["mode"] = "slow" + try: + await P.https_get(url, 0.4, headers=hdr) + res["slow"] = "no error" + except SourceUnavailable as e: + res["slow"] = str(e) + except Exception as e: # noqa: BLE001 + res["slow"] = "LEAK " + repr(e) + await runner.cleanup() + try: + await P.https_get(url, 1, headers=hdr) + res["closed"] = "no error" + except SourceUnavailable as e: + res["closed"] = str(e) + except Exception as e: # noqa: BLE001 + res["closed"] = "LEAK " + repr(e) + # without headers it is the old transport: a status raises aiohttp's error + return res + + +_w = run(_wire()) +check("the token went on the wire as a bearer header, and the body came back", + _w["seen"].get("Authorization") == "Bearer " + TOKEN + and _w["ok"] == b'{"ok": 1}') +check(f"401 and 403 are 'refused the token', with the status and no body " + f"({_w['401']!r})", + _w["401"] == "the price service refused the token (401)" + and _w["403"] == "the price service refused the token (403)") +check(f"429, 5xx and other statuses have their own plain words " + f"({_w['429']!r}, {_w['503']!r}, {_w['418']!r})", + _w["429"] == "the price service says too many requests (429)" + and _w["503"] == "the price service has an error (503)" + and _w["418"] == "the price service answered 418") +check(f"a timeout and a refused connection are SourceUnavailable with the " + f"exception's NAME only ({_w['slow']!r}, {_w['closed']!r})", + _w["slow"].startswith("the price service did not answer (") + and _w["closed"].startswith("the price service did not answer (")) +check("...and nothing the server sent back, token echo included, reached any " + "message", all(TOKEN not in v for k, v in _w.items() + if isinstance(v, str))) + +# =========================================================================== # +print("\n12. restart in the middle of an outage, and the shipped logger names") +_c = make_ctx() +refresh(_c) +_c.net.down() +_hist2 = PriceHistory(_c.hist.path) # a restart: new objects +_c2 = type("C", (), {"clock": _c.clock, "net": _c.net, "hist": _hist2, + "watch": C.PriceWatch(), "o": _c.o})() +_n = mark() +_p = refresh(_c2, NOW + timedelta(hours=1)) +check("a restart with every source down plans from what the file kept", + _p.step == C.STEP_STORED and covers_now(_p, _c2.clock.now)) +check("...and says so once, at WARNING", + since_mark(_n, logging.WARNING) + == ["No price source answers, so the add-on uses the prices it saved."]) +check("the chain logs under goodwe.price.chain", + all(r.name == "goodwe.price.chain" for r in TAP.records[_n:] + if "price source" in r.getMessage() or "price service" + in r.getMessage())) + +for d in _TMP: + shutil.rmtree(d, ignore_errors=True) + +print() +if fails: + print(f"{len(fails)} FAILED: " + "; ".join(fails)) + sys.exit(1) +print("all checks passed") diff --git a/goodwe_controller/translations/en.yaml b/goodwe_controller/translations/en.yaml index 2db6dd1..881a027 100644 --- a/goodwe_controller/translations/en.yaml +++ b/goodwe_controller/translations/en.yaml @@ -199,6 +199,9 @@ configuration: price_feed_url: name: "Tariff · Price feed URL override" description: "Leave it empty to use the source's own feed. An override is sent exactly as typed and often returns only one day." + price_service_token: + name: "Tariff · Price service token" + description: "For the price service only. The token you were given for it. Leave it empty with any other price source." price_horizon_h: name: "Tariff · Price look-ahead (hours)" description: "How many hours of prices to ask the source for." diff --git a/goodwe_controller/translations/nl.yaml b/goodwe_controller/translations/nl.yaml index b48c735..12cfaf8 100644 --- a/goodwe_controller/translations/nl.yaml +++ b/goodwe_controller/translations/nl.yaml @@ -180,6 +180,9 @@ configuration: price_feed_url: name: "Tarief · Afwijkende URL voor de prijsfeed" description: "Laat leeg om de eigen feed van de bron te gebruiken. Een afwijkende URL wordt letterlijk verstuurd en geeft vaak maar één dag terug." + price_service_token: + name: "Tarief · Token van de prijsdienst" + description: "Alleen voor de prijsdienst. Het token dat u ervoor kreeg. Laat leeg bij een andere prijsbron." price_horizon_h: name: "Tarief · Prijzen vooruit (uren)" description: "Hoeveel uur aan prijzen de add-on bij de bron opvraagt."