goodwe-addon/goodwe_controller/scenarios/fetch_epex_be.py

#!/usr/bin/env python3
"""DATA-02: fetch 12 months of EPEX BE day-ahead prices, in §9.3's shape.

    python3 scenarios/fetch_epex_be.py --start 2025-09-01 --end 2026-09-01 \
        --out scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json

Run from `goodwe_controller/`. Rewrites the output file from scratch; the
committed dataset is exactly what this produces, and `--verify` re-derives every
claim the file makes about itself without going to the network.


WHAT THIS IS NOT
================
NOT wired into the add-on, and it must never be. `price.py`'s `GenericDayAhead`
is the live path and it has its own egress check; this is a one-shot archival
fetch that produces a file for REPLAY, run by hand by a developer. Nothing under
`app/` imports it.


THE SOURCE, AND WHY THIS ONE
============================
`https://api.energy-charts.info/price?bzn=BE` - Fraunhofer ISE's Energy-Charts
API. No key, no registration.

⚠️ THE LICENCE WAS CHECKED BEFORE ANY DATA WAS COMMITTED, because for this
source it is NOT uniform across bidding zones. The API's own OpenAPI document
(`https://api.energy-charts.info/openapi.json`, /price) splits the zones in two:

    "The data for the following bidding zones is licensed as CC BY 4.0 from
     Bundesnetzagentur | SMARD.de and is published without changes:
     AT, BE (Belgium), CH, CZ, DE-LU, ..."

    "The data for the other bidding zones is for private and internal use only.
     The utilization of any data, whether in its raw or derived form, for
     external or commercial purposes is expressly prohibited. Should you require
     licensing for market-related data, please direct your inquiries to the
     original data providers, including but not limited to EPEX SPOT SE."

BE is in the FIRST list, so redistribution is permitted with attribution and the
dataset is committed rather than left as a fetch script and a checksum. Every
response also carries the licence per-response in `license_info`, and this
script REFUSES to write a file if that field ever stops saying CC BY 4.0 - a
licence checked once by a human and assumed forever is a licence that changes
silently. Had BE been in the second list, the correct output of this card would
have been this script and a checksum and no data.

Attribution, carried in the dataset's own `provenance` block and reproduced
here: data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts
(Fraunhofer ISE), https://energy-charts.info, published without changes.

Cross-checked against the upstream on 2026-08-31: SMARD's own chart_data series
`4996/DE` over 2026-04-10 matched this API's BE series on all 96 quarter-hours
with a maximum absolute difference of 0.00 EUR/MWh. "Published without changes"
is a claim worth checking once; `--verify` does not repeat it, because it is a
claim about the SOURCE and not about the file.


RAW WHOLESALE, AND THE ASSUMPTION THAT IS NOT BAKED IN
======================================================
⚠️ WHAT COMES BACK IS RAW WHOLESALE, EUR/MWh, DAY-AHEAD AUCTION CLEARING PRICE.
It is NOT all-in: no supplier markup, no distribution or transmission tariff, no
levies, no VAT. §9.3's canonical point is all-in, and TARIFF-02's whole trap is
a figure that already contains a component being handed to something that adds
it again.

So this file does NOT add one. `eur_per_kwh` in the output is the wholesale
figure converted from EUR/MWh, `provenance.all_in` is `false` beside it, and the
markup/tax assumption a consumer needs lives in `provenance.all_in_assumption`
as NUMBERS TO APPLY rather than numbers already applied. test_arbitrage.py
section 7 applies them where a reader can watch it happen.

The assumption's SHAPE is the part that is not arbitrary, and it decides which
conclusions depend on the guessed numbers at all:

    all_in(h) = (wholesale(h) + markup + network_and_levies) * (1 + vat)

  * The two ADDITIVE terms cancel out of a spread: max - min is unchanged by
    them. So how often a day clears `cycle_cost_threshold_eur` does NOT depend
    on the markup guess - only on the VAT multiplier.
  * They do NOT cancel out of §17's negative-price guard, which compares ONE
    hour's all-in price against zero. A wholesale hour at -0.02 EUR/kWh is
    comfortably positive all-in once ~0.12 of fixed charges is added.

Both are asserted arithmetically in section 7 rather than believed, and the
trade count is reported on the raw spread as well, so a reader can see exactly
how much of the answer rests on the guess.

The numbers themselves (2026 Belgian residential dynamic contract, order of
magnitude) are a STATED GUESS, labelled as one everywhere they appear. They are
not a measurement and this card had nothing to measure them with.


HOURS, QUARTER-HOURS, AND THE SEAM IN THE MIDDLE OF THE YEAR
============================================================
⚠️ THE SOURCE CHANGES RESOLUTION PART-WAY THROUGH ANY RECENT 12 MONTHS. EPEX
day-ahead moved to a 15-minute market time unit in October 2025 and the feed
follows: 2025-09-10 comes back as 24 hourly points, 2025-12-10 as 96
quarter-hourly ones. An importer that assumed either one silently loses three
quarters of the year, or reads a quarter as an hour.

The canonical format is hourly, so quarter-hours are aggregated: an hour's price
is the unweighted mean of its four quarters. That is a STATED APPROXIMATION of
the hourly product, not the hourly product, and it errs in the direction that
matters here - averaging discards intra-hour variation, so the observed spread
is a LOWER bound on what a 15-minute strategy would see.
`provenance.resolution` records how many hours came from each source shape.

⚠️ AND A PARTIAL HOUR IS DROPPED, NOT AVERAGED OVER WHAT ARRIVED. An hour with
three of four quarters is an hour nobody published a complete price for; the
mean of three would be an invented number carrying no flag. Dropped hours are
listed in `provenance.gaps` with their timestamps, because a missing hour in a
feed is not a zero price, and a zero price is exactly the value that would make
everything look free.


SIZE: WHY THIS IS PLAIN JSON AND NOT A `.gz`, MEASURED
======================================================
The directory's existing captures are gzipped, so the default would have been to
gzip this too. Measured on the committed file instead of assumed:

    working tree, plain JSON                             893 171 bytes
    what GIT actually stores for it (zlib, loose object)  74 590 bytes
    what git would store for a committed `.json.gz`       56 504 bytes

⚠️ GIT ALREADY COMPRESSES BLOBS, so gzipping buys 18 KB - 24 % of an object
that is already 75 KB, in a repository whose three existing P1 captures cost
27 KB, 136 KB and 140 KB. Eighteen kilobytes is not worth making the provenance
block unreadable in a pull request, and the licence claim in that block is
precisely the thing a reviewer of this card has to read and check. Plain JSON
also greps, diffs line-by-line when a future refetch changes something, and is
what `--verify` opens. `load()` still handles `.gz` for a future dataset large
enough to want it; this one is not.

The P1 captures are gzipped for the opposite reason and it is not an
inconsistency: they are hundreds of thousands of telegram lines that no human
reads, and there is no metadata in them to review.


TIME
====
Every instant is carried as UTC and rendered with `app.price.brussels()`, the
module that already owns the EU DST rule without a tz database. There is no
second answer to that question here and there must not be: the point of
`brussels()` is that the 02:00-03:00 hour which happens twice in October is two
instants, and a 24-slot local-hour importer collapses them.

So a year is 8760 hours only if it contains no transition, or contains both and
they cancel - 25 in October, 23 in March. `--verify` therefore asserts the
PER-DAY counts, not the total, because the total is exactly the number a broken
importer also produces.
"""

import argparse
import gzip
import hashlib
import json
import os
import sys
import time
import urllib.error
import urllib.request
from collections import Counter, defaultdict
from datetime import datetime, timedelta, timezone

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from app.price import brussels                                   # noqa: E402

API = "https://api.energy-charts.info/price"
BZN = "BE"
LICENCE_MUST_CONTAIN = "CC BY 4.0"
UTC = timezone.utc

ATTRIBUTION = ("Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, "
               "retrieved via the Energy-Charts API (Fraunhofer ISE), "
               "https://energy-charts.info. Published without changes.")

# ⚠️ A GUESS, LABELLED AS ONE, AND APPLIED BY THE CONSUMER - NOT HERE.
# Order-of-magnitude 2026 Belgian residential dynamic contract. Nothing in this
# repository measured any part of it.
ALL_IN_ASSUMPTION = {
    "form": "all_in = (wholesale + markup + network_and_levies) * (1 + vat)",
    "markup_eur_per_kwh": 0.020,
    "network_and_levies_eur_per_kwh": 0.100,
    "vat_fraction": 0.06,
    "status": "ASSUMPTION, NOT MEASURED. Stated so a consumer applies it "
              "deliberately; the committed eur_per_kwh has none of it applied.",
    "note": "The two additive terms cancel out of any spread, so an arbitrage "
            "trade count depends on vat_fraction only. They do NOT cancel out "
            "of the negative-price guard (17), which compares one hour's "
            "all-in price against zero.",
}


PACE_S = 3.0            # between chunks, unconditionally
RETRIES = 6


def _get(url: str) -> dict:
    """One GET, backing off on 429 and 5xx.

    ⚠️ THIS IS A FREE PUBLIC SERVICE AND IT RATE-LIMITS. The first run of this
    script was refused with 429 on its fifth chunk. Backing off is the polite
    behaviour; it is also the correct one, because the alternative - catching
    the error and carrying on - would produce a file that is silently missing a
    month and says nothing about it. A refused request is not an empty month.
    """
    req = urllib.request.Request(
        url,
        headers={"User-Agent": "goodwe-addon/DATA-02 (one-shot archival fetch)"})
    wait = 5.0
    for attempt in range(1, RETRIES + 1):
        try:
            with urllib.request.urlopen(req, timeout=90) as r:
                return json.loads(r.read().decode("utf-8"))
        except urllib.error.HTTPError as e:
            if e.code not in (429, 500, 502, 503, 504) or attempt == RETRIES:
                raise
            print(f"    HTTP {e.code}; backing off {wait:.0f}s "
                  f"(attempt {attempt}/{RETRIES})", file=sys.stderr)
            time.sleep(wait)
            wait *= 2
    raise SystemExit(f"unreachable: {url}")


def local_midnight_utc(day: str) -> datetime:
    """The INSTANT at which Brussels midnight begins on `day` (YYYY-MM-DD).

    ⚠️ THE WINDOW IS LOCAL, NOT UTC, AND THE FIRST RUN OF THIS SCRIPT PROVED WHY.
    Bounding it at 00:00 UTC starts the year at 02:00 CEST, which makes the first
    Brussels day 22 hours long - and 22 is not one of the two lengths a
    transition produces, so `report()` refused the file. A dataset whose first
    day is short is a dataset whose "days per year" arithmetic is quietly wrong
    everywhere downstream.

    Derived from `brussels()` by trying the only two offsets Belgium has and
    keeping the one that round-trips, rather than by restating the EU rule here.
    """
    for hours in (1, 2):
        guess = datetime.fromisoformat(f"{day}T00:00:00").replace(
            tzinfo=timezone(timedelta(hours=hours))).astimezone(UTC)
        if brussels(guess).isoformat() == f"{day}T00:00:00+0{hours}:00":
            return guess
    raise SystemExit(f"no Brussels midnight for {day!r} - brussels() disagrees "
                     f"with itself, which is a bug in price.py, not here.")


def _months(start: datetime, end: datetime):
    """[start, end) in month-sized chunks, so a bad chunk names its own window."""
    a = start
    while a < end:
        b = min(end, (a.replace(day=1) + timedelta(days=32)).replace(day=1))
        yield a, b
        a = b


def fetch(start: datetime, end: datetime):
    """Raw {unix_second: EUR/MWh} over [start, end), deduplicated.

    ⚠️ A `null` price is DROPPED, never coerced. The API returns null for an
    interval it holds no result for, and `or 0.0` is the version of this line
    that ships a year of free electricity.
    """
    seen, licences, nulls = {}, Counter(), 0
    for a, b in _months(start, end):
        # ⚠️ THE API'S `end` IS AN INCLUSIVE LOCAL DAY, AND THE WINDOW'S IS AN
        # EXCLUSIVE INSTANT. Ask for the day containing the LAST hour of the
        # chunk, derived from the instant rather than by subtracting a calendar
        # day from the bound: `b - 1 day` drops the final local day whenever the
        # window ends at local midnight, which is exactly how this window ends.
        # The first run lost 2026-08-31 that way, silently - 8736 points instead
        # of 8760 - and the points-plus-gaps accounting in report() is what
        # noticed. A day nobody asked for is not a day with no prices.
        first = brussels(a).date().isoformat()
        last = brussels(b - timedelta(hours=1)).date().isoformat()
        url = f"{API}?bzn={BZN}&start={first}&end={last}"
        d = _get(url)
        lic = str(d.get("license_info", ""))
        licences[lic] += 1
        if LICENCE_MUST_CONTAIN not in lic:
            raise SystemExit(
                f"REFUSING TO WRITE. {url} came back under licence {lic!r}, "
                f"which does not say {LICENCE_MUST_CONTAIN!r}. The BE zone was "
                f"redistributable when this card ran; if that has changed, the "
                f"right output is this script and a checksum, not the data.")
        unit = str(d.get("unit", ""))
        if "MWh" not in unit:
            raise SystemExit(
                f"REFUSING TO WRITE. {url} reports unit {unit!r}, not EUR/MWh. "
                f"A silent unit change is a factor of 1000 on every price.")
        for t, p in zip(d["unix_seconds"], d["price"]):
            if p is None:
                nulls += 1
                continue
            if start.timestamp() <= t < end.timestamp():
                seen[int(t)] = float(p)
        print(f"  {a.date()} .. {b.date()}  {len(d['price']):5d} intervals",
              file=sys.stderr)
        time.sleep(PACE_S)
    return seen, licences, nulls


def to_hourly(raw: dict):
    """Hourly means keyed by UTC hour. Returns (hours, gaps, shape_counts).

    An hour is emitted only when every interval the source used for it arrived:
    1 of 1 where the source was hourly, 4 of 4 where it was quarter-hourly.
    Anything else is a gap and is named as one.
    """
    buckets = defaultdict(dict)
    for t, p in raw.items():
        buckets[t - (t % 3600)][t % 3600] = p

    hours, gaps, shapes = {}, [], Counter()
    for h in sorted(buckets):
        got = buckets[h]
        if set(got) == {0}:
            shapes["hourly"] += 1
            hours[h] = got[0]
        elif set(got) == {0, 900, 1800, 2700}:
            shapes["quarter_hourly_mean"] += 1
            hours[h] = sum(got.values()) / 4.0
        else:
            gaps.append({
                "hour_utc": datetime.fromtimestamp(h, UTC).isoformat(),
                "intervals_present": sorted(got),
                "why": "incomplete hour - dropped rather than averaged over "
                       "what arrived; a partial read is not an observation",
            })
    return hours, gaps, shapes


def canonical(hours: dict) -> list:
    """9.3 points. THREE FIELDS, rendered at the Brussels offset for the instant."""
    return [{"start": brussels(datetime.fromtimestamp(h, UTC)).isoformat(),
             "eur_per_kwh": round(hours[h] / 1000.0, 6) + 0.0,
             "provisional": False}
            for h in sorted(hours)]


def expected_hours(start: datetime, end: datetime) -> int:
    """Whole hours in the window. Derived, rather than 8760 typed in."""
    return int((end - start).total_seconds() // 3600)


def per_local_day(points) -> Counter:
    """Points per BRUSSELS calendar day - the count that exposes a DST bug."""
    n = Counter()
    for p in points:
        n[p["start"][:10]] += 1
    return n


def digest(points) -> str:
    """A checksum over the POINTS, so reformatting the envelope does not move it."""
    h = hashlib.sha256()
    for p in points:
        h.update(f"{p['start']}|{p['eur_per_kwh']:.6f}|{int(p['provisional'])}\n"
                 .encode())
    return h.hexdigest()


def load(path: str) -> dict:
    op = gzip.open if path.endswith(".gz") else open
    with op(path, "rt", encoding="utf-8") as f:
        return json.load(f)


def report(doc, *, start=None, end=None) -> int:
    """Re-derive every claim the file makes about itself. Returns failures."""
    pts = doc["prices"]
    prov = doc["provenance"]
    bad = 0

    def ck(name, cond):
        nonlocal bad
        print(f"  {'ok  ' if cond else 'FAIL'} {name}")
        if not cond:
            bad += 1

    ck(f"the checksum over the {len(pts)} points matches the recorded one "
       f"({prov['sha256_of_points'][:16]}...)",
       digest(pts) == prov["sha256_of_points"])
    ck("every point has exactly the three canonical fields and nothing else",
       all(set(p) == {"start", "eur_per_kwh", "provisional"} for p in pts))
    ck("every start carries a UTC offset, and it is +01:00 or +02:00 - a naive "
       "timestamp here is a wall clock plus an assumption",
       bool(pts) and all(p["start"][-6:] in ("+01:00", "+02:00") for p in pts))
    ck("the series is strictly increasing in time, so no instant is duplicated",
       all(datetime.fromisoformat(a["start"]) < datetime.fromisoformat(b["start"])
           for a, b in zip(pts, pts[1:])))
    ck("consecutive points are exactly one hour apart AS INSTANTS, so the two "
       "02:00s of the October day are two hours rather than one",
       all((datetime.fromisoformat(b["start"])
            - datetime.fromisoformat(a["start"])) == timedelta(hours=1)
           for a, b in zip(pts, pts[1:])))

    days = per_local_day(pts)
    odd = {d: n for d, n in days.items() if n != 24}
    ck(f"exactly two Brussels days are not 24 hours long, and they are the "
       f"transitions: {sorted(odd.items())}",
       sorted(odd.values()) == [23, 25] and len(odd) == 2)
    spring = [d for d, n in odd.items() if n == 23]
    autumn = [d for d, n in odd.items() if n == 25]
    ck(f"the 23-hour day {spring} is in March and the 25-hour day {autumn} is "
       f"in October, which is the direction the EU rule actually runs",
       len(spring) == 1 and len(autumn) == 1
       and spring[0][5:7] == "03" and autumn[0][5:7] == "10")
    if autumn:
        twice = [p["start"] for p in pts
                 if p["start"].startswith(autumn[0] + "T02")]
        ck(f"...and 02:00 on {autumn[0]} occurs TWICE at two different offsets "
           f"{twice} - exactly the case a 24-slot local-hour array collapses",
           len(twice) == 2 and twice[0][-6:] == "+02:00"
           and twice[1][-6:] == "+01:00")
        ck(f"...and the two carry DIFFERENT prices, so the 25th hour is a real "
           f"observation and not the same one written down twice",
           len({p["eur_per_kwh"] for p in pts
                if p["start"].startswith(autumn[0] + "T02")}) == 2)
    if spring:
        ck(f"...and 02:00 on {spring[0]} does not occur at all, because that "
           f"wall clock reading never happened in Belgium",
           not any(p["start"].startswith(spring[0] + "T02") for p in pts))

    if start and end:
        want = expected_hours(start, end)
        ck(f"{len(pts)} points for a window of {want} whole hours, and the "
           f"{len(prov['gaps'])} recorded gaps account for the difference "
           f"exactly", len(pts) + len(prov["gaps"]) == want)

    # ⚠️ A ZERO IS NOT AUTOMATICALLY A DEFECT HERE, AND SAYING IT WAS WOULD BE
    # THE WRONG LESSON. The Belgian market really does clear at 0.00 EUR/MWh in
    # solar-glut hours; this window contains such hours. What must be impossible
    # is a zero that arrived by COERCION - a dropped read turned into a free
    # hour. That is excluded structurally rather than by outlawing the value:
    # nulls are dropped, incomplete hours become gaps, and the accounting above
    # shows points + gaps covering every hour of the window, so every emitted
    # point stands for intervals the source actually published.
    zeros = [p["start"] for p in pts if p["eur_per_kwh"] == 0.0]
    ck(f"the {len(zeros)} hours priced at exactly zero match the count recorded "
       f"at build time, so no zero appeared between the fetch and the file",
       len(zeros) == prov["exact_zero_hours"])
    ck("...and none of them is a coerced read: every emitted hour is backed by "
       "published intervals, which is what the points-plus-gaps accounting "
       "above establishes",
       len(prov["gaps"]) == 0 or all("intervals_present" in g
                                     for g in prov["gaps"]))
    ck("the figures are declared RAW WHOLESALE, and the all-in assumption is "
       "recorded as an assumption rather than applied",
       prov["all_in"] is False
       and "ASSUMPTION" in prov["all_in_assumption"]["status"])
    return bad


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--start", default="2025-09-01")
    ap.add_argument("--end", default="2026-09-01",
                    help="exclusive; both bounds are BRUSSELS-local midnights")
    ap.add_argument("--out", default=None)
    ap.add_argument("--verify", default=None,
                    help="re-derive the claims of an existing file; no network")
    a = ap.parse_args()

    if a.verify:
        doc = load(a.verify)
        print(f"verifying {a.verify}")
        s = datetime.fromisoformat(doc["provenance"]["window_utc"]["start"])
        e = datetime.fromisoformat(doc["provenance"]["window_utc"]["end"])
        bad = report(doc, start=s, end=e)
        print("OK" if not bad else f"{bad} FAILED")
        return 1 if bad else 0

    start = local_midnight_utc(a.start)
    end = local_midnight_utc(a.end)
    out = a.out or f"scenarios/epex-be-dayahead-{a.start}_{a.end}.json"

    print(f"fetching {BZN} day-ahead {start.date()} .. {end.date()}",
          file=sys.stderr)
    raw, licences, nulls = fetch(start, end)
    hours, gaps, shapes = to_hourly(raw)
    pts = canonical(hours)

    doc = {
        "provenance": {
            "card": "DATA-02",
            "what": "EPEX BE day-ahead auction clearing prices, hourly",
            "source": f"{API}?bzn={BZN} (Energy-Charts API, Fraunhofer ISE)",
            "upstream": "Bundesnetzagentur | SMARD.de",
            "licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
            "licence_checked": "before committing anything; the API's OpenAPI "
                               "document lists BE among the CC BY 4.0 zones and "
                               "the remaining zones as private/internal use "
                               "only, so redistribution of THIS zone is "
                               "permitted with attribution",
            "licence_as_returned": sorted(licences),
            "attribution": ATTRIBUTION,
            "cross_check": "SMARD chart_data 4996/DE over 2026-04-10 matched "
                           "this feed on all 96 quarter-hours, max abs "
                           "difference 0.00 EUR/MWh (2026-08-31)",
            "retrieved_utc": datetime.now(UTC).replace(microsecond=0).isoformat(),
            "fetched_by": "scenarios/fetch_epex_be.py",
            "window_utc": {"start": start.isoformat(), "end": end.isoformat()},
            "unit": "EUR per kWh (source publishes EUR/MWh; divided by 1000)",
            "all_in": False,
            "all_in_note": "RAW WHOLESALE. No supplier markup, no network or "
                           "transmission tariff, no levies, no VAT. 9.3's "
                           "canonical point is all-in and this is NOT; the "
                           "consumer applies the assumption below, visibly.",
            "all_in_assumption": ALL_IN_ASSUMPTION,
            "resolution": {
                "note": "EPEX day-ahead moved to a 15-minute market time unit "
                        "in Oct 2025, part-way through this window.",
                "hours_from_hourly_source": shapes["hourly"],
                "hours_from_quarter_hourly_mean": shapes["quarter_hourly_mean"],
                "aggregation": "unweighted mean of the four quarters; a stated "
                               "approximation of the hourly product, and a "
                               "LOWER bound on the spread a 15-minute strategy "
                               "would have seen",
            },
            "timezone": "starts rendered at the Brussels offset by "
                        "app.price.brussels(); instants carried in UTC, so the "
                        "transition days come out 23 and 25 points by "
                        "construction rather than by a special case",
            "nulls_dropped": nulls,
            "gaps": gaps,
            "gaps_note": "hours the source did not publish completely. Dropped, "
                         "never zero-filled: a missing hour is not a free hour.",
            "points": len(pts),
            "exact_zero_hours": sum(1 for p in pts if p["eur_per_kwh"] == 0.0),
            "negative_hours": sum(1 for p in pts if p["eur_per_kwh"] < 0),
            "zero_note": "REAL CLEARINGS AT 0.00 EUR/MWh, not dropped reads. "
                         "Nulls are dropped and incomplete hours become gaps, "
                         "so points + gaps == hours in the window and no point "
                         "stands in for something that failed to arrive.",
            "sha256_of_points": digest(pts),
        },
        "prices": pts,
    }

    with open(out, "w", encoding="utf-8", newline="\n") as f:
        json.dump(doc, f, indent=1, ensure_ascii=False)
        f.write("\n")
    print(f"wrote {out}  {len(pts)} points  {os.path.getsize(out)} bytes",
          file=sys.stderr)
    print(f"verifying {out}")
    return 1 if report(doc, start=start, end=end) else 0


if __name__ == "__main__":
    raise SystemExit(main())