feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata - #735
Draft
HuggeK wants to merge 6 commits into
Draft
feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata#735HuggeK wants to merge 6 commits into
HuggeK wants to merge 6 commits into
Conversation
Radiation-bearing providers previously produced one flat estimate, rated x (W/m2 / 1000), which ignored panel orientation entirely: a south-facing 35 deg roof and a flat one got the same forecast. When per-plane geometry is configured (Weather tab: tilt/azimuth/kWp), global horizontal irradiance is now projected onto each plane via the existing sunpos physics and summed. Providers publishing only GHI get an Erbs correlation to split direct from diffuse first. Sites with no arrays keep the previous behaviour, and Forecast.Solar - which already returns site-calibrated watts - is deliberately left untouched so its numbers are not scaled twice. Split out of the original srcfl#718 per review: this half is user-visible and improves every radiation-bearing provider today. The STRANG client and performance scoring that shared that branch move to a follow-up PR. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
…coverage STRANG becomes a real irradiance source rather than an unreferenced client, every external data source now says where in the world it works, and the location picker moves off Leaflet. STRANG's parameter set was mapped against the live API because SMHI's apidocs pages 404: exactly 116-122 exist. Identification was confirmed by physics rather than by guessing - at solar noon 121 + 122 = 723.0 + 87.5 = 810.5, exactly parameter 117, and 119 caps at 60, i.e. minutes within the hour. STRANG publishes no cloud cover; it is a radiation model. Cloudiness is instead derived from sunshine duration as 1 - minutes/60, which is observed rather than inferred but coarser: blind to thin cirrus, undefined at night. CloudCover() therefore returns an explicit unknown instead of defaulting to clear, because those two lead to opposite decisions. The new coverage registry makes an existing silence explicit. STRANG is Nordic-only and every price provider is European, so sites elsewhere were getting empty results with no explanation (srcfl#726). GET /api/data-sources now reports area, countries, licence and whether each source reaches this site, and the Weather tab renders it under the map. Bounds are advisory: STRANG's grid is rotated, so a lat/lon box can only ever be a superset - all four in-box corners were probed and returned no data. False is definitive, true means worth trying. Scoring now declines to start outside the domain instead of retrying nightly forever. Stacked on srcfl#718, which carries the plane-of-array wiring this builds on. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
A Swedish site can stop typing panel angles in by hand. The new roofmodel/ module reads Lantmateriet LiDAR around the site and recovers each roof face's tilt, azimuth and usable area, which pre-fill weather.pv_arrays. Structured as a separate Python package alongside optimizer/, reached at arm's length: core spawns it, passes coordinates and the operator's own Geotorget credentials, and reads one versioned roof_model.json from stdout. Segmentation drags in a compiled point-cloud stack and runs for minutes, so a time-boxed subprocess keeps it off the control tick and lets it be absent entirely - the normal case, since the data exists only for Sweden. The pipeline follows the SPAN method: iterative RANSAC pulls one surface at a time out of the cloud, then DBSCAN splits faces sharing a plane equation but not a location, because two wings of a building fit the same plane and are not the same roof. Method only - no code is taken from SPAN's GPL QGIS plugin, and the dependencies (numpy, scikit-learn, requests) are all BSD. SWEREF 99 TM is implemented directly rather than via pyproj: it is one projection with fixed parameters, and a full PROJ build is disproportionate for that. Verified against the projection's exact analytic properties - easting on the central meridian is exactly 500000 at every latitude - and by sub-millimetre round trips from Smygehuk to Treriksroeset. Applying derived arrays to config is deliberately a separate act. Derivation is a best guess from a point cloud that may be years old, and silently rewriting an operator's panel config is a change they should make knowingly. Also generalises the __pycache__ ignore rule, which was optimizer-specific and would otherwise need repeating per module. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
The roofmodel tests passed on Windows and failed on Linux, which is where CI
runs them. Two separate bugs, both caused by stubbing the module with a shell
script:
- The stub handed back "sh" as the command, and the service invokes it as
`<cmd> -m ftw_roofmodel ...`. dash reads that as "run the script named
ftw_roofmodel", cannot open it, and exits 2. The stub never ran at all, so
five tests failed against an error that had nothing to do with what they
were asserting.
- The module-error stub echoed an unquoted JSON document. dash strips the
double quotes, so `{"error":"..."}` reached the parser as `{error:...}` and
could not be unmarshalled. cmd.exe keeps them, which is the only reason
that test ever passed.
The stub is now this test binary re-executed with an environment variable
telling it which behaviour to play. Nothing goes through a shell, so there is
no quoting to get wrong and no per-platform behaviour to diverge. The timeout
test consequently runs everywhere instead of being skipped on Windows.
Two assertions the shell stub could not make: that the site, credentials and
radius actually survive the process boundary, and that PYTHONPATH is set --
without it the module is only importable if installed system-wide, which on a
Pi it is not. Also that --vostok stays absent unless configured, so a GPL tool
is never invoked by default.
The Python module's tests did not run in CI at all: the workflow runs
`pytest -q optimizer/tests` and nothing else, so 75 tests covering the plane
fitting every derived tilt and azimuth depends on were never executed upstream.
Added a roofmodel job mirroring the optimizer's, wired into the required-check
gate so a failure blocks rather than being reported and ignored. It installs
without the `geo` extra on purpose: LAZ decoding pulls a compiled backend, and
everything except the point-cloud read is exercised without it.
Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
Settings -> Weather gains a Geotorget credential form and a building picker. Press "Find buildings here" and the footprints near the marker are drawn on the map and listed beside it; click yours, press "Read roof from LiDAR", and the PV arrays fill in with one entry per usable roof face. The form is filled but nothing is saved. The operator sees the numbers, corrects what is wrong and presses Save. FTW does not rewrite a panel configuration on its own: the derivation is a guess from a scan that may be years old, and only the operator knows whether that face has panels on it. Picking a building is not cosmetic. Without a footprint the module segments whatever stands inside its search radius, and the plane fitting is global -- a fitted plane is infinite, so a roof at azimuth 180 is z = f(y) with no x term and extends across the whole tile. A second building sharing that ridge orientation lands inside its inlier band however far away it is, and the two lose returns to each other. Measured on a synthetic pair: a detached garage recovered 93% of its true area and split into two fragments while coplanar with the house, against 100% and one clean face once clipped to its own footprint. The clip buffers the outline by a metre first, because roofs overhang their walls and the eaves carry the lowest returns. Frames are detected rather than assumed. GeoJSON mandates WGS84 but Lantmateriet publishes this catalogue in SWEREF 99 TM and its STAC search takes a SWEREF bbox, so ring coordinates are classified by magnitude -- six- and seven-figure numbers are projected metres, degrees never are. New GET /api/roofmodel/buildings lists footprints as GeoJSON, honouring an explicit lat/lon so the picker can search where the marker is rather than where the last save put it. POST /api/roofmodel/derive accepts a building_id. GET /api/roofmodel reports has_credentials so the UI can stop asking. The Geotorget token now masks and restores like every other secret. It never appears in an API response, and saving an unrelated setting no longer wipes it -- the settings form returns the blank it was given, which without PreserveMaskedSecrets would have deleted the stored credential. docs/roof-geometry.md covers ordering the two Geotorget products, what the derived kWp does and does not mean (an upper bound on what fits, not what is installed), and what each failure message is telling you. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
Both roof-geometry products are STAC APIs behind one Geotorget account, and they differ only in what their items point at: Byggnad Nedladdning, vektor delivers GeoPackage, Laserdata Nedladdning, Skog delivers LAZ organised as COPC. Assets are now selected by declared media type rather than by guessing at asset key names, so a catalogue that calls its asset "punktmoln" instead of "data" keeps working, and a thumbnail is never handed back as a point cloud. Building footprints are read straight out of the GeoPackage with the standard library. A GeoPackage is a SQLite database holding geometry as WKB, both published formats with fixed layouts, so this costs about a hundred lines against a GDAL dependency that will not install on a Pi without a compiler. Previously only inline STAC geometry was handled and the asset-backed case -- the normal one -- returned nothing at all. Because COPC indexes points into an octree, picking a building now also makes the download small: only the octree nodes covering that footprint are range-requested, instead of a 2.5 km tile running to hundreds of megabytes. Plain .laz assets, hosts that ignore Range, and builds of laspy without COPC support all fall back to reading the tile whole -- slower, same answer -- and the model records which path ran as source.fetch, since that also decides what returns_in_radius is counted over. Also corrects the coplanar-absorption measurement quoted in buildings.py and docs/roof-geometry.md. A fresh run does not reproduce the 93%/two-fragment figures: one pass over a house and a garage 40 m apart consumes all 576 of the house's south-face returns and all 256 of the garage's as one surface, and gives identical output at every separation from 3 m to 40 m -- which is the signature of a global infinite-plane fit rather than a proximity effect. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Derives PV array tilt, azimuth and kWp from Lantmäteriet open geodata, so a
Swedish site can stop typing panel angles in by hand.
Shape
roofmodel/is a separate Python package alongsideoptimizer/, reached atarm's length: core spawns it, passes coordinates and the operator's own
Geotorget credentials, and reads back one versioned
roof_model.json.That boundary is the point. LiDAR segmentation drags in a compiled point-cloud
stack, runs for minutes, and is onboarding-time work rather than runtime work.
In a time-boxed subprocess it cannot stall the control tick, cannot leak into the
daemon, and can be absent entirely — which is the normal case, since the data
exists only for Sweden.
Method
Follows the SPAN paper
(Yavuzdoğan, Renewable Energy 2023): iterative RANSAC pulls one roof surface at
a time out of the cloud, then DBSCAN splits faces that satisfy the same plane
equation but sit in different places — two wings of a building fit one plane and
are not one roof.
The method only. No code is taken from SPAN's GPL QGIS plugin; dependencies are
numpy, scikit-learn and requests, all BSD, so the module carries no copyleft
obligation.
Two implementation choices worth flagging
Plane fitting is total-least-squares, not
z = ax + by + c. An ordinaryregression of height on position minimises vertical error, which
systematically biases steep roofs. The smallest singular vector of the
mean-centred points minimises perpendicular distance instead, and stays
well-conditioned as the surface approaches vertical.
SWEREF 99 TM is implemented directly rather than via pyproj. Lantmäteriet
publishes in EPSG:3006 and FTW stores WGS84, so every request crosses that
boundary — but it is exactly one projection with fixed parameters, and shipping
a full PROJ build for that is disproportionate on a Pi. The Krüger series is
~120 lines and verified two ways: against the projection's exact analytic
properties (easting on the central meridian is exactly 500000 at every
latitude, and the origin maps to (0, 500000)), and by sub-millimetre round trips
from Smygehuk to Treriksröset.
What it produces
GET /api/roofmodelreports availability and coverage;POST /api/roofmodel/deriveruns a derive and returns the proposal. Lantmäteriet alsojoins
/api/data-sourcesas a Sweden-only, credential-gated source.Deliberately not wired: auto-applying to config
The RFC sketched automatic auto-fill of
weather.pv_arrays. This PR returns theproposal instead and leaves applying it as a separate explicit act. Derivation is
a best guess from a point cloud that may be years old, packing factor is an
assumption, and silently rewriting an operator's panel config is a change they
should make knowingly. Happy to wire the auto-apply if you'd rather — it is a
small change on top, but it is a behaviour decision, not an implementation
detail.
Testing
57 Python tests. Every geometry case is a synthetic roof whose tilt, azimuth
and area are known exactly by construction, and the generator is the inverse of
the code under test rather than a copy of it — so the assertions check recovered
geometry against ground truth, not against a previous run. Covered: six
orientations, gable splitting, two buildings sharing a plane, sloped-vs-footprint
area, realistic LiDAR noise, wall rejection, determinism.
11 Go tests. The subprocess contract is exercised end-to-end against a stub
module, so argument passing, stdout parsing, stderr error surfacing, schema
rejection and the timeout are all real rather than mocked.
A limitation the tests pin rather than hide
Sweden is a long diagonal, so no lat/lon rectangle traces its border — any box
containing Sweden also contains parts of Norway. Oslo is the clearest case: west
of Sweden, inside the box.
Rather than tighten the box until it starts excluding real Swedish addresses near
the border, the box stays generous and the STAC search is the authority: no tiles
come back, and the module reports "Sweden only". There are tests for both halves —
that Oslo is admitted by design, and that Strömstad, Haparanda, Karesuando and
Smygehuk are never excluded.
Not covered: the live Lantmäteriet fetch. It needs Geotorget credentials for
Byggnad-vektor and Laserdata-skog, which are the operator's own; the HTTP layer
is tested against a fake session instead, including the documented quirk that
properties.datetimeis still being backfilled through 2026 and the laser stripdatumis the fallback. To try it live:Part of the roadmap in discussion #717.