Skip to content

feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata - #735

Draft
HuggeK wants to merge 6 commits into
srcfl:masterfrom
HuggeK:roofmodel
Draft

feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata#735
HuggeK wants to merge 6 commits into
srcfl:masterfrom
HuggeK:roofmodel

Conversation

@HuggeK

@HuggeK HuggeK commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Derives PV array tilt, azimuth and kWp from Lantmäteriet open geodata, so a
Swedish site can stop typing panel angles in by hand.

Stacked on #734 (which adds the coverage registry this registers into), which
is itself stacked on #718.

Shape

roofmodel/ is a separate Python package alongside optimizer/, reached at
arm's length: core spawns it, passes coordinates and the operator's own
Geotorget credentials, and reads back one versioned roof_model.json.

That boundary is the point. LiDAR segmentation drags in a compiled point-cloud
stack, runs for minutes, and is onboarding-time work rather than runtime work.
In a time-boxed subprocess it cannot stall the control tick, cannot leak into the
daemon, and can be absent entirely — which is the normal case, since the data
exists only for Sweden.

Method

Follows the SPAN paper
(Yavuzdoğan, Renewable Energy 2023): iterative RANSAC pulls one roof surface at
a time out of the cloud, then DBSCAN splits faces that satisfy the same plane
equation but sit in different places — two wings of a building fit one plane and
are not one roof.

The method only. No code is taken from SPAN's GPL QGIS plugin; dependencies are
numpy, scikit-learn and requests, all BSD, so the module carries no copyleft
obligation.

Two implementation choices worth flagging

Plane fitting is total-least-squares, not z = ax + by + c. An ordinary
regression of height on position minimises vertical error, which
systematically biases steep roofs. The smallest singular vector of the
mean-centred points minimises perpendicular distance instead, and stays
well-conditioned as the surface approaches vertical.

SWEREF 99 TM is implemented directly rather than via pyproj. Lantmäteriet
publishes in EPSG:3006 and FTW stores WGS84, so every request crosses that
boundary — but it is exactly one projection with fixed parameters, and shipping
a full PROJ build for that is disproportionate on a Pi. The Krüger series is
~120 lines and verified two ways: against the projection's exact analytic
properties (easting on the central meridian is exactly 500000 at every
latitude, and the origin maps to (0, 500000)), and by sub-millimetre round trips
from Smygehuk to Treriksröset.

What it produces

input: 1392 LiDAR returns   (synthetic gable + detached garage)

  tilt  azimuth   area m2  points   truth
  35.0    180.0      87.9     576   35 deg south
  35.0      0.0      87.9     576   35 deg north
  18.0    270.4      23.8     181   18 deg west

-- as candidate PV arrays (north face dropped, too little yield) --
   Roof south       12.31 kWp   tilt 35.0  az 180.0  (88 m2)
   Roof west         3.33 kWp   tilt 18.0  az 270.4  (24 m2)

GET /api/roofmodel reports availability and coverage; POST /api/roofmodel/derive runs a derive and returns the proposal. Lantmäteriet also
joins /api/data-sources as a Sweden-only, credential-gated source.

Deliberately not wired: auto-applying to config

The RFC sketched automatic auto-fill of weather.pv_arrays. This PR returns the
proposal instead and leaves applying it as a separate explicit act. Derivation is
a best guess from a point cloud that may be years old, packing factor is an
assumption, and silently rewriting an operator's panel config is a change they
should make knowingly. Happy to wire the auto-apply if you'd rather — it is a
small change on top, but it is a behaviour decision, not an implementation
detail.

Testing

57 Python tests. Every geometry case is a synthetic roof whose tilt, azimuth
and area are known exactly by construction, and the generator is the inverse of
the code under test rather than a copy of it — so the assertions check recovered
geometry against ground truth, not against a previous run. Covered: six
orientations, gable splitting, two buildings sharing a plane, sloped-vs-footprint
area, realistic LiDAR noise, wall rejection, determinism.

11 Go tests. The subprocess contract is exercised end-to-end against a stub
module, so argument passing, stdout parsing, stderr error surfacing, schema
rejection and the timeout are all real rather than mocked.

A limitation the tests pin rather than hide

Sweden is a long diagonal, so no lat/lon rectangle traces its border — any box
containing Sweden also contains parts of Norway. Oslo is the clearest case: west
of Sweden, inside the box.

Rather than tighten the box until it starts excluding real Swedish addresses near
the border, the box stays generous and the STAC search is the authority: no tiles
come back, and the module reports "Sweden only". There are tests for both halves —
that Oslo is admitted by design, and that Strömstad, Haparanda, Karesuando and
Smygehuk are never excluded.

Not covered: the live Lantmäteriet fetch. It needs Geotorget credentials for
Byggnad-vektor and Laserdata-skog, which are the operator's own; the HTTP layer
is tested against a fake session instead, including the documented quirk that
properties.datetime is still being backfilled through 2026 and the laser strip
datum is the fallback. To try it live:

pip install -e roofmodel[geo]
python -m ftw_roofmodel --lat 59.33 --lon 18.07 --username <user> --token <token>

Part of the roadmap in discussion #717.

claude and others added 3 commits July 31, 2026 10:20
Radiation-bearing providers previously produced one flat estimate,
rated x (W/m2 / 1000), which ignored panel orientation entirely: a
south-facing 35 deg roof and a flat one got the same forecast.

When per-plane geometry is configured (Weather tab: tilt/azimuth/kWp),
global horizontal irradiance is now projected onto each plane via the
existing sunpos physics and summed. Providers publishing only GHI get an
Erbs correlation to split direct from diffuse first. Sites with no arrays
keep the previous behaviour, and Forecast.Solar - which already returns
site-calibrated watts - is deliberately left untouched so its numbers are
not scaled twice.

Split out of the original srcfl#718 per review: this half is user-visible and
improves every radiation-bearing provider today. The STRANG client and
performance scoring that shared that branch move to a follow-up PR.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
…coverage

STRANG becomes a real irradiance source rather than an unreferenced client,
every external data source now says where in the world it works, and the
location picker moves off Leaflet.

STRANG's parameter set was mapped against the live API because SMHI's apidocs
pages 404: exactly 116-122 exist. Identification was confirmed by physics
rather than by guessing - at solar noon 121 + 122 = 723.0 + 87.5 = 810.5,
exactly parameter 117, and 119 caps at 60, i.e. minutes within the hour.

STRANG publishes no cloud cover; it is a radiation model. Cloudiness is
instead derived from sunshine duration as 1 - minutes/60, which is observed
rather than inferred but coarser: blind to thin cirrus, undefined at night.
CloudCover() therefore returns an explicit unknown instead of defaulting to
clear, because those two lead to opposite decisions.

The new coverage registry makes an existing silence explicit. STRANG is
Nordic-only and every price provider is European, so sites elsewhere were
getting empty results with no explanation (srcfl#726). GET /api/data-sources now
reports area, countries, licence and whether each source reaches this site,
and the Weather tab renders it under the map. Bounds are advisory: STRANG's
grid is rotated, so a lat/lon box can only ever be a superset - all four
in-box corners were probed and returned no data. False is definitive, true
means worth trying. Scoring now declines to start outside the domain instead
of retrying nightly forever.

Stacked on srcfl#718, which carries the plane-of-array wiring this builds on.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
A Swedish site can stop typing panel angles in by hand. The new roofmodel/
module reads Lantmateriet LiDAR around the site and recovers each roof face's
tilt, azimuth and usable area, which pre-fill weather.pv_arrays.

Structured as a separate Python package alongside optimizer/, reached at arm's
length: core spawns it, passes coordinates and the operator's own Geotorget
credentials, and reads one versioned roof_model.json from stdout. Segmentation
drags in a compiled point-cloud stack and runs for minutes, so a time-boxed
subprocess keeps it off the control tick and lets it be absent entirely - the
normal case, since the data exists only for Sweden.

The pipeline follows the SPAN method: iterative RANSAC pulls one surface at a
time out of the cloud, then DBSCAN splits faces sharing a plane equation but not
a location, because two wings of a building fit the same plane and are not the
same roof. Method only - no code is taken from SPAN's GPL QGIS plugin, and the
dependencies (numpy, scikit-learn, requests) are all BSD.

SWEREF 99 TM is implemented directly rather than via pyproj: it is one
projection with fixed parameters, and a full PROJ build is disproportionate for
that. Verified against the projection's exact analytic properties - easting on
the central meridian is exactly 500000 at every latitude - and by sub-millimetre
round trips from Smygehuk to Treriksroeset.

Applying derived arrays to config is deliberately a separate act. Derivation is
a best guess from a point cloud that may be years old, and silently rewriting an
operator's panel config is a change they should make knowingly.

Also generalises the __pycache__ ignore rule, which was optimizer-specific and
would otherwise need repeating per module.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
claude and others added 2 commits July 31, 2026 14:59
The roofmodel tests passed on Windows and failed on Linux, which is where CI
runs them. Two separate bugs, both caused by stubbing the module with a shell
script:

  - The stub handed back "sh" as the command, and the service invokes it as
    `<cmd> -m ftw_roofmodel ...`. dash reads that as "run the script named
    ftw_roofmodel", cannot open it, and exits 2. The stub never ran at all, so
    five tests failed against an error that had nothing to do with what they
    were asserting.
  - The module-error stub echoed an unquoted JSON document. dash strips the
    double quotes, so `{"error":"..."}` reached the parser as `{error:...}` and
    could not be unmarshalled. cmd.exe keeps them, which is the only reason
    that test ever passed.

The stub is now this test binary re-executed with an environment variable
telling it which behaviour to play. Nothing goes through a shell, so there is
no quoting to get wrong and no per-platform behaviour to diverge. The timeout
test consequently runs everywhere instead of being skipped on Windows.

Two assertions the shell stub could not make: that the site, credentials and
radius actually survive the process boundary, and that PYTHONPATH is set --
without it the module is only importable if installed system-wide, which on a
Pi it is not. Also that --vostok stays absent unless configured, so a GPL tool
is never invoked by default.

The Python module's tests did not run in CI at all: the workflow runs
`pytest -q optimizer/tests` and nothing else, so 75 tests covering the plane
fitting every derived tilt and azimuth depends on were never executed upstream.
Added a roofmodel job mirroring the optimizer's, wired into the required-check
gate so a failure blocks rather than being reported and ignored. It installs
without the `geo` extra on purpose: LAZ decoding pulls a compiled backend, and
everything except the point-cloud read is exercised without it.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
Settings -> Weather gains a Geotorget credential form and a building picker.
Press "Find buildings here" and the footprints near the marker are drawn on the
map and listed beside it; click yours, press "Read roof from LiDAR", and the PV
arrays fill in with one entry per usable roof face.

The form is filled but nothing is saved. The operator sees the numbers,
corrects what is wrong and presses Save. FTW does not rewrite a panel
configuration on its own: the derivation is a guess from a scan that may be
years old, and only the operator knows whether that face has panels on it.

Picking a building is not cosmetic. Without a footprint the module segments
whatever stands inside its search radius, and the plane fitting is global -- a
fitted plane is infinite, so a roof at azimuth 180 is z = f(y) with no x term
and extends across the whole tile. A second building sharing that ridge
orientation lands inside its inlier band however far away it is, and the two
lose returns to each other. Measured on a synthetic pair: a detached garage
recovered 93% of its true area and split into two fragments while coplanar with
the house, against 100% and one clean face once clipped to its own footprint.
The clip buffers the outline by a metre first, because roofs overhang their
walls and the eaves carry the lowest returns.

Frames are detected rather than assumed. GeoJSON mandates WGS84 but
Lantmateriet publishes this catalogue in SWEREF 99 TM and its STAC search takes
a SWEREF bbox, so ring coordinates are classified by magnitude -- six- and
seven-figure numbers are projected metres, degrees never are.

New GET /api/roofmodel/buildings lists footprints as GeoJSON, honouring an
explicit lat/lon so the picker can search where the marker is rather than where
the last save put it. POST /api/roofmodel/derive accepts a building_id. GET
/api/roofmodel reports has_credentials so the UI can stop asking.

The Geotorget token now masks and restores like every other secret. It never
appears in an API response, and saving an unrelated setting no longer wipes it
-- the settings form returns the blank it was given, which without
PreserveMaskedSecrets would have deleted the stored credential.

docs/roof-geometry.md covers ordering the two Geotorget products, what the
derived kWp does and does not mean (an upper bound on what fits, not what is
installed), and what each failure message is telling you.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
Both roof-geometry products are STAC APIs behind one Geotorget account, and
they differ only in what their items point at: Byggnad Nedladdning, vektor
delivers GeoPackage, Laserdata Nedladdning, Skog delivers LAZ organised as
COPC. Assets are now selected by declared media type rather than by guessing
at asset key names, so a catalogue that calls its asset "punktmoln" instead of
"data" keeps working, and a thumbnail is never handed back as a point cloud.

Building footprints are read straight out of the GeoPackage with the standard
library. A GeoPackage is a SQLite database holding geometry as WKB, both
published formats with fixed layouts, so this costs about a hundred lines
against a GDAL dependency that will not install on a Pi without a compiler.
Previously only inline STAC geometry was handled and the asset-backed case --
the normal one -- returned nothing at all.

Because COPC indexes points into an octree, picking a building now also makes
the download small: only the octree nodes covering that footprint are
range-requested, instead of a 2.5 km tile running to hundreds of megabytes.
Plain .laz assets, hosts that ignore Range, and builds of laspy without COPC
support all fall back to reading the tile whole -- slower, same answer -- and
the model records which path ran as source.fetch, since that also decides what
returns_in_radius is counted over.

Also corrects the coplanar-absorption measurement quoted in buildings.py and
docs/roof-geometry.md. A fresh run does not reproduce the 93%/two-fragment
figures: one pass over a house and a garage 40 m apart consumes all 576 of the
house's south-face returns and all 256 of the garage's as one surface, and
gives identical output at every separation from 3 m to 40 m -- which is the
signature of a global infinite-plane fit rather than a proximity effect.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants