Grid Data Archive

The GDA data lake — read-only.

Direct access to the parquet data lake behind the Twin Scroll Grid Balancer evidence base — nearly 500 billion data points across 701 datasets: ~350 billion of official GB grid feeds, ~150 billion derived and inferred, plus a full power-flow solve of the grid for every second of the last 15+ years. You mount it as a folder and read the parquet. Absolutely read-only: you can browse and download, nothing more.

The lake is served over SFTP from a locked-down account. Download the key, point sshfs or sftp at it, and you have the whole parquet tree as a normal read-only filesystem — query it with DuckDB, pandas, polars, or anything that reads parquet.

~500bn

data points

701

datasets

200+ GB

parquet

~945k

parquet files

15+ yr

solved every second

1 Get the key

Access is by SSH key only. Download it, then keep it somewhere sensible.

mountservice_ed25519

Ed25519 private key · read-only mountservice account

Key fingerprint: SHA256:eKvQNEXJs0EVqhtPAVia6o7GSmHFo5wTtfpqxk2Gx78

Download key

On macOS / Linux, lock the permissions before using it:

chmod 600 ~/Downloads/mountservice_ed25519

2 Connection details

Host	ingest.twinscrollgridbalancer.co.uk
Port	2222
User	mountservice
Protocol	SFTP (SSH)
Path	/lake
Auth	the key you just downloaded

3 Mount the lake as a folder

The easiest way — sshfs mounts the whole lake as a local read-only directory, so every tool just sees files on disk.

macOS / Linux

mkdir -p ~/gda-lake sshfs mountservice@ingest.twinscrollgridbalancer.co.uk:/lake ~/gda-lake \ -p 2222 -o ro,reconnect,IdentityFile=~/Downloads/mountservice_ed25519 # browse it like any folder ls ~/gda-lake # unmount when done umount ~/gda-lake

Needs sshfs — macOS: macFUSE + sshfs; Debian/Ubuntu: sudo apt install sshfs. On Windows use WinFsp + SSHFS-Win, or just use WinSCP (below).

Prefer to browse & pull single files? Use sftp

sftp -P 2222 -i ~/Downloads/mountservice_ed25519 mountservice@ingest.twinscrollgridbalancer.co.uk # then, at the sftp> prompt: cd /lake ls get DataSources/GridWatch/... /local/path/

On Windows, WinSCP (New Site → SFTP → host ingest.twinscrollgridbalancer.co.uk, port 2222, user mountservice, and set the private key file) gives a drag-and-drop window.

First connection shows the server host-key fingerprint — accepting it is normal.

4 What's in the lake

Roughly 500 billion data points across 701 datasets, in three top-level trees:

/lake/DataSources

~350 billion data points. Raw official source feeds — NESO / National Grid, the DNOs (UK Power Networks, SSEN…), GridWatch, Elexon, OSM network geometry, weather and more. The bulk of the lake.

/lake/InferedOrDerivedData

~150 billion data points. Datasets derived or inferred from the raw feeds — cleaned, temporally aligned, gap-filled and enriched. The evidence base most of the analysis runs on.

/lake/GDA-SOLVES

The solved grid-state library — a Newton–Raphson power-flow solution of the GB grid for every second of every day, 15+ years deep (~100% converged), reconstructed purely from the public data. Indexed by date; also downloadable a day at a time from the Solves page.

Everything is parquet, partitioned into dated folders. Only data is published here — none of the ingestion, processing or solve code.

5 Read the parquet

Once mounted, point any parquet-aware tool at the files. A few starting points:

DuckDB (fastest for ad-hoc SQL over many files)

-- glob straight over the mounted tree SELECT * FROM read_parquet('~/gda-lake/DataSources/GridWatch/**/*.parquet') LIMIT 20;

Python — pandas

import pandas as pd df = pd.read_parquet('~/gda-lake/GDA-SOLVES/_index/idx_20260621.parquet') print(df.head())

Python — polars (lazy, out-of-core)

import polars as pl lf = pl.scan_parquet('~/gda-lake/DataSources/**/*.parquet') print(lf.select(pl.len()).collect())

6 The ground rules

Read-only, enforced. The account cannot write, rename or delete anything — the mount is read-only, the SFTP server is read-only, and the account is confined to /lake. Nothing you do can affect the lake.

Look, don't copy. This data is provided for access and analysis — not for redistribution. Some raw feeds (NESO / DNO) carry their own terms. Access is metered: normal reads and random-access querying are fine, but attempting to bulk-download a copy of the corpus trips a per-source data limit and your access is automatically cut off. Don't mirror the lake.

Please be gentle — a couple of parallel connections, paced reads. Need a larger or bulk allowance for a legitimate use? mark@twinscrollgridbalancer.co.uk.