Exporting data

Project admins export a chosen part of their project’s data as an export: a job that runs on its own, packages the data as a plain zip and keeps it behind a link until it expires. Only admins see the Export page; annotators and viewers cannot export.

The data is encrypted in members’ browsers, and the server cannot read it. An export is the one place where it leaves that encryption: an export worker, a program the project trusts, decrypts exactly what the admin chose and writes the archive.

1. Choose an export worker

A project registers the export worker it trusts on its Export page, once. There are two kinds, and the choice decides who can see the data:

• This server’s export worker is run by the operator of this server and is on by default. The server’s operator can see the data while it is packaged, and the archive is kept on this server, unencrypted, until it expires. Registering it takes one click.
• A worker your institution runs, on its own machine: only your institution can see the data while it is packaged, and the archive is kept on the worker’s own storage (or, if your institution registers a descriptor that allows it, on this server). Setup: install the processor SDK, make keys and register them (the SDK README, “Export worker”).

Registering this server’s worker shows what it means first: the operator of this server can see the exported data while it is packaged, and the archive is readable on the server (by anyone with the link, the operator and anyone with access to its storage) until it is deleted. The project’s audit log records export.operator-visible for each export it packages. An institution-run worker keeps the host out: the host never holds its keys. If it also writes the archive to its own storage (the worker destination, the only one the worker the SDK ships offers), the export never exists in the clear on this server at all. Your browser signs each export’s destination and choices, and the worker refuses an export no admin signed, so the server cannot send it elsewhere.

Registering signs a record in your browser, as for processors: what is signed (the worker’s keys and the SHA-256 of its descriptor) is shown first. When a worker is upgraded to a version with other formats, its descriptor changes and its registration stops working: the Export page says so, and an admin registers it again.

2. Select what to export

An export takes a chosen subset, never necessarily everything:

• Data: patients, studies and series, by the filters (site, modality, completion, a clinical form field) and by choosing rows in the tree. A chosen series takes all its images.
• Annotations: the tasks to include and, for tasks with labels, which labels. Annotations of labels left out are not exported; in a segmentation, voxels of labels left out are set to 0.
• Statuses (optional): every status by default, or for example submitted only.
• Process log (optional, off by default): how the readers annotated (items, active time, navigation, tools, strokes, edits, AI tools), from the project’s process log.

The page counts what the export takes as you choose, and estimates its size. The archive counts against the project’s storage quota until it is deleted.

Or export a snapshot

A snapshot freezes a release of the dataset (“v1.0”) so that it can be exported again, identically, later. Make one on the project’s Snapshots page: give it a name and notes (both encrypted), choose whether to apply the consensus and the corrections (where admins recorded a consensus, it replaces the readers’ answers it combines; else the latest correction replaces those it corrects). A snapshot takes submitted annotations. The snapshot lists every patient, study, series and file of the project and each annotation at its current revision, and your browser signs that list. Later edits do not change it: a revised answer stays at the revision the snapshot froze.

To export it, choose Export on its row (or the snapshot under What to export on the Export page): the archive holds the snapshot’s objects and its annotations as they were, and names the snapshot in manifest.json and the dataset card. The page checks the snapshot’s signature against the project’s signed log before it shows, compares or exports it; a snapshot signed by an admin who has since left or changed keys no longer verifies, and an admin makes a new one. Deleting data still removes it, from snapshots too.

The Snapshots page also compares two snapshots: what the later one added, removed or holds at another revision or status, per kind of object and per task.

3. Options

• Formats. Every export holds the native layout (below). Add tables (CSV, JSON Lines, Parquet), NIfTI labelmaps and standard formats (DICOM SEG and SR, nnU-Net, MONAI, BIDS, COCO, Hugging Face) as you need them.
• Destination: this server’s storage behind a link, or the export worker’s own storage (institution-run workers only).
• Keep the archive for 1 to 30 days (7 by default), counted from when it is ready. The date is fixed; it is never extended.
• Protect the link with a code (optional): downloads then need the link and a code made in your browser. Send the code by another channel than the link.
• Folder names: IDs, or the patients’ pseudonym codes and the studies’ and series’ labels.
• Dataset card: a title, summary, notes and licence; tick Include the project’s protocol and the labels’ guidelines to add them to the card.

Everything you type is kept on this device as you type it; nothing needs saving.

4. Read the disclosure, then confirm

Before Confirm, the page says in plain words who decrypts this export and where the archive is stored until when. Confirm names what is exported. Your browser then grants the worker the keys of exactly those objects and seals the options; the server checks the list but never decrypts.

Once the export has started you may close the page or sign out: it goes on without the browser. The exports list shows its progress, and you get an email when it is ready.

5. What the archive holds

• manifest.json: every file of every part, with its part, size, SHA-256 and role.

• dataset-card.md and dataset-card.json: the dataset card (below).

• annotation_log.jsonl and annotation_log.schema.json, when you included the process log: one event per line, with its session and reader (user ID), for the exported items and events of no item.

• project/: the project manifest versions the annotations were made under.

• data/: a folder per patient, study and series with the de-identified originals and the metadata of each (surface meshes keep their .stl, .ply or .obj; recordings their .edf, .bdf or .dcm, and a WFDB record its record.hea and record.dat, so WFDB tools open it as record in its folder).

• annotations/: each annotation as a W3C Web Annotation-shaped JSON file, by task, naming the member who signed it (signedBy, checked by the export worker; answers whose author’s signature does not verify are left out), and under annotations/assets/ the files they reference, such as segmentation masks, attached files and the .wav recordings of dictated answers on tasks that keep voices.

• tables/ (with a table format): tables/tasks/<task>.csv, a row per annotation with its target, reader, status, revision and answer (each option of a multiple choice as its own column, shapes in volume coordinates; on a surface mesh a landmark’s coordinates and a region’s vertex indices), and tables/clinical-form.csv, a row per patient with the clinical form’s answers.

• labelmaps/ (with NIfTI labelmaps): each segmentation as a NIfTI file on its series’ geometry with only the selected labels, and a .json sidecar naming each value’s label.

• geojson/ (projects with whole-slide images, in every export): each reader’s shapes of a task on a slide as a GeoJSON file in the slide’s full-resolution pixels, which QuPath and other pathology tools import; on a stack of planes each shape names its plane.

• Standard formats, each in a folder of its own with a yggdrasil-report.json listing what it could not hold and why:

  • dicom-seg/ and dicom-seg-labelmap/: each segmentation of a DICOM series or DICOM image as a DICOM Segmentation (one segment per label, or a Label Map Segmentation), referencing the de-identified instances, for PACS and DICOM viewers.
  • dicom-sr/: per DICOM series and reader, a structured report (TID 1500) with the shapes, their measurements and the classification and number answers.
  • nnunet/, monai/, bids/: training datasets of volume segmentations, each image converted to NIfTI on the grid its masks lie on: nnU-Net v2 (one label per image: the consensus, else the submitted one), a MONAI datalist (every reader’s label) and BIDS (MR series whose contrast the header tells, the masks as a derivative).
  • coco/: per task, the shapes and masks on 2D images as COCO JSON, naming the originals in data/.
  • huggingface/: a dataset card and one configuration per task, which datasets.load_dataset reads.

Tables are written in each table format you chose (.csv, .jsonl, .parquet), with the same columns. The native export specification defines every file.

The dataset card states the counts per level, modality and site; the de-identification profiles with their tool versions; the annotation protocol (tasks, labels and options with their descriptions, which tasks have a correction stage, the statuses exported); the licence; what the selection left out; how to cite the dataset; and the attribution “Yggdrasil by AImageLab”.

6. Share the link

Copy the link from the exports list (any admin can, at any time). Anyone with the link downloads without signing in; with a protected link, they also need the code. Treat the link as you would the data: whoever holds it can read the archive until it expires.

• Replace link makes a new link (and code); the old one stops at once.
• Revoke stops the link and deletes the archive. It cannot be undone.
• At the expiry date the link stops and the archive is deleted.

The server records each download’s time and coarse network (for example an IPv4 /24), never the full address; refused codes are recorded once per network and hour, never the code typed.

7. Download

A link opened in a browser shows the link page: the parts, their sizes and SHA-256s, the expiry and the commands to copy. A large export is split into numbered parts, each a complete zip; manifest.json in part 1 says which part holds each file.

From the command line, curl downloads any link and resumes an interrupted download (-C -):

# One part
curl -L -C - -o export.zip 'https://<app>/x/<token>'

# Any number of parts, each under its own name
curl -L --remote-name-all $(curl -fsS 'https://<app>/x/<token>?format=txt')

# A protected link: send the code with every request
curl -H 'X-Ygg-Link-Code: <code>' -L -C - -o export.zip 'https://<app>/x/<token>'

8. Check what you downloaded

Compare each part’s SHA-256 with the one the link page lists:

sha256sum export-*-part-*.zip    # Linux
shasum -a 256 export-*.zip       # macOS

Then check the files inside against manifest.json, which lists every file’s SHA-256:

python3 - <<'EOF'
import hashlib, json, zipfile, glob
parts = {int(p.rsplit("-", 1)[1][:4]): zipfile.ZipFile(p) for p in glob.glob("export-*-part-*.zip")}
manifest = json.loads(parts[1].read("manifest.json"))
bad = [f["path"] for f in manifest["files"]
       if hashlib.sha256(parts[f["part"]].read(f["path"])).hexdigest() != f["sha256"]]
print("all files match" if not bad else f"mismatch: {bad}")
EOF

A single-part export saved as export.zip is part 1: open it as parts = {1: zipfile.ZipFile("export.zip")}.