Skip to content
Yggdrasilby AImageLab

Bulk upload from the command line

The ygg command uploads a folder of images to a project. It is for datasets too large to pick in a browser: thousands of files, many patients, uploads that run for hours. It runs the browser’s own de-identification and encryption code, on your machine. The server receives what a browser upload sends: encrypted files and encrypted metadata, never your files or your patients’ real IDs.

It reads the formats the project’s modalities take (DICOM, NIfTI, NRRD, pictures, videos, slides, meshes and signals; see Supported formats) and converts nothing.

Get the command

Section titled “Get the command”

ygg is one file with no dependencies. Until releases ship it, build it from the source (with Deno 2.9 and pnpm):

Terminal window
pnpm install
cd apps/cli
deno task compile        # writes dist/ygg

Sign in

Section titled “Sign in”
Terminal window
ygg login --server https://ygg.example.org --email you@example.org

You get an email with a sign-in code; type it. The first time, the command needs approval from a device where your keys are unlocked, as a new browser does:

  1. ygg prints a link and a code.
  2. On your trusted device, open the link (or open Security and type the code), then Continue.
  3. Both show a 6-digit code. If they match, choose The codes match on the trusted device.

This machine is then one of your trusted devices: it keeps its own device key in ~/.config/yggdrasil/credentials.json, readable by you only (YGG_CONFIG_DIR sets another folder). The Security page lists it with your other devices. Later sign-ins need only the email code.

ygg logout ends the session; ygg logout --forget also deletes this machine’s device key. Remove its slot on the Security page too.

Upload a folder

Section titled “Upload a folder”
Terminal window
ygg projects                                  # your projects; "upload" where you may upload
ygg upload ./dataset --project my-study --mapping mapping.csv
  • Patients. DICOM files are grouped by their Patient ID, study and series. Other files name no patient: the folder that holds a file stands for its patient and study (dataset/BraTS_001/BraTS_001_t1.nii.gz).
  • Pseudonyms come from your mapping file, the same CSV the upload page reads and saves (real_id,pseudonym). It stays on your machine; it is never sent. A patient the mapping file lacks stops the upload. With --assign <prefix>, such patients get new pseudonyms (<prefix>0001, <prefix>0002, …), which are written into the mapping file before anything is sent. (The upload page instead pre-fills such patients’ pseudonyms from the project’s pattern; ygg does not read it.)
  • Site. You upload to your own site; admins name one with --site <name> when the project has several.
  • Timepoints. Each new study gets the patient’s next timepoint in the manifest’s order.
  • Flagged series. Some series may show identifying text in their pixels (for example DICOM’s Burned In Annotation, ultrasound, photos). The upload stops and lists them. Check them, then upload with --include-flagged, or leave them out with --skip-flagged. Masking pixels needs the upload page.
  • --dry-run scans the folder and prints the plan; nothing is sent.

Clinical forms are not filled by the command: new patients get an empty form, which you complete in the web app.

Interrupted uploads

Section titled “Interrupted uploads”

Stop an upload with Ctrl+C, or lose the connection: run the same command again and it goes on where it stopped. Files already stored are not sent again; the rest are de-identified and encrypted again. What the upload has written is kept, encrypted under your keys, in ~/.config/yggdrasil/uploads/, also after it finished: running the command again on the same folder sends only files it has not stored, such as files added since, and writes again what someone deleted from the project in the meantime.

--restart forgets what the folder’s earlier uploads stored and starts over. DICOM instances the patient’s series already hold are still not sent again: they are found by their de-identified UIDs, also when the folder was uploaded from another computer or from the web app. Other formats carry nothing to match on, so without the kept state they make new series.

What it checks

Section titled “What it checks”
  • The project’s signed log is replayed and checked like a browser’s: the first contact pins it, and later runs refuse a log that no longer holds what this machine verified.
  • The manifest must match the project’s signed log.
  • Uploads go only to projects where you may upload, under the project’s current key.

What the command does not do yet: run processors on upload (start them from the Processors page) and make thumbnails of pictures (DICOM series get theirs). The command’s source is in apps/cli.

Yggdrasilby AImageLab