Writing a modality plugin

A modality plugin teaches the platform a kind of data: which files it takes, how the uploader’s browser de-identifies and groups them before encrypting them, and how members view them. It is a package of the plugin catalog like a task plugin: read Plugins and the plugin catalog and Writing a task plugin first, which cover the package, the host, the build and the console.

export default definePlugin((h) => ({
  modalities: [{ descriptor, deid, accept, ingest, viewers }],
}));

The descriptor

A modality plugin is described by a modality descriptor (schema modality/descriptor-v1): its ID and version (manifests name it id@major), its name and catalog category, the selector spaces its viewers use, its formats, its de-identification profiles and options, its layouts and hanging protocols, and the annotation types its viewers can show (annotationTypes, built-in or its own). The wizard’s Modalities step offers exactly what it lists; the app checks the descriptor against the schema when it loads the plugin. Supported formats lists the built-in plugins’ formats, DICOM Modality values and layouts.

deid gives every profile and option the descriptor lists a name and a description in English, and each option’s default. The built-in profiles are ps3.15-basic (DICOM’s Basic Application Level Confidentiality Profile) and strip-metadata; a plugin that takes DICOM uses the first.

Taking files: accept rules and ingest

accept lists the file signatures (first bytes) and names the plugin takes; the upload page routes each file to the first plugin of the manifest that accepts it.

ingest() starts a worker that serves an IngestPipeline (@yggdrasil/plugin-api documents each step): scan groups files into series, review lists what de-identification changes, describe gives the de-identified study and series metadata and the receipt, and prepare gives each asset’s de-identified bytes and metadata, which the app then encrypts and uploads. Every step runs in the uploader’s browser: nothing leaves it before it is de-identified and encrypted. Steps must be deterministic for the same files.

Never convert at ingest. The platform stores the files as they came, with only what identifies the patient removed; viewers read the stored format.

A catalog plugin is one module, so its ingest worker is the same module started again:

import { serveIngest } from "@yggdrasil/plugin-api/ingest";

const pipeline = { scan, review, describe, identify, prepare };
// In the worker: serve the pipeline. In the page: register the plugin.
if (!("document" in globalThis)) serveIngest(pipeline);

const ingest = () => new Worker(import.meta.url, { type: "module" });

The worker has no host: it can use only what the plugin bundles (@yggdrasil/plugin-api/ingest imports nothing). Browsers check no integrity for a worker’s script; the worker runs under the app’s Content Security Policy, which allows WebAssembly but no eval.

Viewing: viewers by layout

viewers maps each of the descriptor’s layouts to a React component. It gets the series’ assets with their decrypted metadata and a read(assetId, start?, end?) function that returns decrypted bytes, and renders the toolbar and side panel it needs through the viewer shell’s slots. Plugins never reach the API or the keys: the host gives them only what the member opened.

The wizard’s preview: sample series

The wizard’s workspace preview shows a draft’s entries on sample images, never on project data. A plugin gives its own with previewSamples(sequences) (plugin API 1.1.0): sample series (one per sequence, if the plugin shows sequences apart), each a list of assets with the metadata its ingest would write and a function making the bytes, synthetic and made in the page. Without it, the preview says it has no sample for the plugin.

Annotations on the data

The descriptor’s coordinateSpaces say where shapes lie (pixel, patient, media-time, slide, mesh, …), and annotationTypes which types its viewers can show. A viewer that draws shapes uses the selectors of annotation/selectors-v1.

Limits

• The server knows only the built-in modality plugins: it checks the IDs a manifest entry picks (formats, profile, layout) for ygg.* plugins only, and leaves a catalog plugin’s to its own checks in members’ apps.
• Processors and the export worker read the stored files as they are; a new format reaches them as its original bytes.
• Thumbnails are built in for the built-in modalities only.