04 / Data

From counts to curves

A detector doesn’t measure science. It measures events — “detector 7041 fired at 12,330 µs”. Turning billions of those hits into a curve someone can publish is called data reduction, and it’s an ETL pipeline you can run with your own hands below.

4.1 — The clock is the instrument

Time of flight: slow neutrons arrive late

ISIS is pulsed: every neutron in a pulse leaves the moderator (the block that slows neutrons to useful speeds — see Facility) at (nearly) the same moment, like runners off a starting gun. Quantum mechanics gives every moving particle a wavelength — the slower it moves, the longer (the Probes page shows why). Fast, short-wavelength neutrons reach the detector first; slow, long-wavelength ones straggle in later.

Why is there a spread of speeds at all? The moderator. Each neutron escapes it after a different random number of bounces — some slip out after a couple of collisions still moving fast, others rattle around until they’re as slow as the molecules they’re bouncing off. So a single pulse emerges as a mixed bag of speeds: a whole spectrum of wavelengths, all launched together.

And what do we get out of timing them? A plain stopwatch tells you each neutron’s wavelength — no extra hardware: λ ≈ 3956 × t / L, arrival time t (seconds) over flight path L (metres) gives wavelength in ångströms (1 Å = 0.1 nanometres — about the size of an atom). The wavelength is the “ruler marking” every later analysis step depends on — the playground’s ConvertUnits step below is exactly this equation applied millions of times. And because timing sorts the mixture instead of filtering it, every neutron in the pulse contributes to the measurement rather than being thrown away. In software terms: the timestamp is the payload.

Drag the slider (or press play) and watch five neutrons of different wavelengths race down an 11 m flight path into the arrival histogram. The 3956 is just Planck’s constant over the neutron’s mass, pre-baked into convenient units — a constant of nature, not a calibration.

moderator 0 m 5.5 m 11 m detector
0.0 ms
FIG 4.1 — five neutrons, λ = 1–8 Å, over L = 11 m. arrival time t = L·λ / 3956.
4.2 — Two shapes of raw data

Events vs histograms

A detector really does catch one neutron at a time. What reaches the file is another matter: on most ISIS instruments the electronics aggregate as they go, so each hit simply adds 1 to the bucket for that detector and that slice of time. Some instruments can instead keep every hit separately — that’s event mode.

In event mode the data file is literally a list: (detector ID, time of flight), one line per detected neutron, millions to billions of lines per run. Nothing is summarized yet — like raw, unaggregated logs.

A histogram is that same data with time chopped into bins and the hits counted per bin — either done for you in hardware as the run proceeds, or done afterwards in software from an event list. Wide bins smooth away detail; narrow bins go noisy because each bin holds only a few counts. Where the events were kept, they stay the ground truth — you can re-bin them any way you like, any number of times.

Why keep the log instead of just the summary? Three wins. ① Choose resolution later — re-bin the same run coarsely for a quick look during the experiment, finely for the final fit, with no re-measuring. ② Filter by time — every event is timestamped, so you can keep only the events recorded while the sample was actually at temperature, or slice one long run into chunks and watch a battery charge or a protein unfold as a sequence of frames. ③ Nothing is ever lost — a histogram can always be rebuilt from the events, but never the other way round; an aggregation mistake costs a re-run of software, not a re-run of beam time.

In Mantid these two shapes are literally two workspace types: EventWorkspace (the log) and Workspace2D (the aggregated histogram) — and the time-slicing trick is the FilterEvents family of algorithms. The two shapes even arrive as two different file types: ISIS histogram NeXus files load with LoadISISNexus, event files with LoadEventNexus.

— µs 50–2000 µs, log scale
FIG 4.2 — 4,000 synthetic events from one detector bank (a group of detectors mounted together at one angle). toggle to histogram, then play with the bin width.
4.3 — Reduction is an ETL pipeline

What raw data is missing

The raw event stream is honest but useless for science as-is:

① It’s in instrument coordinates (microseconds at one particular detector), not physics coordinates (wavelength, energy, d-spacing — the distance between repeating layers of atoms in a crystal). ② The shape of the source spectrum is baked into every measurement — more neutrons arrive at some wavelengths than others. ③ Detectors have quirks: dead pixels, different efficiencies, different positions.

Reduction fixes all three, in a standard order, the same way for every run on an instrument. This is what Mantid does. Its three core ideas map cleanly to software concepts:

Workspace — the data structure (data + instrument geometry + metadata)

Algorithm — a transform: workspace in, workspace out

History — an audit log: every workspace records the exact chain of algorithms (with parameters) that produced it. Full provenance, replayable as a script.

events.nxs raw log Load ConvertUnits + Rebin + Normalize Fit result numbers + errors every step appended to the workspace HISTORY → reproducible
FIG 4.3 — reduction in one line: extract (Load), transform (ConvertUnits/Rebin/Normalize), load (a result you can fit). .nxs = NeXus, the standard facility file format.
4.4 — Hands on

The Reduction Playground

A simulated powder-diffraction run on a toy instrument: a powdered crystal (many tiny grains of the same material), three detector banks at different angles (2θ — just the angle the neutron is deflected through), 11 m flight path. The crystal’s atoms sit in evenly spaced layers; neutrons whose wavelength matches a layer spacing reflect strongly, producing a sharp peak at that spacing — that’s diffraction. Run the five steps in order and watch what each one earns you.

STEP 1

Load

Read the run file into a workspace. Three banks, same crystal — but the peaks don’t line up, because each bank has its own geometry and its own clock-to-wavelength conversion.

STEP 2

ConvertUnits

Convert microseconds → d-spacing using each bank’s known geometry (flight path + angle). Instrument coordinates become physics coordinates — watch the peaks snap into alignment.

STEP 3

Rebin

Put all three banks onto one common set of bins, then sum them into a single pattern — in real Mantid that’s two algorithms, Rebin then DiffractionFocussing. Pick the bin width — too wide blurs the peaks, too narrow goes noisy.

0.010 Å
STEP 4

Normalize

Divide out the source spectrum (measured by a monitor — a small detector sitting directly in the beam). The sloping background flattens and peak heights finally mean something physical.

STEP 5

Fit

Click near a peak in the plot — or focus the plot and press Enter — to fit a Gaussian (a bell-shaped curve) + background. The fitted centre is a spacing between planes of atoms in the crystal, in ångströms. This number goes in the paper.

apply steps 1–4 first
FIG 4.4 — workspace view. apply STEP 1 to load the run.

workspace history

    Simplified — real Mantid reduction adds calibration, masking, absorption corrections and per-technique steps. The real algorithms: Load, ConvertUnits, Rebin, DiffractionFocussing, NormaliseToMonitor, Fit.

    4.5 — Where reduction stops

    Reduction vs analysis

    Reduction

    Facility-side, largely standard per instrument: same steps for every run, increasingly automated. Output: a clean, calibrated curve in physical units with error bars. If reduction is wrong, every user’s science is wrong — which is why it lives in maintained, tested software (Mantid) rather than per-user scripts.

    Analysis

    Science-side and different for every experiment: fit a structural model, compare with simulation, refine parameters. Tools range from Mantid’s fitting interface to technique-specific codes. Output: the numbers and conclusions that go in the paper.

    4.6 — Checkpoint

    Quick check