Measurement Bias: Resolution, Linearity and Drift
A calibration comparison relates the instrument’s output to a reference under the calibration conditions.

| Evidence or term | What it means here |
|---|---|
| Resolution | Whether nearby input values can be distinguished. |
| Linearity | How the measurement relationship behaves across the studied range. |
| Drift | A change associated with time or measurement sequence. |
What does the evidence explain?
Why can calibration and working conditions tell different stories?
A calibration comparison relates the instrument’s output to a reference under the calibration conditions. NIST warns that this does not ensure the same behavior in the working environment. Temperature, humidity, the operator and wear can introduce measurement bias. A disagreement between a buyer’s reading and a factory’s reading should therefore not be assigned immediately to product variation. The measurement environments can also differ.
Bias investigation asks whether there is a directional difference associated with the measuring system. The handbook lists resolution, linearity, drift, instruments, geometries and operators as areas to investigate. The relevant comparison is specific: which result, which reference or other instrument, and under which conditions? A calibration record is useful evidence, but it does not replace the investigation of the actual setup when the disputed measurement was made.
Does a display with more digits necessarily resolve smaller differences?
No. NIST defines resolution in terms of detecting and faithfully indicating small changes in the measured characteristic, and explicitly warns that the number of displayed digits does not establish instrument resolution. Two items that differ slightly may repeatedly produce the same indication even when the display appears very precise. The appearance of a long decimal value is therefore not enough to demonstrate that an instrument can distinguish the product differences relevant to the specification.
A manufacturer’s stated resolution commonly reflects the least significant digit and other instrument mechanisms. The handbook calls for checking it under actual measurement conditions. For sample comparison, distinguish the display format, the claimed instrument specification and the demonstrated ability to separate nearby values. Copying a fine increment from a device brochure into a measurement sheet does not demonstrate that the complete measuring setup can achieve it.
How does the handbook investigate whether nearby values can be distinguished?
The described laboratory approach uses several artifacts with known values, ranging from widely separated to closely spaced. Measurements begin with the pair farthest apart, then progress to pairs with smaller differences. Eventually a pair may repeatedly yield the same indicated result. The difference between that pair’s known values estimates the resolution in the described procedure. The method relies on known artifact differences, not on assuming that nominally identical product samples have exactly the same value.
The consequence is important before conducting a larger comparison study. If the instrument’s resolution is poor relative to the measurement need, the study cannot provide useful discrimination at that scale. Collecting more entries does not reveal distinctions that the measuring system cannot indicate. This laboratory characterization should be designed and performed by someone competent for the measurement. The description is not a claim that ordinary catalogue dimensions are reference standards or that a sourcing team has already established resolution for every instrument it might use.
What does linearity mean in this gauge-study discussion?
The handbook uses a narrow interpretation: equal changes in the stimulus should produce equal response changes, or an existing bias should remain constant over the measurement range. A gauge may behave satisfactorily around one reference value while showing a different scale relationship elsewhere. A check at the middle of the range therefore does not establish the behavior at both ends.
NIST’s described study uses more than four reference standards distributed fairly evenly over the range of interest, with more than one measurement on each. It first plots measured results against the corresponding reference values. The expected one-to-one relationship has slope one; departure from that relationship needs examination. The study counts and layout describe this particular method. They are not a universal calibration schedule, and the page does not supply acceptance tolerances for a mugwort-product characteristic.
How do slope and intercept help distinguish scale error from an offset?
For the fitted relation Y = a + bX + measurement error, Y is the measured result and X the reference value. The intercept a describes an offset in this model, while b describes the response scale. NIST calls for software output that includes the intercept and slope estimates, their standard deviations, the residual standard deviation and a goodness-of-fit assessment. The fitted line should be evaluated with its supporting information, not only displayed as an equation.
A slope different from one departs from the intended one-to-one response. An intercept different from zero indicates bias in the stated model. These statements are about the handbook’s gauge-response interpretation; a straight line with a nonunit slope is still mathematically a straight line. Potential causes listed include endpoint calibration problems, errors in reference values, wear and internal design problems. If the gauge is used only to compare very small differences near one nominal size, the relevant linear range can be that small comparison interval rather than the entire instrument range.
How can measurement sequence interact with instrument drift?
Drift is a slow change in instrument response. For comparator measurements, NIST identifies short-term heat buildup as a frequent cause and points to calibration designs that mitigate linear drift. The page distinguishes this from long-term drift that is effectively constant during a short comparison design and can cancel in difference measurements. That cancellation depends on the comparison arrangement; it is not a general promise that instrument drift never affects an inspection.
This distinction matters when an approved reference is measured at one time and a new product sample much later. The measurement sequence and interval may be relevant to the observed difference. Retain the sequence when investigating disagreement, rather than comparing two isolated values without their timing. The linked drift-resistant calibration designs require their own detailed method review before implementation. Merely citing the introductory page does not establish that an ordinary sequence of factory readings followed such a design.
What can a time record show about drift without extrapolating it forever?
For direct-reading instruments, the handbook describes daily measurements on two or more check standards over a preset period, such as a month, followed by plotting the measurements over time. For instruments corrected by a linear calibration line, it instead points to a control-chart approach with three or more check standards. These are different described monitoring arrangements, not interchangeable minimum counts for any product inspection.
NIST cautions that drift rarely continues indefinitely at the same rate and in the same direction. The purpose of the bounded study is to document the change over a defined interval and support an adjustment plan. A trend observed in that interval should not be extended automatically for years. The page also treats uncorrected drift as an uncertainty contribution. Accordingly, keep the observed interval, treatment of drift and later adjustment distinct; a monitoring graph does not by itself prove that a correction was applied to the results already issued.
Does agreement with the average of several gauges establish the true value?
NIST’s gauge-comparison method averages each artifact’s readings across days for each gauge, then computes an average across gauges for that artifact. Each gauge’s difference from that artifact average is examined. A gauge that remains above or below the others shows a relative high or low bias. The reference in this analysis is the group average. It should not be presented as an independently certified true value simply because several instruments contributed.
The source design uses one operator and several check standards over several days to investigate whether gauge differences are consistent across artifacts and time. Where repeated runs are used, their statistics are computed separately. Its resistivity example identifies one probe as consistently low relative to the others. The transferable point is the comparison structure, not that probe’s numeric offset. For like-kind gauges, the handbook discusses either correcting a specific gauge’s bias with an associated uncertainty or retaining the readings and accounting for inter-gauge differences in uncertainty.
Can a configuration difference change direction between runs?
Yes. The handbook’s wiring example uses one resistivity probe in two configurations and repeats the study after about a month. It reports a configuration difference in both runs, but the direction changes. Neither configuration was established as the correct one. The source therefore treats the differences as an uncertainty contribution and calls for further investigation. This is evidence about that measurement study, not evidence that an unrelated product changed between shipments.
The important comparison is paired: the same artifact and day are used when examining the two configurations. Removing those links can obscure what difference was actually calculated. The example also cautions against assuming a correction established in one run will have the same sign later. If a supplier and buyer obtain different measurements using different setups, preserve the setup identities. The label “same gauge” does not resolve a configuration effect. Before selecting a correction, establish which reference and evidence justify it; agreement in one direction on one occasion is not enough to declare one setup universally correct.
Why should the measurement remedy match the source of error?
Different measurement problems require different responses. NIST discusses repeated measurements to reduce the variability of an average where feasible, or improving gauge performance where repeated measurement is impractical. It lists wear, humidity, temperature excursions and operator technique as possible causes of excess variability. Inadequate resolution requires a better gauge; collecting additional readings cannot create missing discrimination.
For level-dependent response errors, the handbook discusses correction or accepting the remaining effect only when the resulting uncertainty still meets the measurement requirement. Drift correction requires sufficient history to establish its direction and size. Differences among instruments or configurations can be corrected specifically or represented in the uncertainty. These alternatives do not authorize arbitrary changes to reported values. Record the identified problem, chosen treatment and uncertainty consequence together. A correction that hides a result outside a product tolerance without a supported measurement basis is not the remedy described here.
FAQ
Why can calibration and working conditions tell different stories?
A calibration comparison relates the instrument’s output to a reference under the calibration conditions. NIST warns that this does not ensure the same behavior in the working environment. Temperature, humidity, the operator and wear can introduce measurement bias. A disagreement between a buyer’s reading and a factory’s reading should therefore not be assigned immediately to product variation. The measurement environments can also differ.
Why should the measurement remedy match the source of error?
Different measurement problems require different responses. NIST discusses repeated measurements to reduce the variability of an average where feasible, or improving gauge performance where repeated measurement is impractical. It lists wear, humidity, temperature excursions and operator technique as possible causes of excess variability. Inadequate resolution requires a better gauge; collecting additional readings cannot create missing discrimination.
Sources and scope
Sources: NIST/SEMATECH handbook 2.4.5–2.4.5.6. Measurement-system bias, resolution, range response, drift, instrument/configuration differences and remedies. Laboratory methods and historical examples explain evidence; they are not current SKU measurements or supplier approvals. NIST MPC45 NIST MPC451 NIST MPC452 NIST MPC453 NIST MPC454 NIST MPC455 NIST MPC456