Algorithm A (ISO 13528) explained, step by step

Why PT needs robust statistics

PT results often include a few values far from the rest: a unit error, a transposed digit, a contaminated item. The plain mean moves towards them, and the plain standard deviation, which squares every deviation, grows sharply with them. Scored against those figures, every other participant gets a misleading z-score.

Robust statistics describe the bulk of the data and give little weight to values far from it. The simplest are the median and a scaled median absolute deviation. Algorithm A starts from those two and refines them, so that results far from the centre still count, but only up to a limit.

The starting point: median and MADe

• Median: the middle value of the sorted results, or the mean of the middle two when the count is even.
• MAD: the median of the absolute deviations of every result from the median.
• MADe = 1.483 × MAD. For normally distributed data the MAD is about 0.6745 standard deviations, and 1 / 0.6745 = 1.483, so MADe estimates the standard deviation.

The median and MADe are good estimates on their own, and the usual choice when a round has few results. Their weakness is that they use little of the information in the data: the median depends only on the middle one or two values. Algorithm A uses every result while capping how far any one of them can pull.

Algorithm A, step by step

ISO 13528:2022 describes Algorithm A in Annex C, its annex on robust analysis. In plain words, the procedure is:

1. Start. Set x* to the median of the results and s* to 1.483 × MAD.
2. Set the limits. Calculate δ = 1.5 × s*. Replace any result below x* − δ with x* − δ, and any result above x* + δ with x* + δ. Results inside the limits stay as they are.
3. Update. Set x* to the mean of the adjusted values, and s* to 1.134 × their standard deviation.
4. Repeat steps 2 and 3 until x* and s* stop changing. By hand, stop when neither changes in its third significant figure from one iteration to the next.

Pulling values in reduces their spread, so the standard deviation of the adjusted values underestimates the spread of the data. The factor 1.134 makes up for that, so that s* still estimates the standard deviation when the data are normal.

The output is a robust mean, x*, often used as the assigned value x_pt, and a robust standard deviation, s*.

A worked example

Fourteen participants report available phosphorus in a soil test item, in mg/kg. In ascending order:

38.2, 43.1, 43.8, 44.0, 44.4, 44.6, 44.9, 45.1, 45.4, 45.6, 46.1, 46.7, 47.4, 53.9

Two results stand out: 38.2 is low and 53.9 is high. The plain mean is 45.23 mg/kg and the plain standard deviation is 3.30 mg/kg, inflated by those two values.

Step 1: the starting values

With 14 results, the median is the mean of the 7th and 8th values, 44.9 and 45.1, which is 45.0.

The absolute deviations from 45.0, in ascending order, are 0.1, 0.1, 0.4, 0.4, 0.6, 0.6, 1.0, 1.1, 1.2, 1.7, 1.9, 2.4, 6.8, 8.9. Their median is the mean of 1.0 and 1.1, so MAD = 1.05 and the starting s* = 1.483 × 1.05 = 1.557.

Step 2: the first iteration in full

δ = 1.5 × 1.557 = 2.336. The limits are 45.000 minus and plus 2.336: 42.664 and 47.336.

Three results fall outside. 38.2 is replaced by 42.664, and 47.4 and 53.9 are both replaced by 47.336. The other eleven results stay as they are. The 14 adjusted values add up to 631.036, so the new x* = 631.036 / 14 = 45.074. Their standard deviation is 1.454, so the new s* = 1.134 × 1.454 = 1.649.

Step 3: repeat until stable

Each iteration starts from the x* and s* of the one before. The table shows every iteration until the hand stopping rule is met.

Two things happen along the way. First, 47.4 is pulled in on the first iteration but not after it: as s* grows to 1.649, the upper limit moves out to 47.548 and 47.4 falls inside. Second, s* keeps rising for several iterations, because the starting MADe of 1.557 was on the low side for these data.

After iteration 5, x* and s* round to 45.1 and 1.73, the same as after iteration 4 (45.1 and 1.73), so by hand you stop there. Run until nothing changes at all, the algorithm settles at x* = 45.092 mg/kg and s* = 1.734 mg/kg, the same to three significant figures. Compare the plain mean of 45.23 and the plain standard deviation of 3.30.

Pulled in is not thrown out

The adjusted values exist only to calculate x* and s*. Every participant is still scored on the result it reported. Suppose the scheme plan fixed σ_pt at 2.0 mg/kg before the round, and the assigned value is x* = 45.092 mg/kg:

The two results that Algorithm A held at the limits are still flagged, because they are scored on what was reported. The algorithm protects the assigned value from them; it does not hide them.

The provider also checks that the assigned value is known well enough to score against. For a participant consensus, its standard uncertainty is usually taken as u(x_pt) = 1.25 × s* / √p, where p is the number of results. Here that is 1.25 × 1.734 / 3.742 = 0.579 mg/kg. ISO 13528 treats this uncertainty as small enough to ignore when it is no more than 0.3 σ_pt, here 0.60 mg/kg, so this round passes, narrowly. If it did not, the options include finding a better assigned value, or using the z′ score, which allows for the uncertainty.

When to use Algorithm A

Algorithm A fits when the assigned value and the spread both come from the participants’ own results, and:

• the results are quantitative and on one scale;
• there are enough of them: with fewer than about 12, robust estimates become unstable, and ISO 13528 gives separate guidance for small numbers of participants;
• the data form one roughly symmetric group, with at most a minority of outlying results.

Providers often run it on all results and again within each method group, when a scheme reports method-group statistics.

When a reference value is available, from formulation, a reference material with a certificate or a reference laboratory, it can serve as the assigned value, with Algorithm A as a cross-check. A large gap between the consensus and the reference value is a reason to investigate before scoring.

The limits of Algorithm A

• Small rounds. With few results, one participant can move x*, s* is poorly estimated, and the uncertainty of the assigned value is large. A reference value, or a σ_pt fixed in advance, is often the better choice.
• Two populations. When two methods give systematically different results, the data form two clusters. x* lands between them, where nobody reported, and s* swells. Plot the results, as a dot plot or histogram, before trusting any consensus.
• A large share of outliers. Algorithm A is built for a minority of wild results. Many wrong results, especially on one side, drag x* and inflate s*. ISO 13528 also describes estimators that resist a larger share of outliers.
• Many identical results. If more than half the results are the same, the MAD is zero, so s* starts at zero and the limits never open. Results reported with very few digits can cause this. The scheme plan then needs another approach.
• A bias most participants share. A consensus describes what the participants measure, not the true value. If most of them use a method that reads high, x* reads high too, and the few participants who are right are the ones who look wrong.
• σ_pt taken from the same data. If σ_pt is set to the round’s own s*, about 95 % of results will look satisfactory whatever their real performance, because the score then measures agreement with peers, not whether the results are good enough for their use.
• It is not an outlier test. Algorithm A does not decide whether a result is an outlier, and it removes nothing. Outlier tests such as Grubbs’ test answer a different question.

Try it yourself

The free Algorithm A calculator in the EqasOnline Academy runs the same procedure in your browser and shows x*, s* and u(x_pt) for any list of results. Paste in the example results above and check that it gives 45.09 and 1.73. Then change 53.9 to a decimal slip such as 539, and watch the plain mean jump while x* stays exactly where it was: the slipped result is pulled to the same limit as before.