Outliers and Data Cleaning
Identifying extreme values and making informed decisions about data quality
SPEC 2.4 • DATA PRESENTATION AND INTERPRETATION
What is an Outlier?
Definition
An outlier is a data value that is significantly different from the other values in a data set. It lies outside the expected range of the data.
Consider this data set of exam scores:
45, 52, 55, 58, 61, 63, 65, 67, 70, 12
~60
The score of 12 is clearly separated from the main cluster of data
Key Concept
Outliers can occur for various reasons: measurement errors, data entry mistakes, or genuine extreme observations. We need a formal method to identify them.
The IQR Rule for Outliers
The standard method for identifying outliers uses the interquartile range (IQR) to define outlier limits.
Outlier Limits
Outlier Definition
Any data value below the lower limit or above the upper limit is classified as an outlier.
OUTLIER ZONES
Lower Limit Q₁ - 1.5×IQR
Upper Limit Q₃ + 1.5×IQR
OUTLIER ZONE
NORMAL RANGE
OUTLIER ZONE
Important
The IQR rule with multiplier 1.5 is the standard method, but other methods exist (e.g., 2×IQR, 3×IQR, or standard deviation methods). Always check what method the question specifies.
01Worked Example
The times (in minutes) taken by 11 students to complete a puzzle are:
3, 5, 7, 8, 9, 10, 11, 12, 14, 15, 28
Calculate the outlier limits and identify any outliers.
1
Find the quartiles. Data is already ordered, n = 11.
2
Q₁ is at position (11+1)/4 = 3rd value → Q₁ = 7
3
Q₃ is at position 3(11+1)/4 = 9th value → Q₃ = 14
4
IQR = Q₃ - Q₁ = 14 - 7 = 7
5
Lower limit = Q₁ - 1.5 × IQR = 7 - 10.5 = -3.5
6
Upper limit = Q₃ + 1.5 × IQR = 14 + 10.5 = 24.5
Answer28 > 24.5, so 28 is an outlier. No values below -3.5.
Worked Example02
A data set has mean x̄ = 45 and standard deviation σ = 8. Using the rule that outliers are more than 2 standard deviations from the mean, determine whether the values 28 and 65 are outliers.
1
Calculate the outlier limits using mean ± 2σ
2
Lower limit = x̄ - 2σ = 45 - 2(8) = 45 - 16 = 29
3
Upper limit = x̄ + 2σ = 45 + 2(8) = 45 + 16 = 61
4
Check 28: Is 28 < 29? Yes → 28 is an outlier
5
Check 65: Is 65 > 61? Yes → 65 is an outlier
AnswerBoth 28 and 65 are outliers using the 2σ rule.
03Worked Example
The heights (cm) of 40 plants are recorded:
Height (cm)FrequencyCF
10 - 1544
15 - 201014
20 - 251428
25 - 30836
30 - 35440
Determine whether a plant height of 38 cm would be an outlier.
1
Find Q₁ position: n/4 = 40/4 = 10th value. Q₁ is in class 15-20 (CF goes 4→14)
2
Q₁ = 15 + ((10-4)/10) × 5 = 15 + 3 = 18 cm
3
Find Q₃ position: 3n/4 = 30th value. Q₃ is in class 25-30 (CF goes 28→36)
4
Q₃ = 25 + ((30-28)/8) × 5 = 25 + 1.25 = 26.25 cm
5
IQR = 26.25 - 18 = 8.25 cm
6
Upper limit = 26.25 + 1.5(8.25) = 26.25 + 12.375 = 38.625 cm
Answer38 < 38.625, so 38 cm is not an outlier (just within the limit).
Effect of Outliers on Statistics
Different statistical measures are affected differently by outliers:
StatisticAffected by Outliers?Why?
MeanStrongly affectedUses all values in calculation
MedianResistantOnly uses middle value(s)
ModeResistantOnly uses most frequent value
RangeStrongly affectedUses extreme values directly
IQRResistantOnly uses Q₁ and Q₃
Standard DeviationStrongly affectedUses squared deviations from mean
Exam Tip
When data contains outliers, median and IQR are better measures than mean and standard deviation because they are resistant to extreme values.
04Worked Example
A data set has values: 10, 12, 14, 15, 16, 18, 50
Calculate the mean with and without the outlier 50. Comment on the effect.
1
Mean with outlier = (10+12+14+15+16+18+50) ÷ 7 = 135 ÷ 7 = 19.3
2
Mean without outlier = (10+12+14+15+16+18) ÷ 6 = 85 ÷ 6 = 14.2
3
Difference = 19.3 - 14.2 = 5.1
Comment: The outlier increases the mean by 5.1 (from 14.2 to 19.3). The mean with outlier (19.3) is higher than 6 of 7 values, making it unrepresentative.
Data Cleaning Decisions
When you identify an outlier, you have three main options:
Option 1: Keep
When: The outlier is a genuine, valid observation
Example: A record-breaking athlete's performance time
Option 2: Remove
When: The outlier is clearly an error that can be verified
Example: A person's age recorded as 250 years
Option 3: Investigate
When: The cause of the outlier is unknown
Example: Unusually high reading - check equipment, retest if possible
Exam Advice
When asked to comment on outliers, always consider the context. State whether the outlier is likely to be an error or a genuine value, and justify your decision.
05Worked Example
A teacher records test scores. One student scored 15% while all others scored between 45% and 85%. This is identified as an outlier.
Give two possible reasons for this outlier and suggest whether it should be removed.
1
Reason 1: The student may have been ill or had personal circumstances affecting performance.
2
Reason 2: Data recording error - perhaps 15% was meant to be 51% (digits transposed).
3
Decision: Should not be automatically removed. If genuine, it's a valid data point. Investigate the cause first - check with student or verify the recorded mark.
06Worked Example
A scientist measures 10 samples. The lower limit is 12.4 g and upper limit is 28.6 g. A sample has mass 11.8 g.
(a) Is 11.8 g an outlier? (b) Find Q₁ and Q₃ if IQR = 5.4 g.
(a)
Is 11.8 < 12.4? Yes, so 11.8 g is an outlier (below lower limit).
(b)
Lower limit = Q₁ - 1.5 × IQR
•
12.4 = Q₁ - 1.5 × 5.4 = Q₁ - 8.1
•
Q₁ = 12.4 + 8.1 = 20.5 g
•
Q₃ = Q₁ + IQR = 20.5 + 5.4 = 25.9 g
Practice
Question 1: A data set has Q₁ = 18, Q₃ = 34. Calculate the outlier limits and determine whether the values 2 and 55 are outliers.
Step 1: IQR = 34 - 18 = 16
Step 2: Lower limit = 18 - 24 = -6
Step 3: Upper limit = 34 + 24 = 58
Step 4: Is 2 < -6? No. Is 55 > 58? No.
AnswerNeither 2 nor 55 are outliers.
Practice
Question 2: The values 4, 6, 8, 9, 10, 12 have mean 8.17. If the outlier value 25 is added, calculate the new mean and comment on the effect.
Step 1: Original sum = 49
Step 2: New sum = 49 + 25 = 74
Step 3: New mean = 74 ÷ 7 = 10.57
Comment: The outlier increased the mean from 8.17 to 10.57 (increase of 2.4). The new mean is now higher than 5 of the original 6 values, making it less representative.
Practice
Question 3: Most employees earn £25,000-£45,000, but the CEO earns £450,000. This is an outlier. Should it be removed when calculating average company salary? Justify.
This is a genuine outlier - the CEO's salary is a real, valid data point, not an error.
Decision depends on purpose:
For "typical employee salary" → exclude or use median
For "total salary expenditure" → must include
For company comparisons → report both (with and without)
AnswerDon't simply remove. Use median instead of mean, or report mean with and without the outlier.
Summary
Key Concepts
ConceptFormula / Rule
Lower limit (IQR)Q₁ - 1.5 × IQR
Upper limit (IQR)Q₃ + 1.5 × IQR
Lower limit (SD)x̄ - 2σ (or 3σ)
Upper limit (SD)x̄ + 2σ (or 3σ)
OutlierValue outside the limits
Resistant measuresMedian, IQR
Non-resistant measuresMean, range, SD
Key Exam Skills
Calculate outlier limits using IQR or standard deviation methods
Use interpolation to find quartiles from grouped data
Evaluate whether to keep or remove outliers based on context
Key Formulas
IQR Method (Standard)
Standard Deviation Method
where k = 2 or 3 (as specified in question)
Remember
Always check which method the question specifies. Show clear working: calculate limits first, then compare values to both boundaries.
Next: What is an Outlier?