Interpreting and Evaluating Statistics

Correct, and still wrong

A statistic can be perfectly correct and still be used to support something it does not support. Judging that is a skill in its own right, and it is what an examiner is testing whenever a question asks you to comment on a conclusion, to say whether a claim is justified, or to evaluate how an investigation was carried out. You will not be asked to redo the arithmetic somebody else has already done. You will be asked what the number actually says, whether the way it was produced lets it say that, and whether the conclusion drawn from it goes further than the evidence allows. Three habits carry most of the marks here. Read the number back against the question that was actually asked. Look at where the data came from before you trust what it is being used to prove. Check whether the wording of the conclusion claims more than the figures show. Note what this module is not about: it works on finished claims and finished investigations, rather than on how to plan and run one yourself.

The words the marks hang on

Five terms that turn a vague objection into a criticism worth credit.

What is wrong with each claim

How far does the evidence reach?

A study finds that pupils who eat breakfast score higher in tests than pupils who do not. Which conclusion does that evidence support?

• Eating breakfast makes pupils score higher in tests
• Pupils who eat breakfast tend to score higher than pupils who do not
• Pupils who score highly have been made to eat breakfast
• Skipping breakfast is the main reason some pupils do badly

The figures and the claim

Put a claim in two columns. On the left, what was actually measured. On the right, what the sentence adds on top. The gap between the columns is your criticism.

Check it in this order

Five things to do when you are handed a claim to evaluate. Put them into the order that works.

• Read the claim and note exactly what it asserts
• Find the figure the claim rests on, and what that figure measures
• Ask where the data came from and who was included in it
• Compare what the figure shows with what the claim asserts
• State the gap between the two in one specific sentence

Can this sample carry the claim?

A claim about all shoppers in a town rests on one survey. Which THREE of these would weaken what that survey can be used to claim?

• Everyone was asked outside the same shop on the same morning
• Only people willing to stop and answer ended up in the results
• Forty people were asked, and the town has thirty thousand in it
• The questions were written down before the survey began
• The results were recorded to one decimal place

Two things rising together

When a claim leaps from a pattern shared by two measurements to one of them producing the other, the right response is not to say the figures are wrong. The figures may be exactly right. Three other explanations have to be ruled out first, and naming one of them is what earns the mark. A third factor may be driving both, which is why ice cream sales and cases of sunburn climb together every summer without either one producing the other. The direction may be the reverse of the one assumed, so that what has been treated as the effect is really the cause. Or it may simply be chance, which turns up more often than people expect once enough pairs of measurements have been compared. In an exam, one clear sentence naming a specific alternative explanation for this particular pair of measurements is worth far more than a general remark that correlation is not causation. The general remark tells the examiner you have heard the phrase. The specific alternative tells the examiner you can apply it.

Complete the evaluation words

A sample that can reasonably stand for the whole group it is used to describe is called _____. Anything in the way data was collected that pushes the result one way is called _____. Reading a figure against the question that was actually asked is reading it _____. Something about the method that restricts what the results can be used to claim is _____.

Comparing across different tests

Standardised scores exist to do one job. A mark of 45 out of 100 and a mark of 45 out of 60 are not comparable, and neither are two tests where one was far harder than the other. A standardised score measures a result in standard deviations away from the mean of its own test, which puts every test on the same footing. Subtract the mean of that test from the result, then divide by the standard deviation of that test. Work one through. A pupil scores 45 on a test where the mean was 50 and the standard deviation was 10. Subtracting gives minus 5, and dividing by 10 gives minus 0.5. The negative sign is not a mistake and it is the part most often dropped: it says the result sits half a standard deviation below the average for that test. A standardised score of 0 is exactly average, a positive score is above average, and a negative score is below it. Now the comparison becomes possible. Whichever test a pupil sat, the pupil with the higher standardised score did better relative to everyone who sat the same paper.

Standardise the result

Ravi sits a different test and scores 74. On that test the mean was 62 and the standard deviation was 8. Work out the standardised score for Ravi.

Find the sentence that overreaches

Five sentences from one report, in the order they were written. Tap the ONE that claims more than the data can support.

• A researcher surveyed 500 households across the whole of the district, chosen at random from the electoral register.
• Of those asked, 62 in every 100 said they recycled glass every week.
• That figure is higher than the 48 in every 100 recorded in the same district five years ago.
• The rise followed the opening of two new recycling points in the district.
• The new recycling points have therefore increased weekly glass recycling by 14 in every 100 households.

Write the criticism

A vague objection earns little. Choose the option in each gap that turns it into a sentence an examiner can credit.

Evaluation gauntlet

Five quick judgements. Three lives.

Evaluating a finished report

A local newspaper reports that its own survey shows the new bus lane has made journeys faster. The survey timed 30 journeys along the route in the week after the lane opened, and found the mean journey took 4 minutes less than a figure published by the council last year. You have been asked to evaluate the report. Work through three decisions.

• Where do you start?
• You find that the council figure was an annual mean across all times of day, while the 30 journeys were all timed between ten and eleven in the morning. What does that tell you?
• How do you word the evaluation?

Evaluate the claim

A company reports: 92 in every 100 of the people we asked said our product improved their sleep, so our product improves sleep. The people asked were those who had already bought the product and had signed up to the company mailing list. Write an evaluation of that claim. You are not being asked to say the figure is false.

• Say what the figure does establish, and about which group of people
• Name the specific problem with who was asked, rather than only saying the sample was biased
• Say why the wording of the conclusion reaches further than the evidence
• Name one thing that would have to be done differently for the stronger claim to hold
• Finish with what the company could honestly say instead