Reliability, Validity and Questionnaire Design

What makes data trustworthy?

Good statistics need good data. Two ideas measure data quality: reliability and validity. This module covers both, plus how to design a questionnaire that collects fair, useful data.

Key terms

Learn these definitions word for word:

Match each term to its definition

Define reliability

What does reliability mean in statistics?

• The extent to which repeated measurements yield similar results
• The extent to which a test measures what was intended
• The number of people in a sample
• The cost of running a survey

Designing a good questionnaire

A good questionnaire collects trustworthy data. Avoid leading questions that push an answer, and biased or emotive wording. Use closed questions for quick, comparable data and open questions for detail. Always run a pilot first to catch problems, and keep it short so people finish it.

Spot the poor questions

Select the TWO questions that are leading or biased.

• "Do you agree our wonderful park deserves more funding?"
• "How many hours of television do you watch per day?"
• "Most people recycle. Do you recycle too?"
• "Which age group are you in?"

Match each feature to the practice it shows

Open or closed?

Which of these is a closed question?

• "Which of these do you prefer: A, B or C?"
• "What do you think about the new park?"
• "Tell us your views on recycling."
• "Describe your journey to school."

Running a questionnaire

Put the steps of designing and running a questionnaire in order.

• Draft the questions
• Pilot the questionnaire with a small group
• Revise questions that caused problems
• Use the questionnaire for the main survey

Data-quality frame

A measurement is _____ if repeating it gives similar results. A test is _____ if it measures what was intended. Before a main survey, researchers run a _____ to find problems. A _____ question should be reworded because it pushes the respondent towards an answer.

Reliable is not the same as valid

The two ideas are different. A measurement can be reliable but not valid: a scale that always reads two kilograms too heavy is consistent (reliable) but wrong (not valid). Good data needs both: consistent results, and measuring the thing you actually intended.

Reliability or validity?

Decide whether each problem is about reliability or validity.

• A survey gives very different results each time it is repeated. This is a problem with...
• A test meant to measure maths skill actually measures reading speed. This is a problem with...
• A thermometer gives steady readings that are always two degrees too high. Its readings are...

Fix a poor question

A survey asks: "Do you agree that our brilliant town needs a new cinema?" Explain why this is a poor question and rewrite it as a fair one.

• Say what is wrong with the original question
• Give your rewritten, neutral version
• Explain why your version collects more valid data