Break the Code

The question that sorts every error

On the previous topic you met defensive design: validating input, anticipating the ways people misuse a program, writing code somebody else can read. Testing is how you find out whether any of that actually worked. And the first thing a question will ask you to do is say what KIND of error you are looking at. Most people learn two definitions and then hesitate over every real example. There is a much faster way, and it is one question: did it run? If the program would not start, the language could not make sense of something you wrote. It never got as far as doing anything. If the program ran all the way through and gave you the wrong answer, the language understood every single line and did exactly what you asked. The instructions were wrong, not the writing. That one question settles it every time, and the rest of this topic is about choosing data that makes faults show up in the first place.

Words for testing

Your specification names four kinds of test data. Learn the third and fourth together, because telling those two apart is where the marks are.

Two kinds of error, one test

Students lose marks here by guessing from how serious the problem feels. Ask the question in the third column instead and it stops being a judgement call.

How to build a test table

A test table has three columns and they are always the same three: the data you are going to feed in, what kind of data that is, and what you expect to happen. ⚠️ The third column is filled in BEFORE you run anything, and this is the part almost everybody skips. A test you run without deciding in advance what success would look like cannot fail, because whatever the program does you will look at it and think, yes, that seems about right. Writing the expectation first is what turns pressing a button into a test. ⚠️ And choose your data on purpose rather than at random. You want some that should sail straight through, some sitting right on the edges of what is allowed, and some that ought to be turned away. If every row you picked is the kind that works, you have not tested anything - you have just watched the program succeed.

Match each value to what kind of data it is

It ran, and it was wrong

A program runs all the way to the end without complaining, but the total it prints is always exactly one too high. What kind of error is this, and how can you tell?

• A logic error, because the program ran, so the language understood every line and simply did what it was told
• A syntax error, because the output is wrong
• A syntax error, because a line must have been mistyped somewhere
• Neither, since a program that runs without crashing cannot contain an error

True about the two kinds of error

Select the TWO statements that are true.

• A syntax error stops the program running at all, because the language cannot make sense of it
• A logic error can only be found by comparing what the program produced against what it should have produced
• A logic error will be reported by the language before the program starts
• Any program that runs without crashing is correct

A test table, filled in properly

Here is a plan for one field: a box that should accept a whole number from 1 to 10. Three columns, in this order, and read the third one carefully. 5 - normal - should be accepted and used. 1 - boundary - the smallest allowed value, should be accepted. 10 - boundary - the largest allowed value, should be accepted. 0 - invalid - should be refused, with a message explaining why. 25 - invalid - should be refused in the same way. seven - erroneous - should be refused without the program crashing. ⚠️ Two things about that table are worth more than the rows themselves. First, every entry in the third column was written before any of it was run. That is what makes each row a test rather than an observation. Second, look at the spread: two that should work, two edges, and two that should be turned away. A plan made only of the first row proves nothing. ⚠️ And the last row is the one that catches real programs. You met on the data types topic that what comes back from a keyboard is text, and that converting text which is not a number has no sensible answer. The erroneous row is exactly the test that finds a program which assumed every input would convert.

Complete the testing paragraph

An error that stops a program running at all, because the language cannot make sense of a line, is a _____. An error that lets the program run to the end but produces the wrong result is a _____. Testing carried out during development, on parts of a program as they are built, is _____. Testing carried out on the finished program against the original requirements is _____.

Sort the test data

A field should accept a month as a number from 1 to 12. Five in a row, three lives.

How many rows in the plan

A test plan covers 2 normal values, 2 boundary values, 3 invalid values and 1 erroneous value. Add these together to find how many rows the test table has in total.

Spot the true testing facts

Tap the TWO statements that are true.

• A program that runs to the end but gives the wrong answer has a logic error
• The expected result is decided before the test is run
• Erroneous data means data of the right type that falls outside the allowed range
• Final testing is carried out on parts of a program while they are still being built

Three testing calls

Three situations. Choose the response you could defend in writing.

• A student proudly shows a test plan with six rows. Every row is a sensible value that an ordinary user would type, and every one passes. What is wrong with the plan?
• A program refuses to start at all, and the editor highlights one line. What kind of error is it, and what does the refusal tell you?
• A field should accept a whole number from 1 to 100. You have room for one more test and you want to check the top edge. Which value?

Write the testing advice

A friend has finished a program, run it once with a sensible value, and declared it tested. Write them the advice.

• Explain the difference between a syntax error and a logic error, and give the one question that tells them apart
• Name the four kinds of test data and give an example of each for a field that accepts a whole number
• Explain precisely how invalid data differs from erroneous data
• Explain what the three columns of a test table are, and say which one has to be filled in first and why
• Finish by explaining the difference between iterative and final testing, and what each one is for