I came across an article this evening, and based on an experience this afternoon, it made me think. How much software testing is enough?
This all started out this afternoon. A couple of times a week I go out to lunch with my friend Chris, and we spend time talking about all kinds of different things. Sometimes we find amusing things so we like to share them. This afternoon, Chris was showing me that he stumbled across the speech recognition of his Honda Civic Hybrid. He thought that it was mainly to control the GPS navigation, but it turns out that it does a lot more than that. We found ourselves trying to command “radio, 88.2” (nonsensical FM frequency) or “radio, 88.5” (it only goes to 88.7) to see the response. It turns out that you can also turn on/off the air conditioner.
This banter quickly degenerated to “damn car, tune the radio to 870!” which amazingly tuned the radio to 870AM. I tried yelling “Eject! Eject!” but nothing happened. For a brief instant, I was considering “set speed to maximum,” but I did not say it, just in case. The Honda Hybrid does not have a real throttle cable; just a potientiometer that is hooked to the gas pedal. This is essentially an input to the computer, which controls the electric motor & gas engine.
So, it’s quite possible that there could be a software bug, and some condition might cause a seriously undesirable side effect. Thus, the above article. Now, I’m not saying that’s what happened, but it makes you wonder. Is there a governing computer that independently shuts down the system when the wrong outputs are seen? If you peer into a traffic signal computer box, you will find one computer that does the traffic signals, and another that is monitoring everything and in the slightest instance of trouble (say, green lights for perpendicular directions), it shuts the thing down into flashing reds for all directions, and requires human intervention to reset. I think that’s why you tend to see flashing reds in rainstorms — things get wet, currents flow where they should not, and the governing computer senses some malfunction state.
So how do you solve this? Test, test, and more test. The ECZ space station code was path tested for every possible execution path. That’s a lot of testing, but a manned space program requires it. That’s a lot of cost, too. A huge amount. So, what do you do if you have a competitive environment with fast turnaround times and new models always coming out due to customer demand? How far does the testing get taken? When is the test done?