Automatic Bug Filing

I like to automate things. This is welcome trait at APT, as rapidly developing software with an engineering team about 20 in size does not leave much time for manual testing. Out of necessity, we have built a fairly sophisticated automating testing framework, which has been critical in monitoring the integrity of our code. We have software that interacts with our product as if somebody was controlling it themselves. Along the way it checks for errors, or even worse, changed output numbers. The testing code that tells the software what to do is dynamically generated from an object-oriented state-based model abstracted in a database. This allows us to quickly create thousands of test cases that interact with our product in a variety of different ways. These test cases are prioritized and assigned to one of about a dozen automated testing machines which constantly execute them every minute of every day of every week and report on the results.

So there we have it: a distributed prioritized automated testing framework. What more could we want? Well, I found myself spending a lot of time examining the failed tests. If I determined that the problem encountered was not a known issue, I would file a bug a report with the relevant information. Otherwise, I would have to take note that we know about this issue and ignore that test until it is fixed. My coworkers were experiencing the same thing. As we scaled our framework to run more and more tests, we had no analogous expansion of our abilities to monitor and react to the result of these tests. This is where our affinity for automating things comes in. Why not automate responses to our automated tests? And this is exactly what we did.

Now when one of our automated tests hit an error, a check is done to see if it is a new error or not. If it is not a new error, we associate the test with it. This association helps us avoid wasting any more time on subsequent failures as well as logging which tests to use to determine if the problem has been fixed. If it is a new error, we automatically file a detailed bug report with an appropriate priority determined by characteristics of the test case and the error hit.

This automation was not without complexities; in fact we are still working out some kinks. First of all, it hinges on the ability to accurately determine if errors are new or not. Once that is done, you want to be able to filter out errors that are not relevant. Automatic bug filing is a fine line. File too few bugs and you still must spend time going over reports checking for things that may have been missed. File too many bugs and you have to go through them all and weed out the legitimate ones. However once the logic is tweaked the previously manual task of responding to the results of automated tests is now automated itself. The benefits include less time spent looking over reports, as well as zero lag time between the time a problem occurs and the time a bug report is filed, quickly bringing the issue to the attention of product engineers. And of course there is the good feeling you get when you’ve automated something that used to be done manually!