Identifying and Using Oracles

Definition:

An Oracle is a principle or mechanism by which we recognize a problem. It is a special kind of heuristic that helps us to make a particular decision in answer to the question, “Is there a problem here?” as defined by Michael Bolton.

A context where I used this skill:

I had just joined a medium size software development company, in a team of 3 testers and 15 developers. My role was to test a web application being developed in a hybrid-agile form where stories are not kept updated. As documentation was not updated I had no initial reference material to use to learn what the application was supposed to do and to prepare and design my tests. I used the knowledge from developers and UX as oracles as well as Michael Bolton’s FEW HICCUPS heuristics as a first step for identifying and applying oracles, while exploring the application. I found the ‘comparable products’ a helpful oracle as there were a few competitors in the market, as well as the ‘history’ oracle as we were replacing an old platform with a newer one. For testing of a database engine, one concern was the possibility that changing one record might also change other unrelated records. One oracle we used was self-verifying data. The data set needed to be large (100,000,000 records), so entries were generated by generating well-formed records using random numbers. The seed value for each record was embedded within the record, so after a test (or suite of tests) we were able to walk through all of the records and re-generate the expected contents for each using the embedded random number seed. - DHH Sometimes we make problem solving so challenging that we miss the obvious solution. Recently, during the first week of January of 2014, I was working with a test group that was not only remote from me but in multiple locations across a continent. On a video call to discuss some suggestions I had made, there were some questions raised about the approach and why, if we were testing a new, web-based application, would people (testers) need to be able to log into the legacy host system. I asked what they thought they should compare the data in the new system to in order to be certain it was correct? One person suggested the old web-based system that was being replaced. I asked what the data source was for that system. There was a long moment of silence while the participants blinked. The person who had asked, originally, about the legacy system had a look of astonishment on her face.

Of course! The data there is correct. We simply are replicating and reporting it on the new system. We can let the old system tell us what is right or not.

How I’d recommend someone learn this skill:

There are many resources to learn about oracles (see additional resources below). Practice however, is the best way to learn how to identify them and use them. Weekend Testing is a place where testers can come and practice test skills together over Skype. In one of the Weekend Testing sessions, we concentrated in testing with a focus on using oracles. We used FEW HICCUPS as the starting point for identifying and using oracles and testers found that helpful. Full transcript of the session can be found here. For me, this was an exercise in defining problems. The list of resources below (provided by DHH) is very good and I have referred people to several of them. Just as we learned basic addition and subtraction by checking “the answers” when we were done, now I expect it is checking the calculator) this is sometimes something similar. Each of our contexts are slightly different. For fields such as name or address that might display on a screen, I find it helpful to use some of Doug’s self-verifying data (described above and in the resources below.) I have found it helpful when creating test data to give records meaningful, easily identifiable names embedded in specific field in the data. Trial and error, and awareness of context, helped me refine my test data for each environment I worked in. When I needed legitimate addresses, I used addresses of some of my favorite restaurants. If they appear correctly, then I know they are correct.

Additional resources:

By: Pete Walen

Quick Attacks →