Recently I worked on a project that had to support six different browsers; three versions of Firefox (1 through 3) and three versions of Internet Explorer (6, 7, 8). Right away this raised a few questions, and one of the more interesting ones was about testing:

What is the optimal environment for testing two browsers at three versions each?

This is an interesting issue because browsers are not designed to allow more than one version to be installed at once. This makes sense for users — it would be confusing and insecure for them to use an older version by accident — but it's traditionally a big hassle for web developers. It's important for us to be able to test our changes on multiple browsers because different versions of the same browser may not render the same code the same way, and it's important to catch and fix these bugs.

To answer this question, our team of myself (a developer) and two quality assurance experts did some brainstorming and each of us actually came up with our own solution. Here are the three options we considered:

Multiple VMs

This was our senior QA's suggestion. Set up three virtual machines, and on each machine, install a different version of Internet Explorer and Firefox.

Advantages:

• Tried, tested and true. This is a standard QA practice and has been in use for years.
• Can run all six browsers simultaneously (good if we want to test in parallel)

Disadvantages:

• There is some overhead involved in setting up and keeping track of three VMs.
• Three VMs will need to run on their own machine, so an extra box or some server space is needed.

VM Snapshots

This was our other QA's suggestion, which I thought was quite novel. The process goes like this: create a virtual machine and install IE6 and FF1, then take a snapshot (something the VM can revert to upon request). Now, upgrade the browsers to IE7/FF2 and take another snapshot, and perform the final upgrade to IE8/FF3 and take a third snapshot. By toggling between the three snapshots, all six browsers can be tested using only one VM.

Advantages:

• Only one VM is required, so there is less overhead.
• Since there is only one VM, it's possible to run the VM on a developer or QA's machine in a pinch.

Disadvantages:

• Can only test one version of each browser at a time.
• There is probably still a need for some shared machine space to host the VM.

Hack it Together

Predictably, this was the developer's solution (mine). I've used a slightly-unstable beta product called IETester in the past, which allows testing multiple versions of Internet Explorer simultaneously. This meant all I needed was a solution for Firefox, and after some searching I found an outline of how to install multiple versions of FF.

Advantages:

• No virtual machine overhead.
• Everything can be run as needed on the developer's machine.

Disadvantages:

• Highly technical: Setting up and maintaining this environment is not for the faint of heart.
• Can only run one version of FF at a time (though IE can be done in parallel).

Given these options, we decided that the best match for our project's needs was to use VM Snapshots on the quality assurance side, and the hacked-together solution on the development side. The low overhead was a big plus, especially for development, and it was important that QA was able to reliably test real versions of each browser. This worked out well for us, and I would definitely use such methods again.

Have you used these or similar solutions in the past? Are there other solutions out there that we don't know about? Let us know in the comments.

Labels: Browsers, Firefox, Internet Explorer, QA

Sometimes, testing your application isn’t enough: sometimes your test cases need to be tested! Testing doesn’t always give you what you’d expect. In extreme cases, it can even give you a false sense of confidence in a product that is a complete failure.

Recently, I was managing an offshore team developing a component for our client's core enterprise application. We wrote extensive test cases and, because we were concerned about a lack of domain knowledge, we even had architects write some of the test cases. Things were looking good as we approached the end of the development cycle: the product was behaving as it was supposed to; we were passing test cases; we were greenlighted by the QC ‘process’.

Our confidence was high, and we were ready to move on to automated testing. Then we found the problem. It turns out that our test cases were written with a single user in mind. The minute we started to use this component with concurrent user access, the system would pretty much lock up. Our confidence evaporated.

This was a big mistake. Architects and technical leads both reviewed the test cases. Nobody found the oversight at the time. This was compounded by an ‘if it passes the test cases then it works’ mentality.

Unfortunately, a solution to this problem isn’t as easy as ‘follow these three steps’. But if we can take away a lesson learned from this, it is that testing the right way is not only very important to the success of a product, but it also shouldn’t be taken for granted. If you haven’t noticed, QC/QA and testers were not included in the writing of test cases, thus bypassing all kinds of valuable experience that would most likely caught our newbie mistakes.

Bottom line - Don’t underestimate the worth of your testers and their experience, it might just come back to bite you in the future.

Labels: QA, QC, software testing, testing

The very exciting ICSE 2008 (International Conference on Software Engineering) is going to begin in the beautiful Leipzig, Germany. Luckily my paper was accepted at the conference and scheduled in the Telecom Experience Track.

The paper is an abstract of my thesis work done at Queen's University which relates to apply model checking techniques on real industry scale software.

Model checking has been claimed to provide comprehensive coverage to the system it verifies. However, there were few critical studies of the application of model checking to industrial scale software systems by people other than the model checker's own authors. In the paper, I reported my experience in applying the SPIN model checker to the validation of the failover protocols of a commerical telecommunication system, WebArrow.

In my experiment, I used SPIN model checker as the tool as it is the most heavily optimized model checker in the field. The methodology I used as follows:
1. Summarize the failover protocol with UML activity digrams
2. Build Promela (the input language of SPIN model checker) models
3. Check for deadlock
4. Express properties in LTL (Linear Temporal Logic)
5. Verify LTL properties in the SPIN
6. Optimize and simplify the model
7. Debug model using counter-examples
8. Record results

The key study results are that I found there to be a significant gap between the promise of model checking and the reality. Rather than enjoying fully automated checking of properties versus declarative system models, I battled problems of tracebility in the model checker. In addition, the time to create and check the models was incompatible with realities of time-to-market demands. Furthermore, the creation of the model itself took a big chunk of effort as well.

From the results, I can see that there are still some areas in the model checking community that the model checker authors can improve.

Labels: QA