Recently I worked on a project that had to support six different browsers; three versions of Firefox (1 through 3) and three versions of Internet Explorer (6, 7, 8). Right away this raised a few questions, and one of the more interesting ones was about testing:

What is the optimal environment for testing two browsers at three versions each?

This is an interesting issue because browsers are not designed to allow more than one version to be installed at once. This makes sense for users — it would be confusing and insecure for them to use an older version by accident — but it's traditionally a big hassle for web developers. It's important for us to be able to test our changes on multiple browsers because different versions of the same browser may not render the same code the same way, and it's important to catch and fix these bugs.

To answer this question, our team of myself (a developer) and two quality assurance experts did some brainstorming and each of us actually came up with our own solution. Here are the three options we considered:

Multiple VMs

This was our senior QA's suggestion. Set up three virtual machines, and on each machine, install a different version of Internet Explorer and Firefox.

Advantages:

• Tried, tested and true. This is a standard QA practice and has been in use for years.
• Can run all six browsers simultaneously (good if we want to test in parallel)

Disadvantages:

• There is some overhead involved in setting up and keeping track of three VMs.
• Three VMs will need to run on their own machine, so an extra box or some server space is needed.

VM Snapshots

This was our other QA's suggestion, which I thought was quite novel. The process goes like this: create a virtual machine and install IE6 and FF1, then take a snapshot (something the VM can revert to upon request). Now, upgrade the browsers to IE7/FF2 and take another snapshot, and perform the final upgrade to IE8/FF3 and take a third snapshot. By toggling between the three snapshots, all six browsers can be tested using only one VM.

Advantages:

• Only one VM is required, so there is less overhead.
• Since there is only one VM, it's possible to run the VM on a developer or QA's machine in a pinch.

Disadvantages:

• Can only test one version of each browser at a time.
• There is probably still a need for some shared machine space to host the VM.

Hack it Together

Predictably, this was the developer's solution (mine). I've used a slightly-unstable beta product called IETester in the past, which allows testing multiple versions of Internet Explorer simultaneously. This meant all I needed was a solution for Firefox, and after some searching I found an outline of how to install multiple versions of FF.

Advantages:

• No virtual machine overhead.
• Everything can be run as needed on the developer's machine.

Disadvantages:

• Highly technical: Setting up and maintaining this environment is not for the faint of heart.
• Can only run one version of FF at a time (though IE can be done in parallel).

Given these options, we decided that the best match for our project's needs was to use VM Snapshots on the quality assurance side, and the hacked-together solution on the development side. The low overhead was a big plus, especially for development, and it was important that QA was able to reliably test real versions of each browser. This worked out well for us, and I would definitely use such methods again.

Have you used these or similar solutions in the past? Are there other solutions out there that we don't know about? Let us know in the comments.

Labels: Browsers, Firefox, Internet Explorer, QA

# posted by Dan M @ 6:04 p.m. 172 comments