Why Bother?

Assuming you want to do it properly, testing your site to ensure consistency across a range of browsers is a great deal of work, so why would you bother? Well, the difficultly with websites is that they need to be viewed on a large number of different browsers, on a large number of computers. Without proper cross-browser testing your site will not display consistently or without errors on your users’ computers. This is a bad look and not how you want to present yourself to your site’s customers/visitors.

What I won’t cover in this article, but I may cover another time

1. Testing with Javascript turned off: This is important for testing accessibility, but that’s not what we are concerned with today.
2. Mobile Browser, many of the points I cover will be somewhat applicable, but I won’t be covering mobile browsing specifically.
3. Complete end-to-end QA or functional testing, we are focusing specifically on ensuring a constant, quality experience between different browsers. It is assumed that site quality has already been tested against a reference browser, which is not Internet Explorer. I tend to use Firefox 3.5+, but any newish versions of Safari, Chrome, or Opera would work just as well. The important thing though, is to choose a baseline which has minimal quirks.
4. Usability studies, Eye-tracking studies, or any other specific Usability testing. I assume any Usability testing has already been completed against the reference browser before compatibility testing begins.

What Causes these Errors?

Different Browsers

The people that created the browsers are only human, and as such all had different views and opinions. In addition to this, even though there are internationally agreed on standards for building web pages (for example HTML is the main language used), there are still differences of interpretation. This leads to seemingly little discrepancies, that are actually very visible to your visitors.

For instance, the web standards don’t say a whole lot on what the default font size is for a web page. Consequently, the browser vendors all pick their own default size. You can end up with your web page displaying noticeably larger in one browser than another. Here’s a list of the most common browsers. You’ll definitely want to be testing IE, Firefox, and Safari. Ideally, you should also be testing Chrome and Opera but, when time is tight, those two frequently fall by the wayside.

• Internet Explorer (IE)
• Firefox
• Apple Safari
• Google Chrome
• Opera

Different Browser Versions

As progress is marching along, updated versions of the browsers are getting released, with many old errors fixed, a bunch of new features and, usually, a fistful of new errors to contend with. This means that IE 6 and 7 have a different set of bugs and need to be tested separately, as does 8 and 9. The same goes for Firefox 2 and 3, and Safari 3 and 4. Here’s a reasonably complete list of the most significant versions. Exactly which versions you spend the most time testing will depend on what your visitors are using. If you don’t know what your visitors are using, or haven’t launched your site yet, test as much as your can.

• IE 6, 7, 8
• Firefox 3, 3.5
• Safari 3, 4
• Chrome 3, 4
• Opera 10

Whether to Support IE 6 or Not

Internet Explorer 6 is the albatross around all of our necks. It’s now almost nine year old, has a large number of bugs and faults (serious faults, mind you, not minor annoyances), is very slow, and does not support any of the cool bits that all the new browsers do. It is where most of your time and budget you have for testing will be spent. Additionally, fixing IE6 bugs, once they are found, is also time consuming.

If it causes so many problems why do we still support it, especially considering its almost 9 years old? Because we are pragmatists — we have to be. Think of it this way, if you ran a store selling imported, handmade Italian scarfs (beautiful quality, feel the fine weave, why wouldn’t you want one?), then one day, for reasons unknown, 30% of customers started paying in Nepalese Rupees.

Your options are to turn away 30% of your business, or take the money and do the currency conversion yourself. It’s not ideal but at least you still come out ahead.

So if it’s that simple why did Google just drop support for IE6? And why is 37 Signals is phasing out support, and Facebook and Digg, among others, are asking users to upgrade? The question that the creators of each of those sites have asked themselves is: does the percentage of IE6 users make up a large enough lump of your total site traffic to justify the additional expense of maintaining IE6?

Most web stats tend to put current usage of IE6 at between 20% and 40%, but your site could be vastly different. Unless the site that you are testing is already live and you can see how many visitors are using IE6, then you’ll need to base your decision on traffic from similar sites.

• Microsoft: dropping support for IE6 is not an option
• Drop IE6 Support — Give People a Reason to Upgrade
• YouTube Will Be Next To Kiss IE6 Support Goodbye
• Gmail to drop IE6 support this year
• IE6 Death March

Different Operating Systems (OS)

Mostly the different browsers work pretty consistently between OSes, but one area where there are quite noticeable differences is in font rendering and availability. Each OS comes with a different set of font. For example, Mac OS X comes with a font called Helvetica, where as Windows comes with a similar looking font called Arial. Windows uses Arial because they didn’t want to pay to license Helvetica, and although they are close there are a number of visual differences that are not hard to spot.

The upshot of this is that if you specify your site should use Helvetica, Windows users will actually see Arial. There are a number of other fonts that are only available on one OS, but not the others.

By the by, if you’re interested in Type or design in general a really wonderful film was made about Helvetica in 2007. Not so coincidentally, 2007 was also Helvetica’s 50^th birthday!

Another difference that leads to a lack of visual consistency is fundamental, philosophical differences between how the OSes treat type rendering. These differences tend to mean that text on one OS can look more blurry than on another, will cause paragraphs to be laid out differently, or any number of other differences that will lead to your page looking subtly (or not so subtly) inconsistent on different computers. I’ll avoid going into anymore detail regarding this as it starts getting technical pretty quick, but if anyone is interested here’s a couple of links to get you started:

• Font Rendering: Respecting The Pixel Grid
• Font rendering philosophies of Windows & Mac OS X

Bottom Line: the main three ones you need to worry about are:

• Windows
• Mac OS
• Linux

Again, where your time is most spent will depend on what your visitors are using, but it’s a safe bet that you’ll want to spend the most time testing Windows and the least amount testing Linux.

Different Screen Sizes

This one is comparatively simple, usually the site is designed for a minimum screen resolution, currently either 800×600 or 1024×768 pixels are standard, but it depends on your site design. Just make sure that the site looks good (that nothing is cutoff or cropped) at your target resolution in each of the browsers and versions being tested, and that it scales nicely to higher resolutions.

Browser Bugs

As the programmers that created the browsers are only human, we also need to deal with bugs in the browsers themselves. They are unique to each browser and are usually obscure. They typically require almost arcane knowledge of the quirks of the specific browser.

Our biggest problem is IE6 and, to a much lesser degree IE7. IE6 is an enormously buggy browser, and its bugs tend to present in odd, sometimes logic-defying ways. Which is why it’s almost universally hated by web developers everywhere. IE6 bugs tend to be tricky to isolate, and even trickier to fix.

Expect to waste a lot of time on IE6.

A Quick Word on Browser Testing Tools

There are a bunch of nice tools out there to help with testing websites for cross-browser compatibility. A quick search on Google will turn up a swath of them, but one I’m quite fond of is Litmus. Litmus can take screenshots of your site in various Browsers, Browser Versions, and OSes. It won’t help with testing more complex logic, Javascript testing for instance, but it can greatly speed up layout testing.

Types of Errors. What are we Testing for?

When performing general site testing there’s quite a range of things that can go wrong, but when specifically doing cross-browser testing the types of errors we are looking for can be classified into one of two, rather lumpy categories: Display Errors and Logic Errors.

Display Errors

These are usually pretty easy to spot, anytime anyone in your team says anything like “Eh, why is the ad banner displaying over the login screen in IE?” they’ve found a display error. These can be embarrassing but are usually less critical than Logic Errors as they don’t usually prevent bits of the site from working, but they do make a bad impression.

Logic Errors

These are particularly problematic as, not only are they trickier to fix, but they also prevent parts of the site from working. This will likely impact the effectiveness of the site to your visitors. Which will in turn affect the effectiveness of the site to your organisation.

Examples of questions to ask yourself when testing for logic errors:

• Can a User access all necessary functionality?
• Is functionality working as expected?
• Does it respond sanely to invalid input (a bad password, for example)?

Knowing What to Test

The first thing that’s needed is a list of all possible user tasks and all the areas of the site they’ll need to complete those tasks, a site map is useful for the latter, a good understanding of your user roles is critical for the former. What you want to end up with is a list of what you want to test and how to find it on the site. I’ve included a simple example below.

What	Where	How
User Login Page	http://example.com/login	From the login form
Quick User Login	http://example.com	Use the quick user login form in the right column on the home page.
Add to Cart	From any product page	From the product details area, select a quantity and click the “Add to Cart” button.

Reporting Errors, or “Don’t Make Your Developers Want to kill You”

The thing with recording errors is to give enough detail that whoever has to fix the problem will be able to understand at a glance what happened, and then be able to reproduce it themselves. That means you need to include the following in your error report:

• What you were trying to do?
• The page (url) you were trying to do it on?
• Any error messages that were displayed on the screen.
• A screenshot of any errors messages, or visual artifacts that were visible as the error occurred.
• The steps that you used to trigger the error, in enough detail that someone can follow them.

When I say “in enough detail that someone can follow them” I actually mean at least this level of detail:

1. Type “rolly” into the login input box.
2. Type ‘123456’ into the password input box.
3. Click the “Login” button.

Without this level of detail it will be considerably more time-consuming to fix the bug, and your web developers will hate you for making their jobs unnecessarily difficult.

Actually Testing

Work methodically, trying possible combinations, sane data, insane data, recording whats been tested, recording errors.

When testing you must work methodically, and document your progress. As you test each example record what works, and record any errors. Sometimes all that you will need to do will be to glance at a page and make sure the layout matches up with your reference browser. In more complex cases, say a login form or any situation where the visitor needs to interact in a more complex fashion, you’ll need to dig in and actually complete the tasks your visitors would complete.

Using the example of a login form again, it’s important to realise that you need to dig deeper when mimicking your visitors tasks. It isn’t enough to just type a login and password, click the login button and, assuming it doesn’t explode, tick it off your list. You need to identify the major outcomes and test them all.

Here’s a more through example of login form test cases. A lot more could be added, but these cover the most critical cases:

What	How	Outcome
Sucessful User Login http://example.com/login	From the login form: Using username “bob” and password “bob”	The user should be redirected to their dashboard and a message should display saying “Welcome back”
Failed User Login Due to Invalid Password http://example.com/login	From the login form: Using username “bob” and password “invalidpassword”	The user should be redirected back to the login page, above the login form it should say “Your username or password was incorrect, please try again.”
Failed User Login Due to Invalid Username http://example.com/login	From the login form: Using username “invalidusername” and password “bob”	The user should be redirected back to the login page, above the login form it should say “Your username or password was incorrect, please try again.”
User can’t remember their login/password so they click the “Forgot Password” link near the login button http://example.com/login	Click the “Forgot Password” link	The user should be redirected to the “Reset Your Password” page.

Go Test Your Site Now

Even through I feel like I’ve been typing forever, a lot more could be said about this topic, not to mention related topics like Usability and Conversion Optimisation. Treat this as a basic primer into a much larger topic, now you know enough to know what questions to ask next.

Let me know how you get on, or if you have anymore questions.