Early on in any project, I work with the project team to define what browser/operating system (OS) combinations the project will support. I try my best to get the team to limit the number of browser/OS combinations supported because each supported combination adds to the testing and development effort.
For software that’s designed for internal use in a company, it is often easier to limit the number of supported browser/OS combinations. Companies typically try to limit the number of supported combinations so their IT department can efficiently support everyone. If someone has problems with the new software because they are violating company standards, the company can dictate that their machine be updated to the standards.
However, for software designed for public use, there can be a seemingly endless list of combinations that could be supported. Typically, when I ask a client which combinations they would like to support on a public website, their first reaction is "all of them." This first reaction makes sense because nobody wants to lose a potential customer just because the customer is using an old or unusual browser/OS combination. However, when I explain the extra cost of testing and development, the client has always agreed to narrow down the number of combinations.
Narrowing Down the List
The first thing that I do to narrow down the list of supported browser/OS combination is to get data showing how many hits the company’s existing site had for each combination over the past 3 months. I’ve been fortunate that every client I’ve worked with has been able to provide this data. If my client didn’t have this data available, I’d find out the statistics at a more global level at a site such as StatCounter. However, I prefer to see the data for the specific company because there may be something unusual about the demographic of their customers that skews the usage.
At my last client, the list of operating systems that people used to hit the client’s site included Nintendo Wii, Sony PlayStation, and Microsoft Xbox. Unfortunately, I could not convince the client that we needed to support these operating systems, which would have required that we purchase these machines for the test lab.
Once we have hard data of recent usage, the business can make an educated decision about what browser/OS combinations they want to support. Of course, they should keep in mind any trends in usage over time and any new browsers that are being released. Supporting an old browser that has rapidly declining usage may not be worthwhile even if it currently has a decent market share. Similarly, if Microsoft has a new browser available for beta testing, it may be worthwhile to support that browser. So far, I haven’t worked with a client who wanted to take on the extra cost of testing and developing for mobile devices. However, based on the trend of mobile usage statistics, I’m sure that my clients will be looking into testing on these devices soon. This could require testing on Blackberry, iPhone, Android, and Windows Mobile, and may even require testing on different physical devices within each of these operating systems.
IE6 – To Test or Not to Test?
Developers who I work with dread when we have to support Internet Explorer 6 (IE6) as part of our project. They rightly point out that IE6 has security issues that are not problems in IE7 or IE8. In addition, the developers know that that code that works in IE8, IE7, Firefox, Chrome, Safari, and Opera often has problems in IE6. It’s also difficult to test IE6 because you cannot have multiple versions of Internet Explorer on a single machine. Supporting IE6 is a pain for the developers to fix and an added expense for the business.
However, when I run the usage statistics at most clients, it turns out that a significant percentage of users of public websites still use IE6. According to StatCounter, more than 8% of browser usage worldwide over the past year has been IE6. This is too big a percentage for most businesses to ignore.
Because testing IE6 can be a pain, in my next couple of posts, I’ll share some useful, free and easy-to-use tools that I’ve found for testing IE6.
I’d like to know what other methods people have used to determine what browser/OS combinations to test. How have you determined which ones to test?
Have you ever been involved with a project that said that they supported a browser/OS combination but chose not to actually test one or more of the supported combinations before going live? If so, how did it turn out?