Stan’s comment on my last post, "Using UAT as your only testing…is just a poor practice," got me thinking about some the more memorable experiences I’ve had through the years related to User Acceptance Testing. This is the first of several stories. The story you’re about to read is true. The names have been changed to protect the innocent. I’ll call the first user I trained…
Although I was primarily a developer a few years into my career, I was always a strong advocate of Quality Assurance. One of my clients who had previously always "tested in production" decided that they should start testing earlier. Because I had been suggesting to the client that they hire a Quality Assurance specialist, they asked me to train and mentor their users to fill that role in a 3-month rotating basis. This wasn’t quite what I had in mind when I made the recommendation, but this was at least an improvement compared to what they had been doing.
Ms. Penny was the first user who I trained. In addition to her full-time job, she had a part-time job outside of the company. She was an elected public official. She was smart and competent in both of her jobs, but her part-time job required a couple of skills that, in retrospect, were not conducive to a first-time User Acceptance Tester: she was comfortable communicating directly with people in high-level positions, and she thrived on being the center of attention.
I gave Ms. Penny a brief overview on what to look for when testing the newest software changes and asked her to send me an e-mail with any bugs that she found. I would then verify the bugs, enter them into a spreadsheet, and communicate them to the developers. It was a rudimentary process, but like I said, it was an improvement from what they had.
The Sky is Falling
The next day, my manger was at my desk asking me why I didn’t tell him about the major issues that Ms. Penny was reporting. I said that I hadn’t heard anything yet.
It turned out that the first time that Ms. Penny found an error, she first reported it to the director of her department and explained that if the error went live, it could cost the company millions of dollars. The director was quite upset with the possibility of losing millions and chewed out my manager. My manager and I met with Ms. Penny and her director to explain that the purpose of testing was to find errors and correct them before they reached production. We asked Ms. Penny to first report any errors to me before getting the director involved. The director seemed to understand the process and was a little calmer.
The Sky is Falling – Part Two
After our meeting, Ms. Penny started sending bug reports to me, which I recorded and passed on to the developers. However, after a couple of days, my manager was at my desk again asking why I was ignoring Ms. Penny’s urgent bugs. I explained that I wasn’t ignoring them and that we were diligently working on them.
It turned out that when the bugs were not immediately fixed, Ms. Penny went to the director again and told him that the development team was ignoring her reports. The director was furious that we would ignore such important issues and chewed out my manager again. We again met with Ms. Penny and her director to explain that we weren’t ignoring her bug reports. In fact, we were following the process which we had all agreed to. Ms. Penny just needed to understand that it took some time to correct the defects and get a new build to her. The director definitely understood the process this time and seemed rather irritated with Ms. Penny.
Ms. Penny’s director realized that the development team was truly tracking and fixing the bugs reported by Ms. Penny. He told her that she was to work directly with me and only escalate any issues if I specifically said that we were planning to go live with bugs that she felt would pose a risk to the company. Our working relationship improved dramatically, and the quality of our software improved even though our testing methodology was less than ideal.
I took some lessons away from that experience that helped me work with users during User Acceptance Testing on future projects:
- Create a clear communication plan that includes specific conditions for escalation of issues to management and clear paths for that escalation.
- Before testing begins, coach users and all other people on the escalation path about what to expect: errors are normal and are nothing to panic about.
- Track UAT bugs in a public database so the users can easily see the current status of the bugs and progress made towards fixing them.
- Clearly define exit criteria for UAT including who must sign off in order to go live.