Lessons Learned in Teaching QA

Image source: Google
(Title is inspired by Lessons Learned in Software Testing, a book by Bret Pettichord, Cem Kaner, and James Marcus Bach)

Back in December 2009, I started my career as a Jnr. Quality Assurance Engineer and my role was focused on testing the applications that we were enhancing along with the maintenance of QA processes, environments and servers. It’s been almost 8 years and my role has evolved into a strategy maker but implementing QA activities is still the most favorite part of my work.

In January 2016, I was asked to teach a course on Software Quality Assurance to Computer Science undergrads and since then, I have taught three batches at two different universities. During this time, I have worked on revamping the course outline and have added some workshop facilitation techniques in our usual lecture style classes. This blog is documenting my journey to find strategies that work or don’t work while teaching this course and how I made sure to deliver some value to my students.

Usually we are given a course outline that talks about the course objectives including topics, the course book and any reference book (s), if required. We are also given a calendar of classes and exams that will be taken during the semester.

Following is the official course outline for SQA taken from Higher Education Commission’s official website that applies to all the CS undergrad programs. You can see the complete document here.

Course Name: Software Quality Assurance

Credit Hours: 3

Prerequisites: Software Engineering

Course Outline:

What Is Software Quality: Quality Assurance, Quality Engineering

Software Testing: Testing: Concepts, Issues, and Techniques, Test Activities, Management, and Automation, Coverage and Usage Testing Based on Checklists and Partitions, Input Domain Partitioning and Boundary Testing, Coverage and Usage Testing Based on Finite-State Machines and Markov Chains, Control Flow, Data Dependency, and Interaction Testing, Testing Techniques: Adaptation, Specialization, and Integration.

Quality Assurance beyond Testing: Defect Prevention and Process Improvement, Software Inspection, Formal Verification, Fault Tolerance and Failure Containment, Comparing Quality Assurance Techniques and Activities.

Quantifiable Quality Improvement: Feedback Loop and Activities for Quantifiable Quality Improvement, Quality Models and Measurements, Defect Classification and Analysis.

Risk Identification for Quantifiable Quality Improvement, Software Reliability Engineering.

Reference Materials:

1. Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, Jeff Tian, Wiley-IEEE Computer Society Press, 1 st Edition, 2005(or Latest Edition).

2. “Mastering Software Quality Assurance: Best Practices, Tools and Techniques for Software Developers”, Murali Chemuturi, J. Ross Publishing, 2010 (or Latest Edition)

After seeing the above outline, I sat down with one of my teammates and we talked about how disproportionately theoretical it looks and the books mentioned are very one dimensional. All the classes were supposed to happen lecture style and with all this theory, there wouldn’t be any time left for learning anything practical. More recent methodologies were missing from it and there was no room for different schools of thought and comparative studies. We also saw the outline for its prerequisite course and it was full of various document templates and traditional Software Engineering Practices. So we concluded the following:

• We need to add content about Exploratory Testing, Automation Practices, Agile Testing Quadrants, Cross Browser and Cross Platform Testing Techniques, A/B Testing and Monitoring.
• We also decided to add more diverse reading material for students so we made a list of books and blogs that we thought they must read.
• We also decided to add Continuous Delivery and DevOps in the content along with an overview of a Test Management Tool (JIRA) and an Automation Tool (Selenium). Bonus points for doing JMeter.
• Also, Testing will be taught by doing, so we will pick an industry project for students so that they can learn to hunt and report bugs.
• Assignments will include creating Test Plans, Test Cases and Test Scripts. Also, a study on formal methods and their relevancy to the current tech industry will be included.

Pretty ambitious, eh? I know!

I tried to divide all the content into 15 weeks of teaching, 3 hours every week and I realized that I was in trouble. So I made a list of topics that were “must to teach” (just like “must to haves” in our Product Backlogs).

My list looked something like this:

• Quality, Software Quality and it’s a factors, Software Quality Control and Assurance.
• SDLC, STLC and Bug Life Cycle, Bug Reporting and Bug Advocacy, Test Cases, Field Validation Matrices.
• Test Planning, Writing a Test Plan, Software Testing Types, Levels and Approaches.
• Heuristics for Exploratory Testing, Alphabets of Testing, Agile Testing Quadrants.
• Code Coverage, Code Refactoring, Testing Automation and Heuristics for Stop Testing.
• Cost of Quality, Defect Prevention, Defect Detection Techniques.
• Fagan Inspection, Software Inspection, Formal/Informal Reviews.
• Gold Plating, Scope Creep and Change Management, Continuous Integration, Continuous Delivery and DevOps.
• Software Failure Mechanisms, Software Reliability Engineering, Models for Quality Assessments.
• Defect Analysis, Data Integrity, CRC, A/B Testing and CBCT.
• Tool(s) Overview.

I selected the following books as reference reads:

• Lessons Learned in Software Testing by Bret Pettichord, Cem Kaner,and James Marcus Bach
• Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design by James A. Whittaker, Alan Page
• Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin, Janet Gregory
• Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing by Rex Black
• Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement by Jeff Tian

In our first class, my first slide looked like this:

So here comes the lessons that I have learned during the three semesters that I have spent teaching this course. I am learning new things every day and I will try and keep this list up-to-date!

Lesson # 1: Context gives meaning to everything!

Context can be defined as “the formal or informal setting in which a situation occurs; it can include many aspects or dimensions, such as environment, social activity, goals or tasks of groups and individuals; time (year/month/day), etc.” Brown (2010). As we started discussing certain practices and their uses in certain scenarios, I realized that students think more critically when I ingrained in them that without context we cannot determine the validity and efficiency of any solution. They started talking about hypothetical situations and did in-depth discussions about the factors affecting those situations along with the strategies that can be considered to solve the chaos.

Lesson # 2: Visual Aid tools make things fun and easy to learn!

We used mind maps and flow diagrams everywhere. Be it illustrating an application’s features or deciding how to run a strategy meeting, we used visual aids and took visual notes too. I directed them to check the mind maps made by Santhosh Tuppad and his team and they loved browsing through their work. Because we were doing these activities in class we used good old pen and paper to create visual aid.

Lesson # 3: Testing by doing is the way to go!

We picked up products (one each semester) from the local incubators (The NEST I/O) and entrepreneurs, we strategized the testing efforts and went ahead with it. Students were divided into multiple team and were assigned team leads. They took a week’s time to explore and test the products (without any specifications). They filled in bug reports and compared the product with its competitors, wrote their analysis and recommendations and in the end, we shared them with the development team by inviting them to our class. This whole exercise showed them how testing can generate value for a business and the goal is not to undermine or find faults with anyone, but rather improve the quality of the product.

Lesson # 4: Comparing different schools of thought/approaches helps in seeing things critically.

We talked about different schools of thought and used a number of diverse books throughout the weeks which helped with understanding how complex real world problems can be and a person’s ability to understand and evaluate possibilities is a skill that requires rigorous learning and diverse experience. We also checked some definitions and templates used by the practitioners and standards (ISO, CMMI, etc.) and concluded that these templates can’t be used without adopting them to our project, process and people’s context.

Lesson # 5: Sometimes, chewing what you bit becomes impossible.

The changes that I made to the course outline added pressure for me and we had to skim through some of the content because of time shortage. We also couldn’t do a full on tutorial of the test automation tool. So I decided to give them the material and a week’s time to experiment with it. I wanted to call in a Test Manager to talk to the class but the schedule didn’t work for us two out of three times. I also ended up doing more unplanned work due to the complex nature of tasks and high student engagement. But all in all, it was all worth it!  

Lesson # 6: Working in a team is a key lesson!

We hit few problems during the testing assignment where some students were uncomfortable in working with a certain team lead or in a group without their friend(s). Sometimes I let them change their team, other times they had to put up with whatever was given. Another lesson was for team leads to review work done by others and take responsibility for their teams. They presented the findings to the development team after we were done with testing, reporting and sorting of all the information.

Lesson # 7: Humor is a great aid in keeping the class awake and interested!

They say that our sense of humor is connected to our sense of wonder. Students tend to retain better when humor is involved and they also participate in the discussions more. The point to note here is that humor by no means translate to vulgar language or jokes, rather topic related humor that either questions or comments on the concept. We had a strict policy against sexist and racist remarks, crude and sarcastic tone, and infighting.

Lesson # 8: CD and DevOps are geek-gasmic topics for tool buffs!

Deployment pipelines, automated test frameworks, infrastructure maintenance, monitoring and testing in production were well received by students. We also talked about tools for automating most of the error-prone and time consuming tasks. All of them were familiar with GitHub but after the discussion they actually went ahead with researching tools like Docker, Puppet and Chef.

Lesson # 9: Storytelling is an excellent teaching strategy!

There’s something about stories that stick with us and help us in retaining the lessons better. Stories engage our thinking, emotions, and imagination all at once. We used a lot of cases and stories to talk about the topics and I received help from so many fellow testers who shared their glorious successes and screw-ups with me and gave me permission to narrate them in my classes. I also shared stories about my old and current projects with them and a lot of times, the students amazed me by understanding the situation better than I did.

Lesson # 10: Talking about Risk is a must!

Testing requires asking a lot of questions and then trying to find the answers that fit. It also requires excellent listening skills and you need to listen to your client, your team and your C-level management and put the pieces together to create a picture that can help you test better. A lot of these questions and information will be about the things that can go in unexpected ways (that’s risk!) so doing a risk workshop is imperative. We did that before starting testing the product and made sure that we write them down along with the mitigations and contingencies. This activity introduced them to risk-based testing and its significance. Talking about the more risky features (for example, the ones where you are using some external services/APIs) and features that can’t be tested from a specific user persona/interface made them ask some really fascinating questions.

During one of the sessions we saw that the APK file that was given to us didn’t work on some phones and failed system consistency test. We made sure to check the OS version and phone configuration but we couldn’t figure out what was wrong. When we spoke to the development team about it, we got to know that they never installed the app on any phone other the only high-end phone that the team lead had and used it for testing, the app worked fine on it but didn’t work in one out of three cases when we tried to install it on our phones. In one case, we found out that the live app force the user to make a new account every time to try to use the app and that too with a new email account. So yeah, you got the point!

There are so many stories to tell from the classes but I guess I should end this post here. I am trying to document more of my experiences these days and I would love to listen to your suggestions and experiences. Please feel free to reach out by commenting here or tweeting at me!