Mae West once said, “ You only live once, but if you do it right, once is enough.”
Why are we quoting an American actress from the 1930s in an article about Automated mobile app testing? Hold on. We are getting there.
In this Age of Apps, with an estimated 2 million apps on the App store and 2.9 million apps on the Google Play store, you don’t get a second shot at getting your customer’s attention.
So Mae West’s quote can be rephrased as “Your App only gets one chance, and you have to do it right because once is all you get .”If your app freezes or crashes, the number of customers who will give it a second chance drops to 79%, and, if you or your app is having a bad day and it crashes again, the number of repeat users drops down to a whopping 16%, according to this Compuware study.
Talk about the pressure of getting it right the first time. And this pressure is not just for the developers, but a group of people who always work in the shadows and ensure that the app is running smoothly before it goes into production — the testing team. The testers are also known as the ninjas in the world of software development, (one of our testers added this last bit)!
Although testing is seen as overhead by the customer and an annoying part of the development process by the developers, like taxes or traffic fines, testing is a necessity and very much a part of our lives. So deal with it.
Software testing has been around for a few years, but mobile app testing is a relatively new kid on the block and has picked up steam only in the past five years.
In this article, we will look at:
- Mobile app development — Waterfall vs. Agile model and where to use automated testing
- Limitations of manual mobile app testing
- Automated mobile app testing — Why it is the way to go
- A beautiful story involving chairs to help you understand when to use Automated testing (yup, chairs)
The Evolution of Mobile App Development:
Waterfall model
The Waterfall model for software development had a lot of takers early on in the industry, and, by the time smartphones infiltrated into our daily lives and changed the way we interact with each other, the Waterfall model was deeply entrenched in the development psyche.
For complete newbies into this space, a Waterfall model represents a form of Software Development Life Cycle where development flows smoothly, much like a waterfall.
So we have the entire development phase split into well-defined stages, and the project managers have a large amount of control over who is doing what task at which point of time.
For instance, there is the Requirement Gathering stage, the Design stage, the Coding stage, the Integration stage, the Testing stage, the Installation Stage, and finally, the Maintenance stage.
While this method of development gave a lot of structure to the process, in the case of mobile app development, it came with its own set of challenges.
For starters, the needs of the customer keep changing when it comes to a mobile app, and this is a complete no-no in case of the waterfall model. When you need your product to have additional functionality, like a “, Submit a review” button that you thought of at the last moment, the Waterfall model fails, and you can’t have the developer go back and change the code in an earlier stage.
Another issue with the waterfall model is that you can only see the completed mobile app in the end, and if the customer is unhappy with the output, the PM has no choice but to begin from ground zero.
As mobile apps grow more and more complex, the waterfall model is slowly beginning to fall apart, to be quickly replaced by Agile software development.
Agile Methodology — Faster building times, faster shipping times
In Agile Software development, the entire development stage of the app is split into chunks called sprints, and the customer takes part in all the sprint meetings of the team, held every one or two weeks.
With this method, the development and testing happen in tandem, which means a lot of time saved for the customer. You write a piece of code, test it, show it to the client, gain overall feedback, and bam, you are done. You can now move to the next sprint.
Agile also meant that testing was now a collective responsibility, with not just the QA but the developers also sharing responsibility if the code is broken. In the agile model, finding bugs is easier, and since there is better communication between the tester and the developer, complex issues are nipped at the bud.
Since teams now work in sprints, incorporating changes to the product is easier in the Agile method. Customers are in the loop on how the development progresses, and there are no surprises when the final product is shipped.
How Automated testing fits perfectly with Agile methodology:
With Agile, the build times are fast, and that means rapid deployment, which calls for faster testing methods.
For instance, imagine you are building a game such as Candy Crush Saga.
You need to ship new levels out every week, and if you don’t, your gamers are going to move out to the next copy cat game that promises an additional 15 levels.
Your development team pushes out the levels in time, and you need to try now and find the bugs in the new level before the end of the week.
This is where Automated Test cases come to the rescue.
With automated regressive testing, your team can test the new level for all possible bugs, while making sure the previous levels of the game are still stable.
Here’s a quick checklist on when you should use automated testing and when to go manual
For Repetitive Features — Go with automated testing
For Multiple Devices — Automated testing beats manual testing
For Simple Apps — Manual testing because you need fewer resources
A Few Limitations in Manual Testing of a Mobile App
While we have seen how Automated tests make the whole development process more manageable, adding fuel to the fire are the obvious limitations of manual testing a mobile app.
According to Steve McConnell’s book, Code Complete, there is an average of “15–50 errors per 1000 lines of derived code.” For a sophisticated mobile app with thousands of lines of code, this translates to a software tester’s worst nightmare.
Here are a few more reasons manual testing breaks in case of a mobile app:
- Human error: With manual testing, you cannot produce a test report that says the app or the feature is 100 % bug-free. It is, so to speak, humanly impossible. Head out to this post where we show you a few more challenges associated with manual testing in the agile world.
- Time-consuming: Writing a test case and executing it over and over again takes up a lot of time for the testers, valuable time that can be used testing other features or writing new test cases.
- Repetitive and cumbersome: Testing is not exactly the most exciting part of the App development lifecycle, and sooner or later, the testers are bound to feel bored running the same test cases over and over again. This further leads to higher chances of human error creeping into the testing process, making it a vicious cycle.
- Increased costs: With all the points mentioned above, it is evident that manual testing of a mobile app is not exactly the best solution when developing the app in tandem. A final nail to this coffin is that manual testing involves hiring actual human testers, who are susceptible to all of the pitfalls mentioned above. Automated testing also involves a bit of initial investment in the form of tools to perform the test, but, in the long run, for a sophisticated mobile app, an automated test provides higher ROI (Return on Investment).
Automated Mobile App Testing — Overcoming the Challenges of Manual Testing
We now know that manual testing falls flat on its face when tasked with testing a complex mobile app, thanks to all the reasons mentioned above.
The way out? Automated tests. Automating the testing of your mobile application helps you overcome challenges, such as:
Performance Testing
Testing whether your application performs the way it was intended when a set number of users all use it at the same time is one of the most crucial tests you can perform on your mobile app. Automated testing helps speed up this aspect of the testing life cycle.
Consider this for an instant, a 5-minute downtime in 2013 at Google cost the company an estimated $545,000, according to this report by Coralogix.
So we can’t stress how important it is to have your app up and running all the time, and the way to ensure that your app meets the speed, stability, and scalability requirements are by subjecting it to rigorous performance tests.
Automating the performance tests means you can have your test cases run simultaneously from a large number of devices, something which is practically impossible if you are running a manual test.
Automating the performance test also means you can run Endurance tests ( ability to handle the expected load over a long period of time), Spike tests (handling unexpected spikes in the number of simultaneous users), Data volume, and Load tests relatively in a simple manner.
Functional Testing:
Functional testing helps to check if the app functions according to the design (duh!). In a functional test, all that you need to know is if the app is working the way you intend it to work.
In the real world, customers are not just running your app, but 15 different apps at the same time, expecting all of them to function without a glitch.
So you have a customer who is not just playing Candy Crush Saga but is checking his/her Instagram feed in between, texting a friend over WhatsApp, and attending important calls, all at the same time. Your app needs to stand the test of all this, and more.
With functional testing, you are not going into the deep layers of the app, such as the source code, but are just testing how user-friendly the app is.
Developers need a benchmark to test how every function, a “developer-free test,” if you may. Agile development means running tests on the features every day, and this, in turn, means automation.
So functional tests include:
- Smoke testing: Testing the functionality of the software build. This form of testing helps to catch early bugs in the app
- Sanity testing: Once you have the entire software build, which includes bugs that you may have found in earlier testing phases, you conduct a sanity test to see that the bugs you fixed did not lead to any additional issues.
- Regression testing: As explained in the earlier part of this article, Regression testing ensures that any new feature added in the app does not affect the previous functions/ cause the app to crash.
- User Acceptance Testing: UAT’s make the user more confident about the final product and help the engineering team minimize the risk post the implementation of the product.
Chairs and Automated mobile app testing — Are you sitting this one out?
We have been using big words and things have been getting a bit too technical. Let’s take a deep breath, step back for a moment. Now, try and understand why you need to automate your mobile app testing with an analogy.
We looked around the office to find inspiration to write this one, and we found one that every office goer or remote worker has in his/her workplace, chairs.
Imagine you are a manufacturer of chairs. The chairs can be anything ranging from black cushion ones with excellent lumbar support that makes you feel like you are sitting on a cloud or a simple plastic chair that is a sure-fire way to mess up your spine. It doesn’t matter.
All that matters is that you own a factory that makes chairs. Lots and lots of them. This is similar to developing an app that has a lot of lines of codes and different functions.
Your little chair factory makes a thousand chairs a day. Now you know that for every 100 chairs you make, two chairs are defective. A broken leg, back support that is messed up, etc.
This defect represents a bug in the app, a bug that your testing team is supposed to catch.
Now inspecting all the thousand chairs thoroughly to find the few defective chairs might be cumbersome, but can you automate this process?
Can you, for instance, buy a shiny new machine with laser technology that scans each chair in a matter of seconds and tells you if it is defect-free or not?
If your answer is yes, then it probably makes sense to automate it, depending on how established your factory is and if you can invest in a fancy laser machine that inspects chairs.
Again, this depends on the number of chairs that you are producing. If you are making eighty chairs a day, it doesn’t make financial sense to invest in a machine, since inspecting eighty chairs can be done by a couple of factory workers.
This is the same case with Automated testing. Investing in a complex automated testing framework that requires specialized knowledge to implement might not make a lot of sense if you are building a simple app with only a few features that you can easily test manually.
The Rolls Royce of chairs?
If you are a bit more niche player in the field, say building the Rolls Royce of chairs, you don’t need an automated chair testing machine.
This is because if you are an experienced carpenter, there is a high probability that the chairs you produce will be defect-free. Like a Rolls Royce.
But the probability that your developer is an expert carpenter is very rare (not literally, we are just connecting the analogy), and more often than not, you need to test your app for bugs.
If you are a player in the middle segment of chairs, which means you produce premium quality chairs that are mostly defect-free, then automation becomes a necessity. This is similar to an app that is used by more than 1000 users and needs to function flawlessly.
With a mid-range chair, there is an element of exclusivity, and when people pay for your chair (or app), they expect it to be defect-free. So you cannot afford to risk your reputation as a top chair builder (or an app developer) by hiring a few manual testers when the testing needs to be automated.
So we know that automated tests are the way to go if you have a big project with deep pockets and the resources to invest in the long run. But this does not mean that automated testing will completely replace manual testing of mobile apps.
Challenges of Automated Testing
(Yes, they do exist but no company will reveal them)
Automated app testing also comes with a small set of challenges that can only be overcome by manual testing. These include:
- Physical interfaces: How often a person uses the home button while using the app, the force someone uses to press the volume button on his/her mobile, the frequency of switching screens, these are attributes that are difficult to program into a script. In such cases, manual testing still needs to be done.
- Complex tests: Automating a testing cycle requires writing elaborate scripts, that span multiple platforms and take up a lot of effort to code. Writing “hermetic tests” is considered as a test automation best practice. Breaking up a complex test into simpler scripts help re-usability and minimize maintenance, and is often preferred to writing an entire end-to-end system-level automated test.
- Early adoption: Even with Agile development, for a small app or a product that is early in the development stage, you want the app to reach a small number of people so that you can get some constructive feedback. In the ideation/ growth stage, there are going to be a considerable number of changes, which need to be implemented and then tested cyclically. Automated testing is expensive, and you may want to think twice before investing during this phase of the development cycle since it can easily double your engineering costs.
- Cross-App testing: Switching between apps is one of the most challenging tasks in a test automation framework, and is best done manually. An app that requires user input from two different stakeholders, for instance, a buyer and a seller, is a nightmare to script into a testing framework.
- Subjective Evaluation: How good does an app look, the aesthetics, and the “feel” of switching between screens, these are some of the things that software cannot tell, and requires a real human to test.
Despite these minor drawbacks, automated testing is still probably the best solution if you are planning to build a scalable, responsive app using the Agile model, and we would highly recommend automating your mobile app’s testing.
