The Process of Evaluating a Mobile Automation Framework into Applico’s Testing Processes

Testing is one of the most important aspects in the software development life cycle (SDLC) in every organization. At Applico we test throughout the SDLC to improve reliability, performance and other critical factors, which are defined under the requirement specs before the product is shipped to the customer. The QA team is continuously testing throughout the course of the development stage to improve the quality of the product by discovering bugs early in the software, and therefore, regression testing is extremely important. However, regression testing can become very time consuming and costly, so placing an automation framework into Applico’s testing process is the perfect solution to speed things up and deliver bug free applications.

Mobile app testing presents many unique challenges. Native and web apps require different testing approaches, including real network conditions, targeting all of the available devices in the market, and scripting which is the method of defining a test. Native apps are most common as they are arguably the fastest and most reliable, and provide the most responsive experience to users. On the other hand, web apps are limited in what they can do effectively in terms of features and always require a network connection to work. This proves to show that web apps can be slow at times and are less intuitive than native apps, which can interfere with automation scripts.

The QA team took into consideration several important testing features while evaluating the best mobile automation framework. Compatibility with both iOS & Android, a familiar scripting language for the testers, a robust recording functionality, low cost, shallow learning curve, web and native support, and Jenkins and AppThwack integration were the main factors for this evaluation. We then researched the most popular and commonly used tools to compare them amongst each other.

AppThwack provides a full device cloud service for testing both iOS and Android web apps. By simply uploading a completed script, AppThwack allows us to test among all the available iPhone and Android devices, along with older OS versions and under all network conditions (3G, 4G, LTE etc.). A granular report displays if any errors are present in the build, characterized by severity. Jenkins, also an open source tool, is a continuous integration tool written in Java. This commonly used process allows all development work to be integrated at a predefined time when someone pushes new code into the source repository, and is then automatically tested and built, allowing for identification of development errors early in the process. Following this verification process, the latest version of the codebase is released and can be tested on real devices.

MonkeyTalk was the most feasible and proper tool for Applico’s testing needs after a careful and lengthy evaluation using the matrix below. As an open source tool, MonkeyTalk can test apps running on real hardware, can easily connect to iOS or Android simulators, and has an outstanding recording and playback feature that makes it easy for the user to create scripts. Reporting is also made simple, allowing the tester to quickly identify and debug any errors while integrating CSV files with scripts. Most importantly, MonkeyTalk offers integration with AppThwack and Jenkins, which are two common tools used in our current testing process. To evaluate which automation offering best suits your needs, please refer to our evaluation grid below.

Becoming familiar with MonkeyTalk was not an informal process as we initially expected. There is little documentation available and integrating CSV files into the script can prove to be a steep learning curve for first timers with a limited programming background. With some practice with MonkeyTalk on an existing Applico client app, Disney’s TomorrowLand, we identified that touch/swipe gestures were not fully recognized when we played back the script unless the tester modifies the script with timing delays. In addition, the component tree/view hierarchy is not really useful for a QA tester who is not familiar with all the application labels inserted by a developer, thus requiring more effort to identify label names using Xcode.

Monkeytalk Scripts integrate with CSV files for lengthy test runs

With the use of MonkeyTalk, the Applico QA team successfully automated the TomorrowLand app by integrating a CSV file within our script, providing both positive and negative test scenarios. The recorded user actions can be shown in three different views; Table View, Monkey Talk and JavaScript, allowing the user to practice compiling scripts with any of the three languages preferred. Easily integrating our MonkeyTalk scripts into AppThwack allows us to test multiple iOS and Android devices under real world network conditions. Applico’s QA team is proud to incorporate MonkeyTalk automation framework into our testing process, automating every app and shipping bug-free apps to the Apple App Store and Google Play store.