If an important part of your business involves developing mobile apps it’s quite likely you have a box of old mobile phones and tablets lying around so you can test your code for performance and compatibility on different devices and platforms. For most startups who don’t have the luxury of free hardware from manufacturers, the contents of that box may seem more at home in a museum than in a cutting edge technology company.
Emulators don’t always give the same results as real physical devices, and compatibility testing has never been a task that testers relish- particularly when the devices they are doing it on aren’t the latest shiny iPhones or Galaxy Edges. For many, test automation remains a recurring task on the to do list, and this often leaves compatibility testing at the bottom of the priority list. Setting up the devices by hand is fiddly; manual regression (checking that nothing has broken between software versions) is time consuming; and covering the multitude of permissions variants, operating system versions and constantly evolving hardware could be a never-ending task.
AWS Device Farm
Amazon Web Services have come up with a solution that lets you throw away your box of old phones, stops your test team pestering you to splash out on the latest iPad and finally allows you to make some serious inroads into ticking the box of “test automation”.
AWS Device Farm is a service that allows you to run your apps on a wide selection of mobile devices that sit in Amazon’s data centres. Yes, you read that right – Amazon has taken their enormous box of mobile phones and tablets and has plugged them into the cloud!
How it Works
The services lets you run tests for Android, iOS or HTML5 web apps on your choice of around 200 handset/OS version/mobile carrier combinations. You can either choose Remote Access, which lets testers remote control the device through your browser or opt to do automated testing.
I watched Oliver Klein, AWS Senior Solution Architect, perform a live demonstration of the service at the AWS Enterprise Summit in Hong Kong last week. The whole process was remarkable easy to set up and is all configured through simple menu selections in the Amazon control panel. Once you’ve uploaded your application binaries all the hardware is set up for you, clean installed and any application dependences you have specified are pre-loaded onto the devices.
Remote Access is in beta and as you would expect with a remote control solution across the internet there is some lag as you control the screen but it looks like a great option for those hard to reproduce handset specific support problems, or a test cycle on the latest devices you can’t afford.
Test Automation With Little Effort
The most exciting feature, however is the test automation. If you have already developed your own set of automated tests in frameworks like Appium or Calabash you can run these against your selected devices. If you don’t use a framework you can still perform “fuzzy” testing, where a series of randomised swipes, touches and keyboard entries are generated for you. Automated fuzzy testing is a great way of covering a lot of code quickly with little preparation or script maintenance, so if you aren’t automating already this can potentially give you great value with very little effort or cost.
After your tests have run you can pick up error reports including screenshots, performance graphs and on Android devices you can also view a video of your tests as they ran. Any error logs and files generated on the devices can be picked up through the control panel.
Build Your Business Case
There’s no doubt that this is a great tool and has the potential to be a game changer for your test team, however at US$0.17 per device/minute, the costs could soon add up, so you need to think carefully about the scope of its use for your business. As with all testing there is a point where the costs outweigh the benefits, but the speed with which you can get basic automated fuzzy testing working in Device Farm means that you don’t have to spend very much at all to get those initial high value returns.
Steve Hodson is Founder and Principal Consultant at Runlevel 7, a management consultancy that provides business and technical advisory services to startups and growing businesses.