A Brief Introduction to Device Farm

Device Farm is an app testing service that allows testing and interacting with physical, and virtual mobile phones and tablets. Device Farm currently supports iOS, Android and web apps, with desktop support planned for the near future.

While there's support for remotely accessing your android device under-test through the "Remote access" tab; I'll be focusing on "Automated tests".
Interactive testing is found within the "Remote access" tab.

Device Farm provides support for Appium for Android apps. After uploading your test, Device Farm allows test execution in parallel across a collection of user specified devices.

Python-Appium Installation

To install the appium python package run:

pip install Appium-Python-Client

Inspecting elements inside of your android app with the Appium Server Desktop Client allows the team to Inspect Session, and locate elements with strategies including ID, ClassName and so on.

Example Appium Tests

Tests are written against your application under test, so there isn't a cookie cutter option. Instead, here's examples of sanity tests that we prioritized writing to quickly assess confidence of releases.

Platform: Android
App: app-video-debug.apk

Test case 1 : test_skip

  • Skip the shake phone for feedback alert.
  • Verify the skip button is present.
  • Validate the functionality of the skip button.

Test case 2: test_watch

  • Validate the visibility of Watch button of nav bar.
  • Verify the watch button redirects to Watch page.
  • Validate the Shows image displays in Watch page

Test case 3: test_explore

  • Verify the explore button is visible in the navigation bar.
  • Verify the watch button redirects to explore page.
  • Verify the top article list heading is displayed in the Explore page.

Test case 4: test_account

  • Validate the visibility of the Account button in the nav bar.
  • Validate the watch button redirects to the Account page.
  • Verify the visual design of the About header in the Account page.

Build the Test Package

The test package must be uploaded to Device Farm in .zip format, and must contain all required dependencies.

  • Setup and install pytest in a virtual environment.
    virtualenv workspace
    cd workspace
    source bin/activate
    pip install pytest
  • Create your tests inside of the tests/ folder.
    ─ workspace
    └─ tests/ (tests go here)
  • Generate the requirements.txt and wheelhouse to store dependencies. In our tests, there are three packages required:
    - Appium-python-client
    - selenium
    - pytest
  • Create a list of your projects installed packages.
    - pip freeze > requirements.txt
  • Create a wheel, the new standard of Python distribution which has several advantages.
    - pip wheel --wheel-dir wheelhouse -r requirements.txt
  • Zip the folders and requirements.text.
    - zip -r test_bundle.zip tests/ wheelhouse/ requirements.txt.
  • Upload test_bundle.zip file.
This graph shows the current adoption level of the top 360 most-downloaded packages on PyPI.

Setup Device Farm

To set up an automated tests in Device Farm, we need to sign in to the AWS console and create the project for our tests.

  • Create a new run in the new project of Device Farm.
  • Choose Native android app and upload the .apk file.
  • Choose Appium Python to configure the test.
  • Upload the packaged Appium test scripts build to the test run.
  • Start run.

Pros of Appium on Device Farm

  • Many Android, iOS, and Fire OS devices are available.
  • High-level results, low-level logs and screenshots are recorded.
  • Recorded videos are available for each test case.
  • Remote controlling devices is an ideal replacement for owning devices.

Cons of Appium on Device Farm

  • You are required to upload an app apk and to configure the settings manually each time, which is quite expensive.
  • Scheduling tasks is not allowed which prohibits scheduled test execution.
  • There's no functionality provided to store/restore caches from previous builds.
  • Tests can’t be separated to run serially or when the function is called; the entire test suite must be ran across all devices each run.
To run 4 tests it takes around 10 minutes on each device; painfully slow.
  • Test complexity is greatly increased with new models, deprecation of older models is frequent, and there's no way to see what their roadmap is for device acquisition.
  • The report is not informative, it only give us the tests that passed/failed and the logs. Custom reporting had to be implemented to provide more utility.
  • There is no way to add test steps to have a step level report.
  • Integration with JIRA or other services at the reporting level seems impossible.

FAQs

Q: What does Fuzz do?

A: Fuzz will perform fuzz testing on your UI immediately after launching the application under test. It streams random user input (touches, swipes, keyboard input) in a rapid fashion to your app. You can configure the number of events, the delay between events, and the seed used to randomize events. Using the same seed across test runs will result in the same sequence of events.

Q: My app requires dependencies to test all functionality. Can I install other apps?

A: Yes, you can select multiple apps and the order in which to install them. These dependent apps will be installed before your tests begin.