Overview

Firebase Test Lab is a cloud-based infrastructure-as-a-service that offers virtual devices for test automation.

Alternatives

Evaluation Criteria

  • Plan flexibility Vs. alternatives.
  • Plan options & cost.
  • Team requirements Vs. Firebase's offering.
  • Scalability & migration.

Plan Flexibility Vs. Alternatives

Product Free Option Metered Plan Real Devices Dedicated Devices Live Testing Appium Support Parallel Execution
Firebase Test Lab x x x x
Browserstack x x x
AWS Device Farm x x

Plan Options & Cost

Spark (free)

  • 10 tests/day (virtual device)
  • 5 tests/day (real device)

Flame ($25/mo)

  • 10 tests/day (virtual device)
  • 5 tests/day (real device)

The cost covers additional cloud capabilities not related to testing. Flame is the same as Spark in terms of mobile offering.

Blaze (metered)

  • $1/device/hour (virtual device)
  • $5/device/hour (real device)

Team requirements Vs. Firebase offering

Requirement Included/Supported by Firebase
Live testing x
Parallel test execution (~10) x
Real devices ✓*
Unlimited executions x
Scripted Android tests
Scripted iOS tests ✓**
CI Support for Android
CI Support for iOS x
Test screenshot for Android
Test screenshot for iOS x
Most recent Android device / OS version Galaxy S9+ / Android 8
Most recent iOS device / OS version iPhone X / iOS 11.2

* Limited count of tests in real devices (5 tests per day)
** Does not include support for AI-driven tests

Scalability & Migration

Having the test suites housed within the app repository, using the native language and UI framework comes with the advantage that, in the future, the remote device service can be swapped for any other alternative with relatively minimum rework as long as the following criteria are met:

  • The alternative service provides support for app native language (Java and Swift in our use case).
  • The alternative service provides support for the framework(s) implemented in the test suites.

The current situation with alternative services is as follows:

Browserstack

Espresso is supported for android UI tests. Source.

XUITest is supported for iOS tests. Source

AWS Device Farm

While AWS Device Farm supports XCUITest for both Obj-C and Swift, it does not provide support for Android Espresso. Source: 1/2

Saucelabs

Saucelabs has support for both Android Espresso and iOS XUITest frameworks. Source.

Conclusion

  • Firebase's offering is useful immediately at a very small cost.
  • The $25 plan does not increase the limit on device usage, however this is removed when the metered plan is subscribed to.
  • Robo tests automatically cover a large portion of functional tests across all our offerings, across a wide range of android devices.
  • This is a good entry point to Browserstack or AWS device farm down the road.
  • Lowest cost / highest impact.
  • Where others are thousands, this is hundreds or less a year.