Running Firebase Test Lab on Travis CI

Firebase Test Lab is a service that allows you to run your Espresso tests directly on real hardware devices in the cloud. This allows to test on various API levels and hardware without the hassle of maintaining a device wall and keeping these devices charged all the time.

In this post, we'll show how to integrate Test Lab with travis CI so you can automatically run the tests on each git push.

Firebase configuration

Go to https://console.firebase.google.com/ and create an account. As this is a paid service, you will have to upgrade your plan. You can do a dedicated project for testing so you can control your quotas and billing better. The price as I'm speaking is $5 per device hour.

Create a service account for travis to access the project.

  • Go to https://console.cloud.google.com/iam-admin/serviceaccounts/ and select your project.
  • Click 'Create Service Account' at the top.
  • Choose a name and select 'Editor' for 'Role'.
  • Click the 'Furnish a new private key' checkbox (choose JSON for the key type). This will download a json file on your computer. Keep it preciously, you will not be able to download it again.

Encrypting the firebase credentials for travis

Using the Travis client, encrypt your secret json file:

travis encrypt-file secret.json

Beware travis can only encrypt one file so if you already have encrypted files in your repo, you need to make a tar before calling travis encrypt-file

You will end up with a secret.json.enc file, commit this to your repository.

Travis configuration

after_success:
- openssl aes-256-cbc -K $encrypted_78f5c5933ac8_key -iv $encrypted_78f5c5933ac8_iv -in secret.json.enc -out secret.json -d
- wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-127.0.0-linux-x86_64.tar.gz
- tar xf google-cloud-sdk-127.0.0-linux-x86_64.tar.gz
- echo "y" | ./google-cloud-sdk/bin/gcloud components update beta
- ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file secret.json
- ./google-cloud-sdk/bin/gcloud beta test android run --async --type instrumentation --app ./app/build/output/apk/app-debug.apk --test ./app/build/output/apk/app-debug-androidTest.apk --device-ids Nexus5 --os-version-ids 22 --locales fr --orientations portrait --project project-id

The first line decrypts the secret.

The next 3 lines to get the latest version of the gcloud tool. Note that we have to manually download the beta components too: echo "y" | ./google-cloud-sdk/bin/gcloud components update beta.

Then we give gcloud access to the Google Cloud project.

Finally, the actual launching of the tests:

  • --apk is your apk under test
  • --test is your test apk
  • --device-ids is the actual hardware (or vms) you want your tests to run on. You can get a list with gcloud beta test android devices list
  • --os-version-ids is the api levels you want to test
  • --locales the locales you want to test
  • --orientations well... you get it
  • --project is your project id that you can get at https://console.cloud.google.com/home/dashboard

Success!

If you go the the Firebase console, you will see your test running.

A few minutes later your test results are available with the corresponding logs and videos. Now there's no more excuse not to write all these nice espresso tests !