Automated UI tests on Android

I recently fought the Android emulator a lot to get my UI tests to run automatically during the build of Rabbit Escape, so I thought I’d better write down what I did before I forget.

I already have tests that drive the Android UI (see e.g. SmokeTest.java – it’s not pretty, but it seems reliable, and it catches real problems) – this post is about how to run them automatically during our build.

Note that to run the Android emulator I’m fairly sure you need a windowing environment, so I don’t think this could be moved to a headless build server. If course, you could always fight some kind of framebuffer thing.

Here is the part of our Makefile that launches the tests:

android-smoke-tests:
	@echo ". Running Android smoke tests"
	./build-scripts/android-start-emulator "android-8" "3.2in QVGA (ADP2)"
	./build-scripts/android-test "free" "app-free-debug"
	./build-scripts/android-test "" "app-paid-debug"
	./build-scripts/android-stop-emulator

and here is ./build-scripts/android-start-emulator – it starts up and emulator, waits for it to be ready, and unlocks its screen.:

#!/bin/bash

set -x
set -u
set -e

# Args

TARGET="$1"   # E.g. "android-8"
DEVICE="$2"   # E.g. "3.2in QVGA (ADP2)"

# Setup

ADB="${HOME}/Android/Sdk/platform-tools/adb"
EMULATOR="${HOME}/Android/Sdk/tools/emulator"
ANDROID="${HOME}/Android/Sdk/tools/android"
NAME="rabbitescape-${TARGET}"
TMP="/data/local/tmp"

${ANDROID} create avd \
    --force \
    --name "${NAME}" \
    --target "${TARGET}" \
    --abi "armeabi" \
    --device "${DEVICE}"

# Start the emulator
${EMULATOR} -avd "${NAME}" &

# Wait for the device to boot and unlock it
${ADB} wait-for-device shell < ${TMP}/zero
getprop dev.bootcomplete > ${TMP}/bootcomplete
while cmp ${TMP}/zero ${TMP}/bootcomplete; do
{
    echo -n "."
    sleep 1
    getprop dev.bootcomplete > ${TMP}/bootcomplete
}; done
echo "Booted."
exit
ENDSCRIPT

echo "Waiting 30 secs for us to be really booted"
sleep 30

echo "Unlocking screen"
${ADB} shell "input keyevent 82"

Now here is android-test – it launches the JUnit test code on the running emulator::

#!/bin/bash

set -x
set -u
set -e

PKGSUFFIX="$1"   # E.g. "free"
APKNAME="$2"     # E.g. "app-free-debug"

APPID="net.artificialworlds.rabbitescape${PKGSUFFIX}"
TESTAPPID="net.artificialworlds.rabbitescape${PKGSUFFIX}.test"
APK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}.apk"
TESTAPK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}-androidTest.apk"
ADB="${HOME}/Android/Sdk/platform-tools/adb"
DIR="/data/local/tmp/${APPID}"
TESTDIR="/data/local/tmp/${TESTAPPID}"

function run_test()
{
    TMPFILE=$(mktemp)

    ${ADB} shell am instrument \
        -w \
        -r \
        -e class "$1" \
        "${TESTAPPID}/android.test.InstrumentationTestRunner" \
    | tee ${TMPFILE}

    egrep "OK (.* tests?)" ${TMPFILE}
}

${ADB} push "${APK}" "${DIR}"
${ADB} push "${TESTAPK}" "${TESTDIR}"

${ADB} shell pm install -r "${DIR}"
${ADB} shell pm install -r "${TESTDIR}"

run_test rabbitescape.ui.android.DialogsTest
run_test rabbitescape.ui.android.SmokeTest
run_test rabbitescape.ui.android.TestAndroidConfigUpgradeTo1

And here is android-stop-emulator – it shuts down the emulator:

#!/bin/bash

set -x
set -u

echo -e "auth $(cat ~/.emulator_console_auth_token)\nkill" \
    | telnet localhost 5554

echo "Emulator stopped."

Submitting a package to F-Droid

Here’s what I needed to get a dev environment for F-Droid up and running on Ubuntu 16.10, using F-Droid server version 0.7.0 (commit id 8147f9235), so that I could submit a package for inclusion in the F-Droid repository.

Doing this is apparently the best way to get your own package into the repository, since you can provide a direct merge request for the metadata about your package, making it easy for the maintainers.

References:

Setup

Before you start, manually install the Android SDK at ~/Android/Sdk/ – see Download Android Studio. I installed version 23.0.2, but you will probably have a later one and may need to adjust the version number below.

Note: If you’re only planning to contribute a package I’m fairly certain you don’t need to install the Android SDK at all – you can just use the build server by running ./makebuildserver as I outline below.

Also before you start, if you want to contribute to the server project you should fork the F-Droid server project by going to gitlab.com/fdroid/fdroidserver and clicking Fork. When you’ve done that, the git clone command below will need to change to clone your own fork via SSH, instead of the HTTPS one cloning the main repo that is shown below. Do the same for the F-Droid data project, which holds the information about the packages in F-Droid. It’s the data project where you will want to make changes if you are submitting a package.

Run these commands:

# Prerequisites
sudo apt-get install openjdk-8-jdk subversion git git-svn mercurial bzr virtualbox ruby ruby-dev vagrant python3 python3-paramiko python3-pil python3-pyasn1-modules python3-clint
vagrant plugin install vagrant-cachier
ln -s ~/Android/Sdk/build-tools/23.0.2/aapt ~/Android/Sdk/platform-tools/

# Get the code
cd ~/code
git clone https://gitlab.com/fdroid/fdroidserver.git
git clone https://gitlab.com/fdroid/fdroiddata.git
echo 'export PATH="~/code/fdroidserver:$PATH"' >> ~/.profile
source ~/.profile

# Config
cd fdroiddata
cp ../fdroidserver/examples/config.py ./
chmod 0600 config.py
echo 'sdk_path = "$HOME/Android/Sdk"' >> config.py

# Set up Vagrant build box
cd ../fdroidserver
cp ./examples/makebuildserver.config.py ./
./makebuildserver
# Now wait several hours for this to finish

# Build a package (the F-Droid client) just to check it works
cd ../fdroiddata
mkdir repo
fdroid update --create-key
fdroid readmeta  # Should give no output if it worked
fdroid build --server org.fdroid.fdroid

Make your own package

Below I’m using my own package, Rabbit Escape, as an example. Its Android code is inside rabbit-escape-ui-android/app, whereas many programs will just have it directly in a directory called “app”.

Rabbit Escape also builds non-Android-specific Java and other things during its build, so your package may be simpler.

cd ../fdroiddata
fdroid import --url https://github.com/andybalaam/rabbit-escape --subdir rabbit-escape-ui-android/app

Now edit the new file that was created – in my case it was called metadata/net.artificialworlds.rabbitescape.txt.

I set the following info:

Provides:net.artificialworlds.rabbitescape
Categories:Games
License:GPLv2+
Author Name:Andy Balaam and the Rabbit Escape developers
Author Email:rabbitescape@artificialworlds.net
Web Site:http://artificialworlds.net/rabbit-escape
Source Code:https://github.com/andybalaam/rabbit-escape
Issue Tracker:https://github.com/andybalaam/rabbit-escape/issues

Name:Rabbit Escape
Summary:Lemmings-like puzzle/action game
Description:
140 levels of puzzling action!

blah blah blah
.

Repo Type:git
Repo:https://github.com/andybalaam/rabbit-escape
Binaries:https://github.com/andybalaam/rabbit-escape/releases/download/v%v/rabbit-escape-%v.apk

Build:0.10.1,101
    commit=v0.10.1
    subdir=rabbit-escape-ui-android/app
    gradle=paid
    build=cd ../.. && make android-pre

Auto Update Mode:Version v%v
Update Check Mode:Tags v\d+\.\d+(\.\d+)?
Current Version:0.10.1
Current Version Code:101

For more info, see the F-Droid manual.

And then checked it all worked with:

cd ../fdroiddata
fdroid lint net.artificialworlds.rabbitescape
fdroid readmeta
fdroid checkupdates net.artificialworlds.rabbitescape
fdroid rewritemeta net.artificialworlds.rabbitescape

When I got the version stuff right the checkupdates command printed:

INFO: Processing net.artificialworlds.rabbitescape...
INFO: ...updating to version 0.10.1 (101)
INFO: Finished.

Then I made sure it built OK:

fdroid build --server -v -l net.artificialworlds.rabbitescape

Actually, it didn’t work, and I decided I had to request a new package (sox) be installed in the build machine environment (in the fdroidserver project). The relevant commit is here: 19e372026. Actually though, after discussion with the F-Droid devs we agreed I’d be better off not using sox during the build, so I didn’t need this.

Side note: if you do end up needing to modify the build environment for F-Droid, make sure you delete the fdroiddata/buildserver directory when you re-try your build. That one had me stuck for a few days, with the old environment being used no matter what caches I cleared and vagrant commands I ran.

And now I was ready to request my package be included in F-Droid by committing and pushing the changes I had made to the fdroiddata project to my forked repo, and clicking the Merge Request button in the gitlab UI. My merge request is here: gitlab.com/fdroid/fdroiddata/merge_requests/1965

Code for detecting when you leave an Android app

Further to Detecting whether an Android app is stopping (or starting), I implemented code to decide when you are leaving or entering my game Rabbit Escape.

The relevant class is called Lifecycle2SoundEvents. (Yes, it’s a terrible name. Yes, I spent a long time trying to name it, and this is the best I came up with.)

And the tests, which are in TestLifecycle2SoundEvents, look like this:

@Test
public void Press_the_home_button_api10_causes_pause()
{
    Tester t = new Tester( activity1 );

    t.in.onSaveInstanceState( activity1 );
    t.in.onPause( activity1 );
    t.in.onStop( activity1 );

    // When we press home, we must at least pause (really we stop)
    t.assertPaused();
}

which I was reasonably pleased with, because they match my original blog post Order of Android Activity lifecycle events fairly well, without too much noise.

Detecting whether an Android app is stopping (or starting)

I am writing an Android app (called Rabbit Escape), and I want it to start playing music when the user enters the app, and stop when the user leaves.

Not as easy as it sounds because Android largely doesn’t think in apps, but Activities.

Update: real-life code for this is here: Code for detecting when you leave an Android app.

In my previous post Order of Android Activity lifecycle events I tracked the methods that get called on activities on my two test devices when the user does various actions. There was some variation between devices, but we can determine some useful rules.

First, let’s enumerate what we need to detect.

The user leaves by:

  • pressing “home” at any time
  • pressing the power button when the app is running
  • pressing “back” when in the first activity
  • launching an external activity (e.g. web browser)

The user enters by:

  • starting the app (e.g. from the home page, or the “running apps” panel)
  • pressing the power button when the app has been suspended
  • pressing “back” when in an external activity launched by us

From the previous post we can work out these rules:

  • When the user is entering, onResume is always called (on the activity that will be current)
  • When the user is leaving, onStop is always called (on the current activity), except when pressing the power button (to turn off) where you may only see an onPause call.
  • When the user is moving between activities, onResume is always called on the new activity, and onStop is always called on the old one. The onResume call is always before onStop

Thus, if we are OK with leaving ourselves running when the power button is pressed, we can detect leaving and entering the app using the code below.

If we need to stop something when the power button is pressed too, it seems we must be prepared to stop it when we receive onPause, and immediately restart it if we receive onResume, hoping that doesn’t cause a problem (e.g. a break in the music).

// All activities have these two methods:
@Override
public void onResume()
{
    super.onResume();
    LeavingOrEntering.activityResumed( this );
}

@Override
public void onStop()
{
    super.onStop();
    LeavingOrEntering.activityStopped( this );
}

// And we detect leaving and entering like this:
class LeavingOrEntering
{
    private static Activity currentActivity = null;

    public static void activityResumed( Activity activity )
    {
        if ( currentActivity == null )
        {
            // We were resumed and no-one else was running.
            notifyEntering() // Start the music!
        }
        currentActivity = activity;
    }

    public static void activityStopped( Activity activity )
    {
        assert currentActivity != null;

        if ( currentActivity == activity )  
        {
            // We were stopped and no-one else has been started.
            notifyLeaving(); // Stop the music!
        }
        currentActivity == null;
    }
}