Example Android project with repeatable tests running inside an emulator

I’ve spent the last couple of days fighting the Android command line to set up a simple project that can run automated tests inside an emulator reliably and repeatably.

To make the tests reliable and independent from anything else on my machine, I wanted to store the Android SDK and AVD files in a local directory.

To do this I had to define a lot of inter-related environment variables, and wrap the tools in scripts that ensure they run with the right flags and settings.

The end result of this work is here: gitlab.com/andybalaam/android-skeleton

You need all the utility scripts included in that repo for it to work, but some highlights include:

The environment variables that I source in every script, scripts/paths:

PROJECT_ROOT=$(dirname $(dirname $(realpath ${BASH_SOURCE[${#BASH_SOURCE[@]} - 1]})))
export ANDROID_SDK_ROOT="${PROJECT_ROOT}/android_sdk"
export ANDROID_SDK_HOME="${ANDROID_SDK_ROOT}"
export ANDROID_EMULATOR_HOME="${ANDROID_SDK_ROOT}/emulator-home"
export ANDROID_AVD_HOME="${ANDROID_EMULATOR_HOME}/avd"

Creation of a local.properties file that tells Gradle and Android Studio where the SDK is, by running something like this:

echo "# File created automatically - changes will be overwritten!" > local.properties
echo "sdk.dir=${ANDROID_SDK_ROOT}" >> local.properties

The wrapper scripts for Android tools e.g. scripts/sdkmanager:

#!/bin/bash

set -e
set -u

source scripts/paths

"${ANDROID_SDK_ROOT}/tools/bin/sdkmanager" \
    "--sdk_root=${ANDROID_SDK_ROOT}" \
    "$@"

The wrapper for avdmanager is particularly interesting since it seems we need to override where it thinks the tools directory is for it to work properly – scripts/avdmanager:

#!/bin/bash

set -e
set -u

source scripts/paths

# Set toolsdir to include "bin/" since avdmanager seems to go 2 dirs up
# from that to find the SDK root?
AVDMANAGER_OPTS="-Dcom.android.sdkmanager.toolsdir=${ANDROID_SDK_ROOT}/tools/bin/" \
    "${ANDROID_SDK_ROOT}/tools/bin/avdmanager" "$@"

An installation script that must be run once before using the project scripts/install-android-tools:

#!/bin/bash

set -e
set -u
set -x

source scripts/paths

mkdir -p "${ANDROID_SDK_ROOT}"
mkdir -p "${ANDROID_AVD_HOME}"
mkdir -p "${ANDROID_EMULATOR_HOME}"

# Download sdkmanager, avdmanager etc.
cd "${ANDROID_SDK_ROOT}"
test -f commandlinetools-*.zip || \
    wget -q 'https://dl.google.com/android/repository/commandlinetools-linux-6200805_latest.zip'
unzip -q -u commandlinetools-*.zip
cd ..

# Ask sdkmanager to update itself
./scripts/sdkmanager --update

# Install the emulator and tools
yes | ./scripts/sdkmanager --install 'emulator' 'platform-tools'

# Platforms
./scripts/sdkmanager --install 'platforms;android-21'
./scripts/sdkmanager --install 'platforms;android-29'

# Install system images for our oldest and newest supported API versions
yes | ./scripts/sdkmanager --install 'system-images;android-21;default;x86_64'
yes | ./scripts/sdkmanager --install 'system-images;android-29;default;x86_64'

# Create AVDs to run the system images
echo no | ./scripts/avdmanager -v \
    create avd \
    -f \
    -n "avd-21" \
    -k "system-images;android-21;default;x86_64" \
    -p ${ANDROID_SDK_ROOT}/avds/avd-21
echo no | ./scripts/avdmanager -v \
    create avd \
    -f \
    -n "avd-29" \
    -k "system-images;android-29;default;x86_64" \
    -p ${ANDROID_SDK_ROOT}/avds/avd-29

Please do contribute to the project if you know easier ways to do this stuff.

Rabbit Escape 0.12 out now, with water

The newest feature of Rabbit Escape, water, has been brewing a long time, but we now think it’s ready:

Water can flow, it can put out fires, and it can drown rabbits.

Rabbots seem to be immune though…

Check out the 20 new levels we have released! (This makes a total of 180 levels.)

It works on Windows, Linux, Mac OS and Android.

Rabbit Escape 0.11 out now!

The RABBOTS are coming!

Get the latest version of Rabbit Escape:

Make Android Gradle display unit test failure messages

By default, Gradle does not show you what happened when a unit test failed:

$ ./gradlew test
...
MyTest > Black_is_white FAILED
    org.junit.ComparisonFailure at MyTest.java:6
    ^^^ WHAT ACTUALLY FAILED????
...

This is insane, and can be fixed (thanks to mrhaki) by editing build.gradle to add:

// NOTE: this is the non-Android solution - add to build.gradle
test {
    testLogging {
        exceptionFormat = 'full'
    }
}

The above doesn’t work with Android, but something similar does:

// Android solution: add this to app/build.gradle
android.testOptions.unitTests.all {
    testLogging {
        exceptionFormat = "full"
    }
}

And sanity prevails:

$ ./gradlew test
...
MyTest > Black_is_white FAILED
    org.junit.ComparisonFailure:
    expected:<[black]> but was:<[white]>
        at org.junit.Assert.assertEquals(Assert.java:115)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at MyTest.Black_is_white(MyTest.java:6)
...

Files for plain Gradle:

$ cat build.gradle
apply plugin: 'java'

repositories {
    mavenCentral()
}
     
dependencies {
    testCompile 'junit:junit:[4,)'
}

test {
    testLogging {
        exceptionFormat = 'full'
    }
}

$ cat src/test/java/MyTest.java
public class MyTest
{
    @org.junit.Test
    public void Black_is_white()
    {
        org.junit.Assert.assertEquals("black", "white");
    }
}

Files for Android+Gradle

$ cat build.gradle
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.3.1'
    }
}

allprojects {
    repositories {
        jcenter()
    }
}

$ cat app/src/main/AndroidManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="net.artificialworlds.testapp">
</manifest>

$ cat app/src/test/java/MyTest.java
public class MyTest
{
    @org.junit.Test
    public void Black_is_white()
    {
        org.junit.Assert.assertEquals("black", "white");
    }
}

$ cat build.gradle
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.3.1'
    }
}

allprojects {
    repositories {
        jcenter()
    }
}

$ cat local.properties
sdk.dir=/home/andy/Android/Sdk

$ cat settings.gradle
include ':app'

Automated UI tests on Android

I recently fought the Android emulator a lot to get my UI tests to run automatically during the build of Rabbit Escape, so I thought I’d better write down what I did before I forget.

I already have tests that drive the Android UI (see e.g. SmokeTest.java – it’s not pretty, but it seems reliable, and it catches real problems) – this post is about how to run them automatically during our build.

Note that to run the Android emulator I’m fairly sure you need a windowing environment, so I don’t think this could be moved to a headless build server. If course, you could always fight some kind of framebuffer thing.

Here is the part of our Makefile that launches the tests:

android-smoke-tests:
	@echo ". Running Android smoke tests"
	./build-scripts/android-start-emulator "android-8" "3.2in QVGA (ADP2)"
	./build-scripts/android-test "free" "app-free-debug"
	./build-scripts/android-test "" "app-paid-debug"
	./build-scripts/android-stop-emulator

and here is ./build-scripts/android-start-emulator – it starts up and emulator, waits for it to be ready, and unlocks its screen.:

#!/bin/bash

set -x
set -u
set -e

# Args

TARGET="$1"   # E.g. "android-8"
DEVICE="$2"   # E.g. "3.2in QVGA (ADP2)"

# Setup

ADB="${HOME}/Android/Sdk/platform-tools/adb"
EMULATOR="${HOME}/Android/Sdk/tools/emulator"
ANDROID="${HOME}/Android/Sdk/tools/android"
NAME="rabbitescape-${TARGET}"
TMP="/data/local/tmp"

${ANDROID} create avd \
    --force \
    --name "${NAME}" \
    --target "${TARGET}" \
    --abi "armeabi" \
    --device "${DEVICE}"

# Start the emulator
${EMULATOR} -avd "${NAME}" &

# Wait for the device to boot and unlock it
${ADB} wait-for-device shell < ${TMP}/zero
getprop dev.bootcomplete > ${TMP}/bootcomplete
while cmp ${TMP}/zero ${TMP}/bootcomplete; do
{
    echo -n "."
    sleep 1
    getprop dev.bootcomplete > ${TMP}/bootcomplete
}; done
echo "Booted."
exit
ENDSCRIPT

echo "Waiting 30 secs for us to be really booted"
sleep 30

echo "Unlocking screen"
${ADB} shell "input keyevent 82"

Now here is android-test – it launches the JUnit test code on the running emulator::

#!/bin/bash

set -x
set -u
set -e

PKGSUFFIX="$1"   # E.g. "free"
APKNAME="$2"     # E.g. "app-free-debug"

APPID="net.artificialworlds.rabbitescape${PKGSUFFIX}"
TESTAPPID="net.artificialworlds.rabbitescape${PKGSUFFIX}.test"
APK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}.apk"
TESTAPK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}-androidTest.apk"
ADB="${HOME}/Android/Sdk/platform-tools/adb"
DIR="/data/local/tmp/${APPID}"
TESTDIR="/data/local/tmp/${TESTAPPID}"

function run_test()
{
    TMPFILE=$(mktemp)

    ${ADB} shell am instrument \
        -w \
        -r \
        -e class "$1" \
        "${TESTAPPID}/android.test.InstrumentationTestRunner" \
    | tee ${TMPFILE}

    egrep "OK (.* tests?)" ${TMPFILE}
}

${ADB} push "${APK}" "${DIR}"
${ADB} push "${TESTAPK}" "${TESTDIR}"

${ADB} shell pm install -r "${DIR}"
${ADB} shell pm install -r "${TESTDIR}"

run_test rabbitescape.ui.android.DialogsTest
run_test rabbitescape.ui.android.SmokeTest
run_test rabbitescape.ui.android.TestAndroidConfigUpgradeTo1

And here is android-stop-emulator – it shuts down the emulator:

#!/bin/bash

set -x
set -u

echo -e "auth $(cat ~/.emulator_console_auth_token)\nkill" \
    | telnet localhost 5554

echo "Emulator stopped."