I wanted:
- Code coverage to run reliably against the instrumented tests
- Tests to be isolated from each other
I'm new to Android development, but experienced with other development and tooling.
So I looked for the documented options and added the following to my build.gradle:
android {
defaultConfig {testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArguments clearPackageData: 'true'
testInstrumentationRunnerArguments useTestStorageService: 'true'
}
buildTypes {
debug {
testCoverageEnabled true
}
}
I also have a custom task to generate the report:
task coverageReport(type: JacocoReport) {
sourceDirectories.setFrom(fileTree(
dir: "$project.buildDir/src/main/java"
))
classDirectories.setFrom(fileTree(
dir: "$project.buildDir/build/intermediates/javac/debug",
))
executionData fileTree(dir: project.buildDir, includes: [
"outputs/code_coverage/debugAndroidTest/connected/*/*.ec"
])
}
Now what I see when running the tests and generating the report, is that about 10% of the time it hits what looks like a corrupted .ec file. (the rest of the time, the report is fine)
> ./gradlew coverageReport
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:coverageReport'.
> Unable to read execution data file app/build/outputs/code_coverage/debugAndroidTest/connected/Nexus_S_API_33(AVD) - 13/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
Digging into the stack trace:
Caused by: : Unable to read execution data file app/build/outputs/code_coverage/debugAndroidTest/connected/Nexus_S_API_33(AVD) - 13/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
at org.jacoco.ant.ReportTask.loadExecutionData(ReportTask.java:518)
...
Caused by: java.io.IOException: Unknown block type 3.
at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:119)
at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:93)
at org.jacoco.core.tools.ExecFileLoader.load(ExecFileLoader.java:60)
at org.jacoco.ant.ReportTask.loadExecutionData(ReportTask.java:516)
I've seen several other numbers in other projects for the unknown block type, but so far only 3 for this minimal example project.
MD5 sums are the same for the .ec files on and off the device:
> adb shell md5sum /data/media/0/googletest/internal_use/data/data/com.example.coverage/coverage_data/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
89c6b7ac8ad27d4d2f19dce0ac47f028 /data/media/0/googletest/internal_use/data/data/com.example.coverage/coverage_data/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
> md5sum app/build/outputs/code_coverage/debugAndroidTest/connected/Nexus_S_API_33\(AVD\)\ -\ 13/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
89c6b7ac8ad27d4d2f19dce0ac47f028 app/build/outputs/code_coverage/debugAndroidTest/connected/Nexus_S_API_33(AVD) - 13/com.example.coverage.ExampleInstrumentedTest#useAppContext.ec
Re-running the tests will generally clear the issue on this project. On a larger project, several tests fail like this on each run - different tests each time, though apparently some more than others. It doesn't appear to be linked to the running time of the test.
I tried different versions of:
- Gradle (7.4, 7.5, 8.0-rc-1, 8.0-rc-2)
- AGP (7.4, 8.1.0-alpha01)
- Android Studio (Eel, Giraffe)
- emulator SDK version (27, 31, 33) with the same failing on each.
Syncing with Gradle files and cleaning/rebuilding don't help, and I've seen this happen on the first run of a new project.
I can see that:
- Gradle support for Android lags (the new JaCoCo aggregate report plugin is explicitly not supported on Android)
- Gradle's had and fixed some similar issues with JaCoCo in recent releases (e.g. https://github.com/gradle/gradle/issues/20532)
- a similar issue's been seen with JaCoCo in other contexts (e.g. JaCoco MVN unknown block type which suggests the JVM's being hard-killed between tests)
Turning off clearPackageData doesn't stop these failures.
I was expecting code coverage of instrumented tests to be a core tool in my development process. So far it looks like either this error is a "me" problem, or Android devs don't see much value in having this working reliably. Any insights into why it's not that valuable or can be easily replaced would be as welcome as a fix.
