Adventures in Testing Gradle: A Cautionary Tale

My team recently had some down time right after a release. We decided to use our time trying to clean up as much of our tech debt as we could. We were trying to slay all of those little TODOs that had added up and do those minor refactoring projects that probably aren’t strictly necessary, but will make everyone happier in the long run. As a part of one of these, I found myself facing the great task of ‘clean up our gradle files.’ Yay! So for anyone who isn’t familiar, gradle is a build tool that allows you to declare in code what your build tasks are, what depends on what, etc. It’s super powerful and super handy. However, like any build tool, most people get their build working and then try not to touch it. We are most people. Each of us had learned just enough to modify the file in the ways that we needed and then promptly forgot everything and tried not to touch our build files again. In fact, one of my teammates jokingly (or maybe she wasn’t joking) told me that she was trying to get me to do all of the gradle tasks so that I could be our in-house expert. Needless to say that I was far from an expert when I finally found myself facing the task to make our gradle files beautiful. (As an aside, there’s actually a really good article by Doug Borg which I used for most of my inspiration.)

I jumped into my task with gusto. I started to untangle our gradle files and separate things into new files focused on singular aspects of our build process (e.g. put all of the things related to publishing together). As I was doing this, I realized that I was pulling out some of the exact same code into the exact same files in several of our repositories. To the engineer in me, this was a flashing red light. I decided I would do the right thing (after all, we’re fixing tech debt right now anyway, right?) and I would pull it out into a plugin. My very first gradle plugin!

Things started off not too badly, I got the plugin written. As it turned out, that was the last of the easy part. My plugin was in a new repository with it’s very own pipeline and, wait a minute, I had to build that. I found myself with gradle files to build my new repo and gradle files that were the content of my new repo. Okay, that’s slightly confusing, but nothing that I can’t wrap my head around. Now, like any good developer, I decided to write some tests for my new plugin. I looked around for some examples of how to do this. The first couple of plugins I happened to look at didn’t have tests. I naively judged the developers who made these choices (who checks something in with no tests!?). I eventually found a few test examples and tried to get started. That’s where everything started to get interesting.

So, as it turns out, if you want to test a gradle file, there are a few standard ways of doing it. At it’s heart, however, you essentially need to have some sort of test build.gradle file that is mimicking how your code might be used in a different repository’s actual build.gradle file. So now I have a build.gradle file to build my project and a fake build.gradle file for my tests. For both unit and functional tests, there is a gradle test kit that will help set up a temporary build.gradle file and write in whatever you want as defined in the test, so at least people before me have tried to make this as easy as possible.

Now, just to review, when we run ‘./gradlew build’ from the command line, it’s going to run the plugin’s build.gradle file in order to build the plugin. Then it’s going to run the plugin’s tests (still as a part of running the build file), which are going to run the build.gradle files that are created as a part of each test. So we have multiple build files getting run each time we build. Now that this is happening, when you suddenly get a random build failure, it’s not always obvious which build file the error is coming from and what’s actually happening. Be careful about where you’re adding the fix for any build failures! It won’t do anything in the wrong build file (not that I would know from experience or anything).

Okay, since that wasn’t messing with my head enough, I needed to do more. The plugin I was writing actually was a wrapper for the Jacoco plugin, which calculates code coverage. We have some common custom logic written on top of this plugin and consistent settings across all of our repositories, so it made sense to encapsulate this (thus my new plugin). From a high level, my plugin (as viewed by anyone using it) was calculating code coverage. Now if I want to test that I’m correctly doing this, I needed to find a way to actually calculate coverage. From a unit testing perspective, a lot of this can be handled by faking the report files and dealing only with those. That logic mostly lives within Jacoco, which they’ve already thoroughly tested, so I had no real need to test any of that. What I did want to test, to be reasonably sure that I had implemented everything correctly, was at least one happy path integration test (and maybe a few sad paths). Ok, so as a part of my test, I now need a project to go along with my build.gradle file and I need the tests to run on that project to make sure that the code coverage is calculated and aggregated. So now, I have a test project (complete with a build file and tests) living inside my plugin project. Then, when I build my plugin, I’m now running the plugin’s build file, which runs the plugin’s tests, which builds this test project’s build file, which runs the test project’s tests. If any part of this breaks, good luck figuring out where it’s coming from.

Unfortunately a few things were broken. I was running into a few errors specific to Jacoco. I began by trying my standard method of searching the internet. My only slight problem was that when I search for ‘testing the Jacoco plugin’ or ‘writing tests for Jacoco’, it quickly found things about using Jacoco on my tests. It turns out that searching for testing a framework that runs on tests was never going to find me anything. The internet had failed me. From here, I mostly just spent lots and lots of time trying to untangle exactly where my problems were coming from and reading lots of documentation to try to figure out enough about gradle to reason through what was going on.

In the end, I did manage to get my tests running and debug/get rid of all of my errors but not without a few scars and a lot of time spent on the process. Needless to say that I learned a ton about both gradle and the gradle test kit and now understand both much better than I ever expected too (although it still manages to surprise and stump me randomly). However, even with my new found knowledge, I will still think twice before assuming any future gradle changes will be easy. I conquered gradle this time but gradle definitely won a few rounds.