Make minitest/unit Great Again

My journey with testing started with Ruby. My first tests were on a Rails 3 application I was working on before starting SINAPTIA. Probably around 2011. It was a good experience and I learned a lot. The tests were messy, but they helped me understand testing better.

After that experience, I started looking at minitest/spec and RSpec looking for something more clever than the standard Test::Unit framework that came with Rails (it was still Test::Unit back then!). I used minitest/spec a lot until we started working with Alliants. In the project we started collaborating they had RSpec already set up. So I started using it and getting comfortable with it.

Fast forward to 2024. If you haven’t already, read our previous post for context.

The situation described in our previous post had an impact on me. I started asking myself what was wrong with our test suite, and if there was a better way for junior devs to approach testing. Not only that, it was hard for someone more experienced to understand tests quickly.

What’s wrong with RSpec

I want to be clear. There’s nothing intrinsically wrong with RSpec, it’s a nice framework and it’s widely used. However, the way it’s designed allows you to over-engineer your tests so you can write more tests with less code. Nested contexts, shared examples, let and let!, are all accomplices of this crime: one-sentence-tests.

In RSpec is common to find tests like this:

RSpec.describe InvoicesController do
  describe "#create" do
    context "when the user is not logged in" do
      # tests
    end

    context "when the user is logged in" do
      let(:user) { create :user, role: role }

      before do
        sign_in user
      end

      context "and it's admin" do
        let(:role) { "admin" }

        # tests
      end

      context "and it's finance" do
        let(:role) { "finance" }

        # tests
      end

      context "and it's customer" do
        let(:role) { "customer" }

        # tests
      end
    end
  end
end

As you can see, tests are read beautifully. But understanding what a single test does means scanning the whole suite looking for breadcrumbs.

Back to the basics

After the situation described in our previous post happened, I started being critical not only of our usage of RSpec but testing in general.

What’s the purpose of testing?

This is the first question we need to ask ourselves. What do we want to achieve with the tests we write? To me, it’s confidence. Confidence that the code we’re producing works as expected. Confidence that the feature we’re adding to a project is not breaking other features.

An important aspect of testing is maintainability because a test suite difficult to work with will harm confidence: to sustain a high level of confidence, tests must be easy to change, fix, and debug. Applications evolve and their test suites must evolve along with them. When tests get harder to change and fix, the team’s productivity drops, and rather sooner than later people will tend to test the minimum possible, or avoid testing at all.

So what can we do about it? Good old Four-phase tests to the rescue!

Four-Phase test

The Four-Phase test is a testing pattern. It consists of structuring each test with four distinct parts executed in sequence. The idea is to make what we’re testing obvious. The four phases are:

1. Setup: We prepare the subject under test and anything we need to observe the desired outcome of the test.
2. Exercise: We interact with the subject under test.
3. Verify: We do whatever is necessary to determine whether the expected outcome has been obtained.
4. Teardown: We put everything back into the state we found it.

The Four-Phase test is a great pattern for people interested in testing for the first time, but also for people who like things simple. Using Rails’ sugar, the most purist approach to the Four-Phase test would be to write a test like this:

class InvoicesControllerTest < ActionDispatch::IntegrationTest
  test "admin user can create invoices" do
    # setup
    @user = create :user, role: :finance
    invoices_count = @user.invoices.count

    # exercise
    post invoices_path, params: {invoice: {}} # assuming we're sending the right params

    # verify
    assert_equal invoices_count + 1, @user.invoices.count

    # teardown
    @user.destroy # assuming this also destroys associated invoices
  end
end

Let’s compare this pattern to the code above. Where’s the setup? where’s the exercise, and where’s the verification? Well, they are all over the place. Setting up a test in the “and it’s admin” context, means scanning up to the parent contexts and describes looking for before blocks, if the before block uses a variable not defined in it then you must scan for a let, and if there’s a let! it will be always executed. Then, in the actual test, a variable could be referenced that has never been defined, so you need to scan the file again for said variable.

In this purist example, we have one method doing the four phases, very easy to read. Readability improves a lot, even if tests get longer. However, it’s too purist for my taste.

In minitest/unit, you can share common setup and teardown methods, and add as many test_ methods as you want (which act as exercise and verify). Moreover, Rails adds a small layer on top of Minitest which helps reverting your changes to the database. Similar to the example above, in minitest/unit (using Rails’ sugar) we could write:

class InvoicesControllerTest < ActionDispatch::IntegrationTest
  setup do
    @user = create :user
  end

  test "can't create an invoice if the user is not logged in" do
    # the actual test
  end

  test "admin user can create invoices" do
    # This line acts as part of the setup. One could argue that it's confusing, but having just a bit of setup in the test is not bad.
    @user.role = :admin
    # the actual test
  end

  test "finance user can create invoices" do
    @user.role = :finance
    # the actual test
  end

  # and so on...
end

As you can see, we are testing the same things as in the RSpec example but more direct, and less layered. It’s easier to understand, and maintain and encourages new contributors to write more tests for the features they add, which is always desirable.

Final thoughts

At the end of the day, it doesn’t matter which testing framework you choose or if you choose not to test at all (though we don’t encourage this). What matters is that you are confident about the code you put in production, and that can be accomplished in many ways and with many frameworks. What I will certainly try to do is encourage people to write tests in the simplest way possible and I think that can be achieved much easier with minitest/unit.

But that’s not the end of the story. Not all projects are subject to change the testing stack, or suddenly change the testing culture of a team. We’ve been through these situations too. That doesn’t mean there’s nothing to do to improve your test suite’s and team’s testing culture. Want to know what we did? Stay tuned for next week’s post where we’ll talk about strategies you can apply to your rspec test suite to avoid running into this kind of issues in the long run.