By David Krammer in Tech —

Orchestration vs Transformation Code - Choosing the right Unit Testing strategy

Tests are important, I don't have to tell you that. And while TDD is trying to move us in a direction where we write meaningful tests that don't just get the coverage up, we have to take a step back and start thinking about our test approach before we write the tests at all. But what does that even mean?

You might say that your tests work totally fine and this is possibly true, however, is it a slog to write them? Are there dangers to writing them in a certain way? Can you even structure your production code better by thinking about how you write tests for it?
Let's answer that by at first looking at the problem that we want to solve: What kind of input data do we use to test our SUT? There are two different approaches

  1. Static Data
    Static data gives you the confidence that you can reliably reproduce a failed test. You hard-code the data in your test setup. If there is a different test case, you write another test. Easy. But what if your SUT is specifically tailored to that test data (willingly or not). Wouldn't it be cool if we could use
  2. Random Data
    This is something that libraries like AutoFixture or Bogus (both C#) made easily accessible and configurable. They generate random data for you that you sometimes have to tweak here and there in order to fit your use-case, but all in all you get different values each time you run the tests. It sounds nice at first, but it leads to two unwanted consequences. On the one hand, flaky tests. You maybe didn't think your test data generation 100% through and in every 200th or so run, you get a data set that will fail your test. But that is the smaller problem as you can fix the flaky test once you get knowledge of it. The much bigger fish here is that random data tempts the developer to replicate production code in the test in order to get the expected output data.

So, what do we do? Just use static data and be mindful to cover all test cases, maybe use multiple data sets for the same test case? Or do we always use random data and simply check that we generally calculate the expected output in a different way than the production code?
Or maybe both together? But if we do that, when do we use which? This is the time when you have to take the initially mentioned step back and ask yourself: What code am I writing here?

I would split code generally in two clusters: Orchestration Code & Transformation Code.

Orchestration Code

Orchestration code is code that does not do any calculation itself, but orchestrates the communication between different components.
Let's look at an easy example and say you have a class ItemPriceUpdateCommandHandler that looks like this

public class ItemPriceUpdateCommandHandler : ICommandHandler<ItemPriceUpdateCommand>
{
    private readonly ITransactionGenerator _transactionGenerator;
    private readonly IItemUpdateService _itemUpdateService;
    private readonly IItemRepository _itemRepository;
    public UpdateItemWithTypesCommandHandler(
        ITransactionGenerator transactionGenerator,
        IItemUpdateService itemUpdateService,
        IItemRepository itemRepository)
    {
        _transactionGenerator = transactionGenerator;
        _itemUpdateService = itemUpdateService;
        _itemRepository = itemRepository;
    }
    public async Task HandleAsync(ItemPriceUpdateCommand command)
    {
        using var transaction =
            await _transactionGenerator.GenerateAsync();
                
        var item = await _itemRepository.LoadAsync(command.ItemId);
        _itemUpdateService.UpdateItemPrice(item, command.Price);
        await _itemRepository.SaveAsync(item);
        await transaction.CommitAsync();
    }
}

See what's happening here? This command handler is only orchestrating calls to different services in HandleAsync but it doesn't do any data transformation itself. This is a prime candidate for unit tests with random test data as you're mocking the services behind the interfaces and just verify that the expected calls with the expected values (ItemId, Price, Item) are made in the correct order. What the exact values ItemId, Price & Item are? No one cares. You can randomly generate them. It's only important that the generated values are indeed passed.
Don't make the mistake here to assert for any argument (instead of the exact value) passed into the methods and by that only verify that the method is called at all. What if you made a mistake and accidentally passed command.OldPrice into the update service instead of command.Price? The test wouldn't notice it.

Transformation Code

Transformation code is the opposite. It does actual data transformation through any kind of logic.

Let's for a second pretend that this is how seat reservation works in a Redux component with an immutable state.

public static State ReserveNextAvailableSeat(State state)
{
    var allSeats = state.Seats.ToList();
    var seat = allSeats.FirstOrDefault(s => !s.IsReserved);
    if(seat is null)
        throw new Exception("No seat available anymore");
    var seatIndex = allSeats.IndexOf(seat);
    var newSeat = seat.Reserve()
    allSeats[seatIndex] = newSeat;
    
    return state with {
        Seats = allSeats.ToList()
    };

}

What do we want to test on a happy path here?

  1. No exceptions are thrown
  2. The first available seat is reserved
  3. No other changes were made (e.g. a seat disappeared from the list)

The first and last one are easy. You assert that no exception is thrown by the code and compare the entire actual result (that's returned by the method) to your expected result, which means your expected result is a State, not a Seat (FluentAssertions for example has a great deep-compare method that always compares the values of two objects, not their reference).

So far, so good. But how do we test the second (and main) requirement? Let's look at our two data options.

  1. Static Data
    With static data this is pretty straight forward. You hard-code the creation of a list of seats in the state and set one to not reserved. Because you know all the values, you can also hard-code the expected result. Done.
[Fact]
public void ReserveNextAvailableSeat_WithSeatAvailable_ShouldReserveSeat()
{
    // Arrange
    var state = new State
    {
        Seats = [
            new Seat(id: 1, reserved: true),
            new Seat(id: 2, reserved: false)
        ]
    };
    var expectedResult = new State
    {
        Seats = [
            new Seat(id: 1, reserved: true),
            new Seat(id: 2, reserved: true)
        ]
    };
    
    // Act
    var actualResult = SeatReducer.ReserveNextAvailableSeat(state);
    
    // Assert
    actualResult.Should.BeEquivalentTo(expectedResult);

}

But what if the implementation always uses the last seat? That would be wrong. So, that's a different test case, let's add it.

[Fact]
public void ReserveNextAvailableSeat_WithSeatAvailable_ShouldReserveSeat()
{
    // Arrange
    var state = new State
    {
        Seats = [
            new Seat(id: 1, reserved: true),
            new Seat(id: 2, reserved: false)
        ]
    };
    var expectedResult = new State
    {
        Seats = [
            new Seat(id: 1, reserved: true),
            new Seat(id: 2, reserved: true)
        ]
    };
    
    // Act
    var actualResult = SeatReducer.ReserveNextAvailableSeat(state);
    
    // Assert
    actualResult.Should.BeEquivalentTo(expectedResult);
}

[Fact]
public void ReserveNextAvailableSeat_WithFirstSeatAvailable_ShouldReserveSeat()
{
    // Arrange
    var state = new State
    {
        Seats = [
            new Seat(id: 1, reserved: false),
            new Seat(id: 2, reserved: true)
        ]
    };
    var expectedResult = new State
    {
        Seats = [
            new Seat(id: 1, reserved: true),
            new Seat(id: 2, reserved: true)
        ]
    };
    
    // Act
    var actualResult = SeatReducer.ReserveNextAvailableSeat(state);
    
    // Assert
    actualResult.Should.BeEquivalentTo(expectedResult);

}

That's quite a lot of code. And it'll be even more for more complex SUTs. We're surely somehow able to extract the state generation into setup methods, but that's not the focus here. However, it does its job.

  1. Random Data
    With random data, that's not so easy because you have to have knowledge about the state you initially create in order to calculate the expected result. But when you randomly create the state, you have no knowledge about its value composition. What if you create a state where all seats are already reserved? That would result in an exception and, thus, fail the test.
    But hey, frameworks like AutoFixture have you covered here. You can write configuration with which you can ensure that at least one seat is not reserved yet. And once we made sure that is the case, we can calculate the expected result. Awesome, let's do that.
[Fact]
public void ReserveNextAvailableSeat_WithSeatAvailable_ShouldReserveSeat()
{
    // Arrange
    var state = ...;// initial random state generation
    var availableSeat = state.Seats.First(s => !s.IsReserved);
    var seatIndex = state.Seats.IndexOf(availableSeat);
    var expectedSeats = state.Seats.ToList();
    expectedSeats[seatIndex] = new Seat(availableSeat.Id, true);
    var expectedResult = new State { Seats = expectedSeats };
    
    // Act
    var actualResult = SeatReducer.ReserveNextAvailableSeat(state);
    
    // Assert
    actualResult.Should.BeEquivalentTo(expectedResult);
}

Ok, works. But hang on. This kind of looks like our productive implementation of ReserveNextAvailableSeat. If we have a bug in there, we most likely also have a bug in the test - which kind of defeats the purpose of the test, doesn't it? Plus, the next developer that comes along has to do quite some thinking before they understand what's happening here and are able to modify the test. But there is another option for tackling at least the first issue.

  1. Random Data. Backwards.
    In the previous sections we randomly generated the state and calculated the expected result, which is the same direction as the production code goes (it gets the initial state and modifies it). What if we do it backwards? Randomly generate the expected result and calculate back what the initial state must be in order to get this result. Let's try it.
[Fact]
public void ReserveNextAvailableSeat_WithSeatAvailable_ShouldReserveSeat()
{
    // Arrange
    var expectedResult = ...;// random state generation that ensures that a random amount of n seats at the beginning of the list are reserved
    var seatIndex = ...; // randomly choose one of the n seats  
    var initialSeats = expectedResult.Seats.ToList();
    initialSeats[seatIndex] = new Seat(availableSeat.Id, false);
    var state = new State { Seats = initialSeats };
    
    // Act
    var actualResult = SeatReducer.ReserveNextAvailableSeat(state);
    
    // Assert
    actualResult.Should.BeEquivalentTo(expectedResult);
}

That looks a lot better, and we eliminated the need for a .First(...) and an .IndexOf(...). We still have some logic in there for calculating the initial state, but you won't be able to eliminate this to a certain degree if you want to use random data.
Of course this singular test does not absolve you from testing different test cases explicitly. What if all seats are available? What if the first seat is available? What if only the last seat is available? And so on, you get the idea.

Choosing the correct strategy

We've now taken a look into orchestration & transformation code and suitable unit testing strategies behind them. But you might say that sometimes orchestration and transformation is merged. What if the method loads and saves the data from/to a repository but does the transformation by itself? The first question that you should probably answer here is "Is this necessary or can we restructure the code into proper orchestration/transformation separation?". If the answer is "No", then you'd probably have to go with your desired transformation code test strategy as this is the lowest common denominator between both.

In the end it all boils down to the question of what you like better. While orchestration code can be easily tested with both static and random data, transformation code is a tougher decision.