DDT: Directory Driven Tests

Of course you know TDD which stands for test driven development but do you also know DDT?

To be fair, this is an acronym that I just made up and which stands for directory driven tests. I think this term accurately describes the following approach of organizing test input and output.

In the simplest form, a test might require a single input and it produces a single output from it as in this pseudo code snippet:

@Test
void test1() {
    final String testInput = "very large string literal";
    final String testOutput = subject.codeUnderTest(testInput);
    assertThat(testOutput).isEqualTo("large result string");
}

@Test
void test2() {
    final String testInput = "another, even larger string literal";
    final String testOutput = subject.codeUnderTest(testInput);
    assertThat(testOutput).isEqualTo("another, way larger result than the first one");
}

These tests come with some inconveniences. First, the two test cases only differ in parameterization. It would be better to express them as @ParameterizedTest:

@ParameterizedTest
@CsvSource(delimiter = ";", value = {
    "very large string literal;large result string",
    "another, even larger string literal;another, way larger result than the first one"
})
void test(String testInput, String expectedOutput) {
    final String testOutput = subject.codeUnderTest(testInput);
    assertThat(testOutput).isEqualTo(expectedOutput);
}

This version of the test is sweet as it explicitly expresses that it executes multiple tests that only differ in parameterization. It is more or less obvious how to add further input/output combinations to test. The still not so nice part are those very large and even larger string literals that the codeUnderTest apparently processes. They add a lot of clutter to the test class and make it harder to read. Also it is hard to spot where in a single csv line the input ends an the output begins. To put it short: when working with large strings you’d normally want to externalize them into a file instead of putting them directly into your source code.

It would be great to have a JUnit5 @ArgumentSource implementation that is able to list files within a directory and provide them as input to a @ParameterizedTest. Assume the following directory structure:

src
 + test
 |-+  resources
   |-+  test_cases
     |- test-case1.input
     |- test-case1.output
     |- test-case2.input
     |- test-case2.output

We want a test that iterates the *.input files, runs their contents through the codeUnderTest and compares the result against the corresponding *.output file. Like this:

@ParameterizedTest
@FilesFrom(directory = "test_cases", extension = ".input")
void test(TestFile inputFile) {
    final String testInput = inputFile.asText(StandardCharsets.UTF_8);
    final String expectedOutput = inputFile.siblingWithExtension("output").asText(StandardCharsets.UTF_8);
    
    final String testOutput = subject.codeUnderTest(testInput);
    assertThat(testOutput).isEqualTo(expectedOutput);
}

This is a nice and clean way to write a parameterized test where both the input and the output is externalized into files. Let’s examine a test that is just a little bit more complex and requires two (or more) large inputs. Putting all files in the same directory will soon become very confusing if you add more and more test cases. In such cases it would be way better to drive the test from directories where each sub directory represents a single test case and has the same defined structure.

src
 + test
 |-+  resources
   |-+  test_cases
     |-+ test_case1
       |- input1.txt
       |- input2.txt
       |- output.txt
       + test_case2
       |- input1.txt
       |- input2.txt
       |- output.txt

Now we want a test that iterates the directories within test_cases and provides each sub directory as paramter to the test method:

@ParameterizedTest
@DirectoriesFrom(directory = "test_cases")
void test(TestDirectory testDirectory) {
    final String input1 = testDirectory.resolve("input1.txt").asText(StandardCharsets.UTF_8);
    final String input2 = testDirectory.resolve("input2.txt").asText(StandardCharsets.UTF_8);
    final String expectedOutput = testDirectory.resolve("output.txt").asText(StandardCharsets.UTF_8);
    
    final String testOutput = subject.codeUnderTest(input1, input2);
    assertThat(testOutput).isEqualTo(expectedOutput);
}

That is a very clean and concise way of defining test cases. That is also due to the various convenience methods that the TestFile and TestDirectory classes provide.

Unsurprisingly, this approach can very well be combined with snapshot testing. That is why the snapshot-test library comes with an extra module that provides exactly the features as explained above. The directory-params module is currently marked experimental and its latest coordinates can be found on the GitHub release page.