Throughout this semester (Spring 2021) I’ve made a habit of writing automated testing suites for each assignment we’ve received in 01:198:112 (Data Structures). Since, for this course, AutoLab does not indicate where our code fails (only that it does), having an automated testing suite that you can run any amount of times without penalty substantially reduces the amount of stress and manual labour involved in ensuring that your submission is correct. Of course, this is heavily dependent on the tests themselves being correct. Tests were created using the test case information in this GitHub repository. When the provided test cases were not accompanied by solutions, we had someone whose code passed all the AutoLab tests provide their outputs to create correct unit tests.
In this post I explain the process by which any individual can create their own testing suite. However, we operate under the assumption that
Note that the 2nd prerequisite does not require that you have direct access to anyone else’s assignment code. Having them generate outputs from the inputs you give them is more than enough for the purpose of writing unit tests.
We will write unit tests for the first data structures assignment, Polynomial, using JUnit 5. In order to use JUnit 5 we will use a build tool for Java called Gradle. Gradle is a tool used in professional as well as open-source Java projects for managing the project and its dependencies. It greatly simplifies the process of including external libraries and tools in our project. Install Gradle by following the directions linked above and ensure that it is in your path.
Create a new directory in which the project will be stored, and open a command-line in that directory. Run gradle init and you will be greeted by an interactive command-line menu asking you to provide some details about your project. We will make the following selections:
[rosalogia@gdon PolynomialTestExample]$ gradle init
Welcome to Gradle 7.0!
Here are the highlights of this release:
- File system watching enabled by default
- Support for running with and building Java 16 projects
- Native support for Apple Silicon processors
- Dependency catalog feature preview
For more details see https://docs.gradle.org/7.0/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
Select type of project to generate:
1: basic
2: application
3: library
4: Gradle plugin
Enter selection (default: basic) [1..4] 2
Select implementation language:
1: C++
2: Groovy
3: Java
4: Kotlin
5: Scala
6: Swift
Enter selection (default: Java) [1..6] 3
Split functionality across multiple subprojects?:
1: no - only one application project
2: yes - application and library projects
Enter selection (default: no - only one application project) [1..2] 1
Select build script DSL:
1: Groovy
2: Kotlin
Enter selection (default: Groovy) [1..2] 1
Select test framework:
1: JUnit 4
2: TestNG
3: Spock
4: JUnit Jupiter
Enter selection (default: JUnit 4) [1..4] 4
Project name (default: PolynomialTestExample): Polynomial
Source package (default: Polynomial): polypoly. This is important. We keep the package name consistent with the package name in the assignment so that the user’s code will still be a valid submission while they’re testing it. If we changed the package name to one that the assignment doesn’t use, the user would be forced to change the package name back before submitting their code.That was a lot, but now we have a project set up in the current directory. A lot of files and folders have been generated. Here are some comments explaining the important ones:
.
├── app # This folder contains all our code, as well as a build.gradle file which is where we configure how our tests run
│ ├── build.gradle
│ └── src # This folder actually contains all our code, no gradle files.
│ ├── main # This is where the code for the assignment goes. Everything should theoretically be under the poly folder.
│ │ ├── java # This folder is superfluous and annoying, but Gradle generates it for us and removing it is a hassle. It won't cause problems.
│ │ │ └── poly
│ │ │ └── App.java
│ │ └── resources # This folder exists in case our project needs access to certain files (e.g. images, text documents, etc.) to run. It does not.
│ └── test # This is where our tests go
│ ├── java # This is where the code for our tests goes
│ │ └── poly
│ │ └── AppTest.java
│ └── resources # Again, exists in case our tests need access to certain files. Our tests do need access to test files, so we will use this.
├── gradle # Don't worry about this folder. It contains files necessary for others to be able to use the "Gradle wrapper" program (gradlew).
│ └── wrapper # That being said, don't forget to commit it to git if you're pushing this project to a repository. Users will need access to it.
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew # gradlew is a script/program that lets users of your project use gradle commands without installing gradle on their computer.
├── gradlew.bat # This one is a batch script for Windows users.
└── settings.gradle # For configuring Gradle settings, though we won't mess with it.Notice that, by default, Gradle generates a working Hello World project in your directory that you can build, run, and even run tests on. Try the following commands:
$ gradle build
$ gradle run
$ gradle testGenerally, this is how we will interact with our project.
Now we want to copy all the code from our Polynomial assignment into the appropriate sub-directory in our project. Particularly, we want to move all the files in poly/src/ from the original assignment into ./app./src/main/java/poly/. You can do this however you prefer, for me the command is cp ~/University/Courses/CS112/poly/src/* ./app/src/main/java/poly/. It will be different for you depending on where your original assignment lives. However, you can feel free to use a GUI if you’d like.
We now want to remove the file ./app/src/main/java/poly/App.java, since now our main method lives in the Polytest.java file. After doing this, we must update ./app/build.gradle to reflect the change we just made. Open it up in your preferred editor. The contents should look more or less like this:
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
dependencies {
// Use JUnit Jupiter API for testing.
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.1'
// Use JUnit Jupiter Engine for testing.
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
// This dependency is used by the application.
implementation 'com.google.guava:guava:30.0-jre'
}
application {
// Define the main class for the application.
mainClass = 'poly.App'
}
tasks.named('test') {
// Use junit platform for unit tests.
useJUnitPlatform()
}Notice this particular bit:
application {
// Define the main class for the application.
mainClass = 'poly.App'
}We want to change 'poly.App' to 'poly.Polytest' to reflect that our main class is the Polytest class. While we have this file open, we will make some more changes before we begin writing our tests. Below is what your build.gradle file should look like after maknig these changes. All the changes are commented so as to explain their purpose, while unchanged portions are truncated:
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
// CHANGE: We add a plugin called test-logger written by Adarsh Ramamurthy
// This plugin automatically displays the results of our unit tests cleanly
// when we run the gradle test command.
id 'com.adarshr.test-logger' version '3.0.0'
}
// …
dependencies {
// Use JUnit Jupiter API for testing.
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.1'
// CHANGE: We add the jupiter "params" library to enable "parameterized" testing,
// which we'll get into later
testImplementation 'org.junit.jupiter:junit-jupiter-params:5.7.1'
// Use JUnit Jupiter Engine for testing.
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
// This dependency is used by the application.
implementation 'com.google.guava:guava:30.0-jre'
// CHANGE: We add AssertJ as a dependency. This is my library of
// choice for doing "assertions" in tests. You will see what this
// means later on.
testImplementation "org.assertj:assertj-core:3.12.2"
}
// …
// CHANGE: We add this snippet of Groovy code that uses the test-logger plugin
// to display clear output regarding the test results when tests are run. You
// may customize this to your liking.
test {
afterTest { desc, result ->
logger.quiet "Executing test ${desc.name} [${desc.ClassName}] with result: ${result.resultType}"
}
}
// CHANGE: We add this snippet to make sure that when users run their code interactively,
// Gradle directs standard input (the command-line) to the application instead of expecting
// input from a file
run {
standardInput = System.in
}
// ...With that, we have properly configured our Gradle project for testing. Now we can dive into the code.
This part is more creative than you may have imagined. Writing tests can be tedious so we will try and automate as much of it as we can.
First, open up and observe the file ./app/src/test/java/poly/AppTest.java. It is a remnant of the project template: a unit test for the now non-existent App.java. Its structure is as follows:
Test annotation from the org.junit.jupiter.api package. This “annotation” is how we specify that a method is actually a unit test.AppTest. The name of this class is not important, we can name our test classes whatever we want.AppTest class, it defines a void method called appHasAGreeting, annotated with @Test which marks it as a unit test.appHasAGreeting method, a new App is created, and the method “asserts” that the result of calling .getGreeting() on the new instance of App does not return null. If it does, the test fails and the output of the test is "app should have a greeting".An assertion is a thing your test expects to be true; it is the test’s “condition”. When we write tests, we set up all the objects and data we need to verify that the code behaves as we expect it to in the case being covered by the test. Then, we write an assertion which succeeds and does nothing if the code does in fact behave as we expect, and throws an error if the code does not behave as we expect.
Now that we have some insight into how a unit test looks, we can delete ./app/src/test/java/poly/AppTest.java since we won’t be using it. Now we can create a file ./app/src/test/java/poly/EvaluateTest.java. In it, we will write unit tests for the evaluate method in our Polynomial class.
Before we jump into writing our tests, lets look at their source. The linked page contains the test cases we will be incorporating into our testing suite. There are some obvious questions that arise:
Notice that the format of the polynomials on the page above are compliant with the format of the testing files our assignment comes with. This means that in order to load them into our tests, we simply need to paste the contents of each test case’s polynomial input into a file and have the testing code read it with Polynomial.read.
As for knowing what the test cases should return in the case of success: this is where having access to a person who has already passed all the AutoLab tests is helpful. You can ensure that your unit tests are accurate by matching up their expectations with the outputs of someone who has already gotten a 100 on the project.
However, this is not our only option in this case. Since, after all, anyone can manually evaluate a polynomial at a given value of x, we can figure out what the correct outputs should be by hand. This sucks, though, so if we can avoid it we should. One alternative is to write a Python script that parses a polynomial file and uses a symbolic mathematics library called sympy to tell us what the correct outputs should be. This way we aren’t just reimplementing our Java code in Python, since that could be error prone. This is more akin to parsing the polynomial file and automatically plugging it into WolframAlpha from within Python. Since this is fun and interesting we will explore this option. However, feel free to skip this section, since it’s really not necessary.
In this section we will write a short Python script that reads a polynomial file and uses sympy to figure out what the correct output should be based on what it is that our test case is testing.
Suppose, to begin with, that we have a file poly1.txt that looks like this:
4 6
5 3
-11 1
5 0For now, we want to be able to call our script with 2 arguments:
Our script should read the file and use the information inside of it to build a polynomial using sympy. Then it should evaluate the polynomial at the given value and output the result. If you intend to follow along, make sure you have Python installed. You can install sympy with pip install sympy.
Before we can begin reading and parsing our polynomial files, we need to know what kind of input sympy is expecting. Let’s play around with an example where we directly encode a polynomial for sympy to work with:
from sympy import *
x = symbols("x")
polynomial = 4 * x ** 6 + 5 * x ** 3 + -11 * x ** 1 + 5 * x ** 0
print(polynomial)If everything is working, the output of the above should be the following: 4*x**6 + 5*x**3 - 11*x + 5. Interestingly, SymPy has figured out that x**1 (that is, x to the power of 1) and x ** 0 are redundant and has simplified our expression a bit as a result. SymPy is a powerful library and we’re only using some of its most basic features. Let’s expand our script above to also evaluate our polynomial at a point x = 1:
# …
polynomial = 4 * x ** 6 + 5 * x ** 3 + -11 * x ** 1 + 5 * x ** 0
print(polynomial.subs(x, 1))The output should be 3, which is correct. If we wanted to do this by hand, we’re already in pretty good shape. However, that really isn’t what we want. Instead, we’ll now begin writing some code to parse the poly1.txt file and automatically build SymPy polynomials and evaluate them for us.
First, lets create an empty polynomial and set up some code that reads one line from the file at a time:
from sympy import *
from sys import argv, exit
if len(argv) < 3:
exit("Please provide a polynomial file as well as a value at which the polynomial should be evaluated.")
x = symbols("x")
polynomial = 0
with open(argv[1]) as poly_file:
for line in poly_file:
# Do something with the lineNow it’s time to actually parse a line of the file and correctly add to the polynomial. Here’s the plan:
coeff degree. That is to say, the first number in the line is a coefficient of x, whereas the second number in the line is the power to which x is raised.polynomial the result of raising x to the power of degree and multiplying the result by coeff.Here’s the code that does this:
# …
polynomial = 0
with open(argv[1]) as poly_file:
for line in poly_file:
coeff = int(line.split()[0]) # split defaults to splitting by spaces
degree = int(line.split()[1]) # We know this could throw an error, but this is a quick script. It's fine.
polynomial += coeff * x ** degree
print(polynomial)If you save this script as poly.py and run it as python poly.py poly1.txt 0, you should see the polynomial expression SymPy gave us before outputted to your terminal window. This is great, it means that the parsing was successful. All we have to do is slightly edit the script to evaluate our polynomial at the specified point.
# …
print(polynomial.subs(x, int(argv[2])))
# or if you'd prefer to break it up ...
evaluate_at = int(argv[2])
result = polynomial.subs(x, evaluate_at)
print(result)Both options should work equally well, and if you run the code again you should see that our polynomial is correctly evaluated at the given input.
We’ve now fully automated the process of generating correct outputs for evaluate. When it’s time to write tests for add and multiply, we will come back to this. For now, let’s return to the task of actually writing tests for evaluate!
We want to test each case on the previously linked page of test cases. Let’s begin by setting up our EvaluateTest.java file. Start the file with the following lines:
package poly;
import org.assertj.core.api.Assertions; // Importing the assertion library we specified as a requirement in build.gradle
class EvaluateTest {
// …
}Now we’re ready to begin. Let’s lay out our plan:
fromFile that takes a filename as input and returns a Node as output.resources directory for the tests to draw fromLet’s tackle these one at a time, beginning with the fromFile method. We want to write a method that will call Polynomial.read for us and return a Node object. You can look at the Polynomial.java file to see that Polynomial.read does not accept a file path, but rather it accepts a Scanner. In this method, we’ll simply construct a scanner from the given filepath and pass it to Polynomial.read.
// ...
import java.io.File;
import java.io.IOException;
import java.util.Scanner;
class EvaluateTest {
// We create a string variable to hold the path that contains
// our polynomial files so we don't have to repeat it anywhere
private static String resourceRoot = "./src/test/resources/evaluate/";
private Node fromFile(String fileName) throws IOException {
// Java does have type inference, and I like to use it.
// Obviously (from the constructor), the type of polyFile is File.
// I will only use the var keyword when context makes it obvious what
// the type of something is
var polyFile = new File(resourceRoot + fileName);
var reader = new Scanner(polyFile);
Node polynomial = Polynomial.read(reader);
reader.close();
return polynomial;
}
}Now the body of our tests will be cleaner, since they won’t be cluttered with file reading logic. Next, we’re going to prepare the data our tests will need to verify that our code works correctly. First we’ll create files in ./app/src/test/resources/ containing our test-case polynomials. Create a directory under resources called evaluate. In it, create three files case0.txt, case1.txt and case2.txt containing the test-case polynomials on the page linked above. Make sure that what you paste in only contains the numbers in the polynomial, and doesn’t include any other text or blank lines.
The next part of the data preparation step is creating some hard-coded data containing the values we’re evaluating our polynomials at as well as the expected output. We will do this by creating two arrays with an equal number of indices so that one index can be used to index into both arrays. Notice that we also named our case files in a way that’s conducive to this. The reason for using arrays in this case is that we will be writing “parameterized” tests. This means that, instead of writing out three separate test methods for each case (which involves quite a bit of repetition), we will write a single “general” test method that depends on a variable to generate various cases. In this case, that variable will be an array index. If this is confusing, hang on and see how the code works and you might understand better.
// …
class EvaluateTest {
// …
private static float[] inputs = { 0.0f, 8.0f, -1.0f };
// Generated by our script
private static float[] outputs = { 5.0f, 342391.0f, 1.0f };
}Now we’re ready to write our tests. In reality, we will only write one test, but we will parameterize it such that it will run the same test for 3 separate cases.
// …
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
class EvaluateTest {
// ...
@ParameterizedTest
@ValueSource(ints = {0, 1, 2})
void testEvaluate(int caseNumber) throws IOException {
// Retrieve the correct polynomial based on the caseNumber
Node polynomial = fromFile(resourceRoot + "case" + caseNumber + ".txt");
// Assert that the result of evaluating the polynomial at its corresponding input results in the corresponding output
Assertions
.assertThat(Polynomial.evaluate(polynomial, inputs[caseNumber]))
.isEqualTo(outputs[caseNumber]);
}
}That’s all there is to it. We’ve successfully written a parameterized test for our evaluate method that runs through 3 test cases with ease. You can test it out by running gradle test in your project root. Now we will move on to testing other methods.