In this entry, I aim to illustrate some of the issues that developers may encounter as they migrate their existing JUnit 3.x test suites to JUnit 4.

For this illustration, we'll migrate the tests for Jaggregate, a Java-5-only collections library that is modeled after the ANSI Smalltalk collection protocols.  As of this writing, there are over 3700 JUnit test methods in the project.  They are run by an AllTests class which exposes a suite() method.  The suite() method uses the JUnit Addons DirectorySuiteBuilder to collect a suite of all test classes in a given directory.  The project's Ant build file compiles all the test classes to a particular build directory; so the DirectorySuiteBuilder interrogates this directory for the test classes it seeks.

package test.jaggregate;

import junit.framework.Test;
import junitx.util.DirectorySuiteBuilder;
import junitx.util.SimpleTestFilter;

public final class AllTests {
    private AllTests() {
        throw new UnsupportedOperationException(
            "Do not instantiate this utility class" );
    }

    public static Test suite() throws Exception {
        final DirectorySuiteBuilder suiteBuilder = new DirectorySuiteBuilder();
        suiteBuilder.setFilter( new SimpleTestFilter() );

        return suiteBuilder.suite( "build/test-classes" );
    }
}

To start with, we obtain the latest source for JUnit 4 (as of November 8, 2005) from CVS, and build it using its Ant build file.  The dist target of JUnit's build file creates junit.jar, which contains all the classes needed to create and run JUnit tests.

At this point, I realize how much the modern Java IDE has spoiled me.  The baked-in JUnit support - with custom test runners that allow one-click execution of any slice of the JUnit test tree, one-click navigation to locations mentioned in stack traces, and the like - is so handy that having to go without it for purposes of this demonstration really feels like "roughing it."  I can't remember the last time I had to invoke a JUnit test runner by hand.  OK, here goes...

The first order of business is to ensure that all the tests run fine with the JUnit 3.8.1 console test runner, junit.textui.TestRunner.  In the root directory of Jaggregate's source distribution, ensuring that Jaggregate's test class files, Jaggregate's test subject class files, and the dependencies thereof (JUnit 3.8.1's junit.jar, and junit-addons.jar) are on the classpath, we fire up the old-style runner:

> java -cp build\classes;build\test-classes;lib\junit.jar;lib\junit-addons.jar junit.textui.TestRunner test.jaggregate.AllTests

.........................................
.........................................
[snip]
.......................................
Time: 1.625

OK (3729 tests)

Fantastic, all the tests pass.

Next, let's try to run the tests using JUnit 4's version of the JUnit 3.x-style console test runner.  This execution will look much like the first, except we'll swap in JUnit 4's junit.jar for 3.8.1's junit.jar on the classpath:

> java -cp build\classes;build\test-classes;c:\java\junit
4\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar junit.textui.TestRunner test.jaggregate.AllTests

.F
Time: 0.015
There was 1 failure:
1) warning(junit.framework.TestSuite$1)junit.framework.AssertionFailedError: Class test.jaggregate.AllTests has no public constructor TestCase(String name) or TestCase()

FAILURES!!!
Tests run: 1,  Failures: 1,  Errors: 0

In JUnit 3.8.1, if the class you handed to the test runner declared a public static Test suite() method, the runner would execute that method and run() the resulting suite, regardless of whether the providing class was a derivative of TestCase.  With JUnit 4, it seems as if the test runner is expecting the target class to be a derivative of TestCase with the attendant constructors.

It turns out that JUnit 4 introduced a defect into junit.textui.TestRunner.start(String[] args).  Apparently there is a new command line option -m, which when given a fully qualified test method name, will attempt to instantiate the class on which the method is indicated to reside, and execute that single test. start() looks like this (some irrelevant details elided):

public TestResult start(String args[]) throws Exception {
    String testCase= "";
    String method= "";

    // parse args here...if -m, set testCase and method
    // set testCase to next arg if no more options

    if (method != null)
        return runSingleMethod(testCase, method, wait);

    Test suite= getTest(testCase);
    return doRun(suite, wait);
}

If start() doesn't receive a -m option with argument, then method will remain pointed to the empty string, which is not null, which will cause start() to try to instantiate test.jaggregate.AllTests.  This instaniation fails, since it wants a public zero-argument or one-argument (String name) constructor on the target class.  Oops!

Initializing method to null instead of the empty string gets us around the issue.  So, let's make the change and rebuild JUnit 4.  Now when we fire off the command line above:

> java -cp build\classes;build\test-classes;c:\java\junit
4\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar junit.textui.TestRunner test.jaggregate.AllTests

.........................................
.........................................
[snip]
.......................................
Time: 0.609

OK (3729 tests)

Goodness.

Now, let's try to run the JUnit 3.x-style tests unmodified, this time using the new JUnit 4 test runner.  The JUnit 4 analogue to JUnit 3.x's console test runner is org.junit.runner.JUnitCore.  So, let's fire it up thusly:
C:\java\jaggregate-trunk>java -cp build\classes;build\test-classes;c:\java\junit4\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar org.junit.runner.JUnitCore test.jaggregate.AllTests
JUnit version 4.0rc1
E
Time: 0
There was 1 failure:
1) Test mechanism failure
java.lang.Exception: Test class should have public zero-argument constructor
        at org.junit.runner.internal.TestIntrospector.validateTestMethods(TestIntrospector.java:33)
[snip]

FAILURES!!!
Tests run: 0,  Failures: 1

Maybe we got a little overzealous by making our AllTests class uninstantiable.  Let's remove the private constructor altogether, and let the compiler provide a default public zero-argument constructor for us.  Now, we run again:

C:\java\jaggregate-trunk>java -cp build\classes;build\test-classes;c:\java\junit4\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar org.junit.runner.JUnitCore
JUnit version 4.0rc1

Time: 0.016

OK (0 tests)

It seems that JUnitCore isn't much interested in our suite().  Instead, it instantiates AllTests using its default public zero-argument constructor, looks for test methods on it, finds none, and stops.

When deciding how to run a named test class, JUnitCore makes a RunnerFactory which first looks for a RunWith annotation on the test class; if present, the factory will instantiate the class named in the annotation using its two-argument (RunNotifier, Class) constructor, and the test gets run using the newly instantiated runner.  Otherwise, if the test class is a pre-JUnit 4 test-so determined by deciding whether it is a derivative of junit.framework.TestCase-the runner factory yields an OldTestClassRunner to run the test.  If the test class has no RunWith annotation and is not a pre-JUnit4 test class, it is run using a TestClassRunner.

Since AllTests has no RunWith annotation and is not a derivative of TestCase, it is run using JUnit4's TestClassRunner.  This runner tries to find methods annotated with Test, Before, After, and the like to execute.  Since AllTests has none of those, no tests get executed.

Maybe we can "fool" JUnitCore into recognizing and executing AllTests' suite()  by modifying AllTests to extend TestCase:

public final class AllTests extends TestCase {
    // ...
}

Here we go:

C:\java\jaggregate-trunk>java -cp build\classes;build\test-classes;c:\java\junit-4.0rc1\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar org.junit.runner.JUnitCore test.jaggregate.AllTests
JUnit version 4.0rc1
.E
Time: 0.015
There was 1 failure:
1) junit.framework.TestSuite$1.warning()
junit.framework.AssertionFailedError: No tests found in test.jaggregate.AllTests

        at junit.framework.Assert.fail(Assert.java:51)
        at junit.framework.TestSuite$1.runTest(TestSuite.java:93)
        at junit.framework.TestCase.runBare(TestCase.java:130)
        at junit.framework.TestResult$1.protect(TestResult.java:110)
        at junit.framework.TestResult.runProtected(TestResult.java:128)
        at junit.framework.TestResult.run(TestResult.java:113)
        at junit.framework.TestCase.run(TestCase.java:120)
        at junit.framework.TestSuite.runTest(TestSuite.java:228)
        at junit.framework.TestSuite.run(TestSuite.java:223)
        at org.junit.runner.internal.OldTestClassRunner.run(OldTestClassRunner.java:29)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:72)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:55)
        at org.junit.runner.JUnitCore.runMain(JUnitCore.java:45)
        at org.junit.runner.JUnitCore.main(JUnitCore.java:24)

FAILURES!!!
Tests run: 1,  Failures: 1

This change at least got JUnitCore to run AllTests with OldTestCaseRunner.  It looks as though a TestSuite was constructed and executed; but it was found to have no tests, so the runner complained and stopped.  What happened?  It seems AllTests' suite() method wasn't invoked after all.  Instead, when the OldTestCaseRunner was constructed, it built a TestSuite using TestSuite's one-arg (Class) constructor, which interrogates the class for the usual public void test*() methods.  Again, AllTests has none of these.  It expects a test runner to ask for its suite().

Perhaps we can extend JUnit 4's notion of what constitutes a "pre-JUnit 4" test class: Either it is a TestCase derivative, or it has a public zero-argument suite() method whose return type is junit.framework.Test.

...Then again, maybe we can just use org.junit.runner.extensions.AllTests.  It looks like this is a test runner which interrogates the target class for a suite() method, and invokes it to get a set of tests to run, then runs them with an OldTestClassRunner.  Sweet.  I bet if we annotate test.jaggregate.AllTests like so:

@RunWith( org.junit.runner.extensions.AllTests.class )
public final class AllTests {
    // ...
}

being sure to make Jaggregate now depend on JUnit 4.0's junit.jar instead of JUnit 3.8.1's, we'll see that:

C:\java\jaggregate-trunk>java -cp build\classes;build\test-classes;c:\java\junit-4.0rc1\junit\junit4.0rc1\junit.jar;lib\junit-addons.jar org.junit.runner.JUnitCore test.jaggregate.AllTests
JUnit version 4.0rc1
......................................................................................................................
[snip]
.................................................
Time: 0.688

OK (3729 tests)

We're back in the saddle again.  In fact, I can even put back test.jaggregate.AllTests' private constructor to prevent instantiation, and no longer need to extend TestCase.  The AllTests runner doesn't need to instantiate the suite() provider.

Let's summarize what we've discovered so far.

* We can run our AllTests using JUnit 4's version of junit.textui.TestRunner without modifying our tests, but needed to fix a bug to make the execution succeed.
* The JUnit 4 console test runner, org.junit.runner.JUnitCore, honors only those JUnit 3.x tests that are instantiable derivatives of TestCase.  To run a class that provides a suite() without extending TestCase, annotate the suite provider class with @RunWith(org.junit.runner.extensions.AllTests.class).

In the next installment, we'll see what happens when we migrate test classes to JUnit 4 style (needn't extend TestCase, use @Test, etc.), and try to figure out a way to collect tests like we did with JUnit 3.x.