JUnit 5 – Empty test suite in IntelliJ IDEA when using JUnit Jupiter engine

How to execute All Suite tests with JUnit 5 in IntelliJ IDEA v2016.2.2?

I get Empty test suite running this code:

import org.junit.platform.runner.IncludeEngines;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.runner.SelectPackages;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
@IncludeEngines("junit-jupiter")
@SelectPackages("<eu...package>") //I confirm that <eu...package> is ok.
public class AllTests {
}

I receive:

INFORMAZIONI: Discovered TestEngines with IDs: [junit-jupiter, junit-vintage]
Empty test suite.
Empty test suite.

[root]
JUnit Jupiter
JUnit Vintage

OR

import eu.....services.ServiceTest;
import eu.....repository.DAOTest;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
        ServiceTest.class,
        DAOTest.class
})
public class AllTests {
}

I receive:

INFORMAZIONI: Discovered TestEngines with IDs: [junit-jupiter, junit-vintage]
Empty test suite.

[root]
|+--JUnit Vintage
|   +--eu.....AllTests
|+--JUnit Jupiter

I was able to run suite with JUnit 4, but it doesn’t work with JUnit 5.

Answer

Short Answer

If you are using IntelliJ IDEA 2016.2, it is currently not possible to execute a test class annotated with @RunWith(JUnitPlatform.class) within the IDE.

Long Answer

Based on the behavior you reported, after some painstaking investigative work, I believe I have the answer to your question…

If you are using IntelliJ IDEA 2016.2 which has built-in support for JUnit 5, then the following is what is happening.

1. IDEA launches the JUnit Platform via the Launcher API, selecting the test class annotated with @RunWith(JUnitPlatform.class) (let’s call it TestSuite).
2. The Launcher detects both the junit-jupiter and junit-vintage TestEngine implementations.
3. The JUnit Jupiter engine ignores TestSuite since it is technically not a JUnit Jupiter test class.
4. The JUnit Vintage engine also ignores TestSuite since it is annotated with @RunWith(JUnitPlatform.class).
5. The end result is that neither registered test engine claims that it can run the TestSuite class.

The non-intuitive part is that the JUnit Vintage engine ignores TestSuite, when it in fact looks like a JUnit 4 based test class since it’s annotated with @RunWith(). The reason it is ignored is to avoid infinite recursion, which is explained in the source code for DefensiveAllDefaultPossibilitiesBuilder:

if ("org.junit.platform.runner.JUnitPlatform".equals(runnerClass.getName())) {
    return null;
}

The fact that the above code returns null in such scenarios results in an empty suite.

Of course, it would certainly be better if the user were informed of such scenarios — for example, via a log statement. I have therefore opened issues for both JUnit 5 and IntelliJ to improve the usability in such scenarios.

On the plus side, since you’re using IntelliJ IDEA 2016.2, you don’t need to use the test suite support. Instead, you can simply right-click on src/test/java in the project view in IDEA and select Run 'All Tests', and that will run all your tests.

Regards,

Sam (JUnit 5 core committer)