JUnit's @TestMethodOrder annotation not working

Question

I'm having a problem with following integration test

import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;

@SpringBootTest
@ActiveProfiles("test")
@TestMethodOrder(OrderAnnotation.class)
public class FooServiceIT {
    @Test
    @Order(1)
    void testUploadSuccess() { ... }
    @Test
    @Order(2)
    void testDownloadSuccess() { ... }
    @Test
    @Order(3)
    void testDeleteSuccess() { ... }
}

I'd expect when I run the test that the execution order would 1, 2, 3 but for some reason, the actual execution order is 2, 3, 1.

Tbh, I'm clueless why the annotation is not working. I'm using Spring Boot 2.1.3 with JUnit 5.4.

theoretically you should write your tests the way that they can run in any order... why do you need to order them? — Ruth, Mar 01 '19 at 15:31
You have configured wrongly. Try to put `@TestMethodOrder(MethodOrderer.OrderAnnotation.class)` [https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/TestMethodOrder.html](https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/TestMethodOrder.html) — Valijon, Mar 01 '19 at 15:33
@Valijon I've updated the imports, I was already using the `OrderAnnotation` class properly. @RuthiRuth the methods are manipulating files on a remote server, obviously, there the download/delete method will fail if I don't upload the file on the server first (I'd also like to keep the method separate - i.e. not calling the upload method in the download test). — toucheqt, Mar 01 '19 at 15:37
Can you please share the rest of your imports, in particular for the `@Test` annotation? — Sam Brannen, Mar 01 '19 at 17:29
One reason for ordering tests is to resolve cases where running single tests in a suite succeeds but running multiple tests fails on, say the second one because the test harness has not been torn down after the first test as expected. This can be a real pain to debug, especially with the latest versions of the Maven test plugins, where the fork configurations seem sometimes to interact non deterministically with junit 5. — maw, Nov 19 '19 at 21:46
Use this annotation ```@TestMethodOrder(MethodOrderer.OrderAnnotation.class)``` — Anupam Haldkar, Aug 05 '20 at 16:40

Valijon · Accepted Answer · 2019-03-02T14:35:39.293

36

You need to configure correctly your IDE.

Requirements

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0</version>
</dependency>

Do not use JUnit 5 that offers your IDE. If you add it as library, you will get:

No tests found for with test runner 'JUnit 5' 
==================== and this exception ===================
TestEngine with ID 'junit-vintage' failed to discover tests
java.lang.SecurityException: class "org.junit.jupiter.api.TestMethodOrder"'s signer information does not match signer information of other classes in the same package

So just include mentioned dependency only and your code will work as you expect:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class FooServiceIT {

    @Test
    @Order(1)
    public void testUploadSuccess() {
        System.out.println("1");
    }

    @Test
    @Order(2)
    public void testDownloadSuccess() {
        System.out.println("2");
    }

    @Test
    @Order(3)
    public void testDeleteSuccess() {
        System.out.println("3");
    }
}

JUnit result:

1
2
3

edited Mar 02 '19 at 14:35

answered Mar 02 '19 at 14:27

Valijon

12,667
4
34
67

Thanks! This wasn't the actual problem but your answer pointed me to right direction. The problem was that I've used the `junit-jupiter` artifactId as an aggregator dependency which for some reason did not contain the `junit-jupiter-engine` in the 5.4.0 version but in 5.3.2 version. After I've used the dependencies in correct versions, the tests did not run at all but adding platform launcher as suggested in https://stackoverflow.com/a/54652059/3750108 solved the issue. Now the tests works, and they work in correct order. – toucheqt Mar 04 '19 at 08:00
You just saved me from being extra tilted due to a wrong dependency version. Cheers! – Kata May 31 '19 at 11:39
1

I am having the same issue. Just replaced `junit-jupiter-engine` version to 5.4.0 and still having the reverse order execution. Can you please provide some feedback? Many thanks. – Deminem Mar 21 '20 at 18:37
Just want to add, this thread https://github.com/junit-team/junit5/issues/1776 also mentioned the same thing. Still No luck - any help would be appreciated. Thanks. – Deminem Mar 21 '20 at 18:40
@Deminem make sure `junit-jupiter-engine` is taking correct version, not from IDE. If you can share your code, I can take a look – Valijon Mar 21 '20 at 18:48
@Valijon - I've just confirmed that using the `junit-jupiter-engine` correct version. Please see yourself. `com.personalitytest.demo:personalitytest:jar:1.0-SNAPSHOT \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0:test ` – Deminem Mar 22 '20 at 03:43
@Valijon - please check the code which I've posted in the separate thread. Thanks. – Deminem Mar 22 '20 at 03:46
`junit-jupiter-engine` comes with the `spring-boot-starter-test` dependency. no need to add it extra. please check if you annotated the class with `@TestMethodOrder(MethodOrderer.OrderAnnotation.class)` before adding another dependency – Fzum Jun 21 '21 at 13:34
Using a recent Eclipse (2021-09 (4.21.0)), the built-in junit-jupiter-engine (appears to be 5.7.1) is good enough, all that is required is the `@TestMethodOrder` annotation. – xenoid Feb 02 '22 at 11:19

score 13 · Answer 2 · edited May 30 '20 at 05:30

13

I have faced the same issue. But, I found the problem where exactly is on my case. Wrongly import the "Order" class.

Wrong One

import org.springframework.core.annotation.Order;

Right one

*import org.junit.jupiter.api.Order;*

Also, please verify the following five classes with proper import

import org.junit.jupiter.api.Assertions;

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;

import org.junit.jupiter.api.Order;

import org.junit.jupiter.api.Test;

import org.junit.jupiter.api.TestMethodOrder;

edited May 30 '20 at 05:30

Aman Srivastava

1,007
1
13
25

answered May 29 '20 at 16:24

user1448964

131
1
4

dude...you saved me lots of hours man. thank you for this. – javalove Apr 22 '23 at 16:02

score 10 · Answer 3 · answered Mar 01 '19 at 16:28

10

From the docs https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/TestMethodOrder.html

 @TestMethodOrder(MethodOrderer.OrderAnnotation.class)
 class OrderedTests {

     @Test
     @Order(1)
     void nullValues() {}

     @Test
     @Order(2)
     void emptyValues() {}

     @Test
     @Order(3)
     void validValues() {}
 }

answered Mar 01 '19 at 16:28

Jason Portnoy

757
8
23

4

How your answer addresses the problem? – Diego Marin Santos Nov 07 '19 at 16:06
This is the thing which I was missing. Thank you! – Hishara Dilshan HishD Apr 18 '23 at 13:13

score 8 · Answer 4 · answered Aug 14 '20 at 00:16

8

If you have/had JUnit 4, check the import for annotation @Test: import org.junit.Test;

For JUnit 5 import should be: import org.junit.jupiter.api.Test;

It was my issue for this question

answered Aug 14 '20 at 00:16

x3hree

81
1
2
4

score 1 · Answer 5 · answered Feb 28 '23 at 06:36

Please make sure you import libraries from org.junit.jupiter.api instead of org.junit

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(OrderAnnotation.class)
class OrderedTestsDemo {

    @Test
    @Order(1)
    void nullValues() {
        // perform assertions against null values
    }

    @Test
    @Order(2)
    void emptyValues() {
        // perform assertions against empty values
    }

    @Test
    @Order(3)
    void validValues() {
        // perform assertions against valid values
    }

}

Code Sources: https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order-methods

score 0 · Answer 6 · answered Mar 08 '20 at 11:39

you need to configure correctly your pom.xml. See mine :

<properties>
    <!-- Dependency versions -->
    <junit.jupiter.version>5.6.0</junit.jupiter.version>
    <maven-surefire-plugin.version>3.0.0-M4</maven-surefire-plugin.version>

    <!-- Java 10 -->
    <maven.compiler.source>1.10</maven.compiler.source>
    <maven.compiler.target>1.10</maven.compiler.target>

    <!-- Encoding -->
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

<!-- Jupiter API for writing tests -->
<dependencies>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-engine</artifactId>
        <version>${junit.jupiter.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<!-- Maven Surefire plugin to run tests -->
<build>
    <plugins>
        <!-- plugin to run test cases from maven -->
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin.version}</version>
        </plugin>
        <!-- Maven plugin to use perticular java version to compile code -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.1</version>
            <configuration>
                <source>${maven.compiler.source}</source>
                <target>${maven.compiler.target}</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Now your Junit 5 annotations must function correctly

This solution just tried and doesn't work – Arefe Jan 24 '22 at 04:39 — Arefe, Jan 24 '22 at 04:39

Deminem · Answer 7 · 2020-03-22T04:33:44.203

After applying all the settings based on previous recommended solution. I'm still having the same reverse or random @order execution.

Pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.3.RELEASE</version>
</parent>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0</version>
    <scope>test</scope>
</dependency>

Maven Dependency Tree:

mvn dependency:tree -Dverbose -Dincludes=org.junit.jupiter:junit-jupiter-engine

[INFO] com.personalitytest.demo:personalitytest:jar:1.0-SNAPSHOT
[INFO] \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0:test

JUnit Test:

@SpringBootTest
@ActiveProfiles("test")
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class JUnitOrderTest {

    private static final Logger log = LoggerFactory.getLogger(JUnitOrderTest.class);

    @Test
    @Order(1)
    public void testUploadSuccess() {
        log.info("Junit - Order 1");
    }

    @Test
    @Order(2)
    public void testDownloadSuccess() {
        log.info("Junit - Order 2");
    }

    @Test
    @Order(3)
    public void testDeleteSuccess() {
        log.info("Junit - Order 3");
    }
}

Output:

Running com.personalitytest.demo.security.JUnitOrderTest
08:48:35.470 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 2
08:48:35.480 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 3
08:48:35.481 [main] INFO com.personalitytest.demo.security.JUnitOrderTest - Junit - Order 1

@Valijon - Please have a look to the code. Any help would be appreciated - Thanks. — Deminem, Mar 22 '20 at 03:36
@Valijon - It was strange for me, before I noticed that you're running the tests against Junit5. @MethodExecutionOrder only work with Junit5 and any version of SpringBoot below `2.0` uses Junit4 for testing purposes. So I excluded the Junit4 from Spring settings and installed the Jupiter (Junit5) along MockitoJupiter(Mockito5) and now everything works as per expectation. Thanks for your help much appreciated! — Deminem, Mar 22 '20 at 19:04
Just want to add for other's, @Test annotation works differently for both Junit4 and Junit5. When you've spring boot configured below than 2.0, then it uses Junit4 and you don't realise it while running the tests through `mvn clean test`. — Deminem, Mar 24 '20 at 08:52

score 0 · Answer 8 · answered Jun 13 '22 at 09:46

0

same issue.

For my case. First, it's the @Test annotation, should be org.junit.jupiter.api.Test, not org.junit.Test.

Second, I added @RunWith(SpringRunner.class) in the head of the test class, after I remove it, the order of test cases run as expected.

answered Jun 13 '22 at 09:46

yogurtzzz

1
1

RunWith comes with JUnit4, I am not sure it is a good idea to mix up JUnit 4 & 5 features. Might be better to use @ExtendWith(SpringExtension.class) – Hamza Khattabi Aug 12 '23 at 11:45

Nick Borges · Answer 9 · 2020-04-11T20:13:20.730

-3

For example, by using Spring Boot you can use @FixMethodOrder(MethodSorters.JVM) instead of @TestMethodOrder(MethodOrderer.OrderAnnotation.class). All of tests are run in the order they appear.

@FixMethodOrder(MethodSorters.JVM)
public class MyTest{

   @Test
   public void zzzz(){}

   @Test
   public void cccc(){}

   @Test
   public void aaaa(){}

   @Test
   public void bbbb(){}
}

Order execution is:

zzzz()
cccc()
aaaa()
bbbb()

edited Apr 11 '20 at 20:13

answered Apr 11 '20 at 16:07

Nick Borges

477
5
3

Not true. Read the documentation. "Note that the order from the JVM may vary from run to run" – Ran Oct 30 '20 at 08:34
@FixMethodOrder is in Junit4 not Junit5 ! – S_intg Apr 19 '22 at 12:13

JUnit's @TestMethodOrder annotation not working

9 Answers9

Linked