How do I control order of test execution within a single integration test file?

Question

I’m using Maven 3.0.3, Failsafe plugin v2.17 and JUnit 4.11. Currently I have an integration test with tests in the following order

@RunWith(SpringJUnit4ClassRunner.class)
public class MyTests {

    @Test
    public final void testAdd() {
        …
    }

    @Test
    public final void testUpdate() {
        …
    }

    @Test
    public final void testDelete() {
        …
    }

Currently when I run the tests through Maven as part of a “mvn clean install” run, the “testDelete” is getting run before the “testAdd” or “testUpdate”. If I change the name to “testZZZDelete”, then it gets run last but I don’t like that.

How do I get the tests to run in the order that I specify them in the file? My failsafe configuration is like so:

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.17</version>
        <configuration>
                <reuseForks>true</reuseForks>
                <argLine>-Xmx4096m -XX:MaxPermSize=512M ${itCoverageAgent}</argLine>
        </configuration>
        <executions>
                <execution>
                        <goals>
                                <goal>integration-test</goal>
                                <goal>verify</goal>
                        </goals>
                </execution>
        </executions>
</plugin>

The best is to use a integration test framework instead of a unit test framework. Use [testng](http://testng.org) where you can control dependencies etc. of the integration tests which means you can define the order of execution of the test methods no matter how they are named by annotations like `@test(dependsOnMethods={"m1", "m2§´"}`. JUnit is not intended for integration tests. No need to make assumptions on alphetical order etc. — khmarbaise, Jun 20 '14 at 19:53
I was brought into this project late and at this time, don't have the luxury of converting hundreds of tests to TestNG, although will definitely recommend it to the group when we have time to spare. In the emantime, any ideas on how to solve the problem given the constraints? — Dave, Jun 20 '14 at 21:31
Duplicate of http://stackoverflow.com/questions/3693626/how-to-run-test-methods-in-specific-order-in-junit4 — Raedwald, Sep 23 '14 at 12:16

score 6 · Answer 1 · edited May 23 '17 at 12:02

Short answer: no, there isn't anyway apart from renaming the tests so that they execute in order. You can use the @FixMethodOrder(NAME_ASCENDING) annotation to ensure that the tests are executed in alphabetical order.

Long answer: As I'm sure you know, surefire/failsafe gives you the option of ordering test classes through the runOrder configuration parameter. This controls the order in which each test class is executed. So you can run class foobar.Test1 before foobar.Test2 or the other way around.

For the order of execution of methods within a class, the problem that you are facing is that the JVM doesn't return the list of methods in the same order that they are declared in the file. With Java 6, the order in which they were returned was usually the order of declaration, but this changed with java 7. So, with the release of JUnit 4.11, the default ordering was changed to be based on the hash of the method name, to give a deterministic, but not predictable ordering. This is why you are getting testDelete run before anything else.

After a long discussion, we added the FixMethodOrder annotation to JUnit 4.11 to allow someone to at least be able to rename their methods. This seems to work with the SpringJUnit4ClassRunner - at least with the latest version 4.1.0.RELEASE. I haven't tried other releases.

So, to have a predictable ordering, you could, where necessary, rename your methods to be executed in the order you want and add the @FixMethodOrder annotation to the class.

@RunWith(SpringJUnit4ClassRunner.class)
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class MyTests {

@Test
public final void step1Add() {
    …
}

@Test
public final void step2Update() {
    …
}

@Test
public final void step3Delete() {
    …
}

For more information, please see Has JUnit4 begun supporting ordering of test? Is it intentional?.

score 1 · Answer 2 · answered Jun 22 '14 at 12:59

I think TestNg gives an option to execute the tests in user-defined sequence.

@Test(dependsOnMethods='testAdd')
public final void testAdd() {
    …
}

@Test(dependsOnMethods='testAdd')
public final void testUpdate() {
    …
}

@Test(dependsOnMethods='testUpdate')
public final void testDelete() {
    …
}

So, if the compiler tries to execute the "testUpdate" test, there is a restriction added to the execution. The testUpdate function is depending on the "testAdd" function. So Compiler will first perform the "testAdd", then proceed with the "testUpdate" function.

Hope the tests should be migrated from JUNIT to TestNg.

score 0 · Answer 3 · edited May 23 '17 at 12:26

0

This JUnit patch implements this feature https://github.com/adko-pl/junit/commit/4421261dbdcaed8ff0a82f4d5229ac8ad6c97543, thought it seems that the author didn't add the order param to the org.junit.Test annotation, using a patched JUnit will be way more easy than adapting all your current tests in a different framework IMHO.

See How to run test methods in specific order in JUnit4?

edited May 23 '17 at 12:26

Community

1
1

answered Sep 24 '14 at 14:50

isalgueiro

1,973
16
20

SO is your answer to add the "@FixMethodOrder(NAME_ASCENDING)" annotation? I can't tell by following this sequence of links but that's what it would appear to me. – Dave Sep 24 '14 at 19:03
That JUnit patch uses a new `org.junit.Test` attribute to specify test ordering. With that you could add `@FixMethodOrder(METHOD_ORDER)` and `@Test(order=1)`, `@Test(order=2)` etc to your test methods. See the changes in src/test/java/org/junit/internal/MethodSorterTest.java. – isalgueiro Sep 25 '14 at 08:22

score -1 · Answer 4 · edited May 23 '17 at 10:29

-1

First off, having tests depend on each other is typically bad practice, but i realise this is a purest view and that if it is really so slow to get to that state that you have to continue on then I would say you should have a look at the answers provided over on this post personally I would pick the second option as annotations make it nice and explicit.

edited May 23 '17 at 10:29

Community

1
1

answered Jun 20 '14 at 21:57

Paul Harris

5,769
1
25
41

Dependant tests are bad whatever type of testing you do and if they can be avoided, they should be. A normal scenario is say you have a login test, which ends with an assertion, this assertion might be important for the login scenario but has no relevance to all the other tests you have that depend on it. Someone breaks this one test, now it breaks every test you have checked in making it hard to determine what your check in did and did not break. Also dependant tests start to add in incidental details, did doing the action in the test break it or does it only break because of the series etc.. – Paul Harris Jun 21 '14 at 07:53
If the only reason he wants dependant tests is speed, then i would just recommend adding parallelisation into your test suite. However I would agree with your comment and say using testng wouldn't be a bad idea, lets be honest changing a set of tests from junit to testng would probably be a ten minute job. – Paul Harris Jun 21 '14 at 07:57
This seems like a good solution but I already had a @Runwith annotation defined (question has been edited). Is there some trick to running a test class with two RunWith annotations? – Dave Jun 22 '14 at 16:40
instead of BlockJUnit4ClassRunner extend the runner you are using currently in the run with annotation. – Paul Harris Jun 24 '14 at 13:54
I see what you're saying about extending. But how would I write a comparator to tell the system to execute test methods in the order in which they appear in teh file? Would I have to hard code the list for each file I'm dealing with? – Dave Jun 25 '14 at 19:41
I would use the other answer with annotations – Paul Harris Jun 30 '14 at 21:24
"tests depend on each other is typically bad practice," - looks like you have never written integration tests. – RamPrakash May 16 '23 at 18:55

How do I control order of test execution within a single integration test file?

4 Answers4