Re-run failed test using Junit

Question

I'm trying to improve an existing system of automated Selenium test. My goal is to repeat the tests that fails because of connections problem. I've found and tried to follow this thread How to Re-run failed JUnit tests immediately? that revealed itself quite useful.

In my case the suite is composed by classes, so I've tried to substitute @Rule with @ClassRule, in order to repeat for each try also the @Before and @After parts. I'm sorry for my ignorance, but where am I supposed to place this rule? In my Suite class? Or in the Classes representing the test?

Answer 1

I am the original answerer of How to Re-run failed JUnit tests immediately?

If I understand correctly, the problem that you are having is due to the @Before being executed before the code in the RetryRule, and the @After being executed afterwards.

So your current behaviour is something like:

@Before
@Retry
test code
@Retry
@After

But you can implement your @Before and @After as a rule - there is a rule ExternalResource which does exactly that. You would implement @Before and @After as a rule:

@Rule public ExternalResource beforeAfter = new ExternalResource() {
    public void before() {
        // code that was in @Before
    }

    public void after() {
        // code that was in @After
    }
}

Then you don't need the @Before and @After. You can then chain these rules using RuleChain. This forces an order of execution to your rules:

@Rule public RuleChain chain= RuleChain
                       .outerRule(new LoggingRule("outer rule")
                       .around(new LoggingRule("middle rule")
                       .around(new LoggingRule("inner rule");

so your final solution would be something like:

private ExternalResource beforeAfter = ...
private RetryRule retry = ...

@Rule public RuleChain chain = RuleChain
                               .outerRule(retry)
                               .around(beforeAfter);

Note that if you are using RuleChain, you no longer need the @Rule annotation on the ExternalResource and RetryRule, but you do on the RuleChain.

Answer 2

Here is my solution based on the one mentionned in the question.

It's a combinaison of a @Rule, FailedRule and a @ClassRule, RetryRule

public class RetryTest
{
    public static class FailedRule implements TestRule
    {       
        @Override
        public Statement apply(final Statement base, final Description description)
        {
             return new Statement()
             {
                    @Override
                    public void evaluate() throws Throwable
                    {
                        try
                        {
                            base.evaluate();
                        }
                        catch (Throwable t)
                        {
                            System.out.println(description.getDisplayName() + " failed");
                            retry.setNotGood();
                            if (retry.isLastTry())
                            {
                                System.out.println("No more retry !");
                                throw t;
                            }
                            else
                            {
                                System.out.println("Retrying.");
                            }
                        }
                    }
            };
        }
    }

    public static class RetryRule implements TestRule
    {
        private int retryCount, currentTry;

        private boolean allGood = false;

        public RetryRule(int retryCount)
        {
            this.retryCount = retryCount;
            this.currentTry = 1;
        }

        public boolean isLastTry()
        {
            return currentTry == retryCount;
        }

        public void setNotGood()
        {
            allGood = false;
        }

        public Statement apply(final Statement base, final Description description)
        {
            return new Statement()
            {
                @Override
                public void evaluate() throws Throwable
                {
                    // implement retry logic here
                    for (; currentTry <= retryCount && !allGood; currentTry++)
                    {
                        allGood = true;
                        System.out.println("Try #" + currentTry);
                        base.evaluate();
                    }
                }
            };
        }
    }

    @ClassRule
    public static RetryRule retry = new RetryRule(3);

    @Rule
    public FailedRule onFailed = new FailedRule();

    @BeforeClass
    public static void before()
    {
        System.out.println("Before...");
    }

    @AfterClass
    public static void after()
    {
        System.out.println("...After\n");
    }

    @Test
    public void test1()
    {
        System.out.println("> test1 running");
    }

    @Test
    public void test2()
    {
        System.out.println("> test2 running");
        Object o = null;
        o.equals("foo");
    }
}

It gives :

Try #1
Before...
> test1 running
> test2 running
test2(RetryTest) failed
Retrying.
...After

Try #2
Before...
> test1 running
> test2 running
test2(RetryTest) failed
Retrying.
...After

Try #3
Before...
> test1 running
> test2 running
test2(RetryTest) failed
No more retry !
...After

If I am commenting the o.equals("foo"); in test2, everything runs fine in the firt try :

Try #1
Before...
> test1 running
> test2 running
...After 

Answer 3

You decorate the test name itself with the @After or @Afterclass attributes:

@After
@Test
@Category(SmokeTests.class)
public void testProductPageOnly() throws TimeoutException {
   //Some tests here.
}

@Afterclass
public static void SomeTest {
   //Some test here.
}

Something to note, @Afterclass will always run; even if you are using a @Beforeclass that throws an exception.

Answer 4

May this can solve problem:

1) Test class should be inherited from junit.framework.TestCase

2) Run your tests with something like this

YourTestClass testClass = new YourTestClass();
TestResult result = testClass.run();
Enumeration<TestFailure> failures = result.failures();
if (result.failureCount() != 0)
{
   TestFailure fail = failes.nextElement();
   junit.framework.Test test = fail.failedTest();
   test.run( result );
}

At the end result will contains last results of test running, so after analyzing what test was failed you can run it again.