TestNG or JUnit


For many years now, I have always found myself going back to TestNG whenever it comes to doing Unit Testing with Java Code. Everytime, I picked up TestNG, people have asked me why do I go over to TestNG especially with JUnit is provided by the default development environment like Eclipse or Maven. Continuing the same battle, yesterday I started to look into Spring’s testing support. It is also built on top of JUnit. However, in a few minutes of using the same, I was searching for a feature in JUnit that I have always found missing. TestNG provides Parameterized Testing using DataProviders. Given that I was once again asking myself a familiar question – TestNG or JUnit, I decided to document this so that next time I am sure which one and why.

Essentially the same

If you are just going to do some basic Unit Testing, both the frameworks are basically the same. Both the frameworks allow you to test the code in a quick and effective manner. They have had tool support in Eclipse and other IDE. They have also had support in the build frameworks like Ant and Maven. For starters JUnit has always been the choice because it was the first framework for Unit Testing and has always been available. Many people I talk about have not heard about TestNG till we talk about it.

Flexibility

Let us look at a very simple test case for each of the two.

package com.kapil.itrader;
import java.util.Arrays;
import java.util.List;
import junit.framework.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

public class FibonacciTest
{
    private Integer input;
    private Integer expected;

    @BeforeClass
    public static void beforeClass()
    {
        // do some initialization
    }

    @Test
    public void FibonacciTest()
    {
        System.out.println("Input: " + input + ". Expected: " + expected);
        Assert.assertEquals(expected, Fibonacci.compute(input));
        assertEquals(expected, Fibonacci.compute(input));
    }
}

Well, this is example showcases I am using a version 4.x+ and am making use of annotations. Priori to release 4.0; JUnit did not support annotations and that was a major advantage that TestNG had over its competitor; but JUnit had quickly adapted. You can notice that JUnit also supports static imports and we can do away with more cumbersome code as in previous versions.

package com.kapil.framework.core;
import junit.framework.Assert;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class BaseTestCase
{
    protected static final ClassPathXmlApplicationContext context;

    static
    {
        context = new ClassPathXmlApplicationContext("rootTestContext.xml");
        context.registerShutdownHook();
    }

    @BeforeSuite
    private void beforeSetup()
    {
       // Do initialization
    }

    @Test
    public void testTrue()
    {
        Assert.assertTrue(false);
    }
}

A first look at the two code, would infer that both are pretty much the same. However, for those who have done enough unit testing, will agree with me that TestNG allows for more flexibility. JUnit requires me to declare my initialization method as static; and consequently anything that I will write in that method has to be static too. JUnit also requires me to have my initialization method as public; but TestNG does not. I can use best practices from OOP in my testing classes as well. TestNG also allows me to declare Test Suite, Groups, Methods and use annotations like @BeforeSuite, @BeforeMethod, @BeforeGroups in addition to @BeforeClass. This is very helpful when it comes to writing any level of integration testing or unit test cases that need to access common data sets.

Test Isolations and Dependency Testing

Junit is very effective when it comes to testing in isolation. It essentially means that there is you can not control the order of execution of tests. And, hence if you have two tests that you want to run in a specific order because of any kind of dependency, you can not do that using JUnit. However, TestNG allows you to do this very effectively. In Junit you can make workaround this problem, but it is not neat and that easy.

Parameter based Testing

A very powerful feature that TestNG offers is “Parameterized Testing”. JUnit has added some support for this in 4.5+ versions, but it is not as effective as TestNG. You may have worked with FIT you would know what I am talking about. However, the support added in JUnit is very basic and not that effective. I have modified my previous test case to include parameterized testing.

package com.kapil.itrader;

import static org.junit.Assert.assertEquals;

import java.util.Arrays;
import java.util.List;

import junit.framework.Assert;

import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class FibonacciTest
{
    private Integer input;
    private Integer expected;

    @Parameters
    public static List data()
    {
        return Arrays.asList(new Integer[][] { { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 } });
    }

    @BeforeClass
    public static void beforeClass()
    {
        System.out.println("Before");
    }

    public FibonacciTest(Integer input, Integer expected)
    {
        this.input = input;
        this.expected = expected;
    }

    @Test
    public void FibonacciTest()
    {
        System.out.println("Input: " + input + ". Expected: " + expected);
        Assert.assertEquals(expected, Fibonacci.compute(input));
        assertEquals(expected, Fibonacci.compute(input));
    }

}

You will notice that I have used @RunWith annotation to allow my test case to be parameterized. In this case, the inline method – data() which has been annotated with @Parameters will be used to provide data to the class. However, the biggest issue is that the data is passed to class constructor. This allows me to code only logically bound test cases in this class. And, I will end up having multiple test cases for one service because all the various methods in the Service wil require different data sets. The good thing is that there are various open source frameworks which have extended this approach and added their own “RunWith” implementations to allow integration with external entities like CSV, HTML or Excel files.

TestNG provides this support out of the box. Not support for reading from CSV or external files, but from Data Providers.

package com.kapil.itrader.core.managers.admin;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.uhc.simple.common.BaseTestCase;
import com.uhc.simple.core.admin.manager.ILookupManager;
import com.uhc.simple.core.admin.service.ILookupService;
import com.uhc.simple.dataprovider.admin.LookupValueDataProvider;
import com.uhc.simple.dto.admin.LookupValueRequest;
import com.uhc.simple.dto.admin.LookupValueResponse;

/**
 * Test cases to test {@link ILookupService}.
 */
public class LookupServiceTests extends BaseTestCase
{

    @Test(dataProvider = "LookupValueProvider", dataProviderClass = LookupValueDataProvider.class)
    public void testGetAllLookupValues(String row, LookupValueRequest request, LookupValueResponse expectedResponse)
    {
        ILookupManager manager = super.getLookupManager();
        LookupValueResponse actualResponse = manager.getLookupValues(request);
        Assert.assertEquals(actualResponse.getStatus(), expectedResponse.getStatus());
    }
}

The code snippet above showcases that I have used dataProvider as a value to the annotations and then I have provided a class which is responsible for creating the data that is supplied to the method at the time of invocation. Using this mechanism, I can easily write test cases and its data providers in a de-coupled fashion and use it very effectively.

Why I choose TestNG

For me the Parameterized Testing is the biggest reason why I choose TestNG over Junit. However, everything that I have listed above is the reason why I always want to spend a few minutes in setting up TestNG in a new Eclipse setup or maven project. TestNG is very useful when it comes to running big test suites. For a small project or a training exercise JUnit is fine; because anyone can start with it very quickly; but not for projects where we need 1000s of test cases and in most of those test cases you will have various scenarios to cover.

http://kapilvirenahuja.com/tech/2011/08/07/testng-or-junit/

About these ads
  1. Thomas
    1 September, 2011 at 9:17 pm

    The @Before annotation in Junit seems to translate to the @BeforeMethod in TestNG

  2. Tristan
    22 September, 2011 at 2:04 am

    but not for projects where we need 1000s of test cases and in most of those test cases you will have various scenarios to cover.

    Given many many people do just that with JUnit I think that’s a bit of a leap.

    As for things like dependencies in tests – one of the reasons I prefer JUnit is it does not let you have such things easily. Dependencies between tests is a test smell, and often indicative of a code smell. That’s a valuable pointer when it comes to designing code.
    Interdependencies between tests makes your testing more fragile and can allow your code to be more fragile.

    I’m not sure what the problem with @BeforeClass is, I never use it, @Before suffices and is not static.
    Also, I see no problem with setup methods being public, the test framework needs to access them and making them private surely means that the framework needs to use reflection hackery to get round that.

    There are now alternatives to the Parameterized runner (which is lacking), and JUnit is even more flexible now it has Rules.

    When it comes down to it I don’t think there is much difference. I prefer JUnit because the philosophy behind it is geared towards helping me write better code (just like I prefer JMock to Mockito as a design tool), but there isn’t that much in it.
    I just feel your crticisms are not really justified.

    • 4 January, 2012 at 1:34 pm

      Hi Tristan – I want to start off my apologizing to the fact that your response remained unapproved for a long time; something I did not intent to do, but just happened. Also, thank you for presenting your view point.

      Given many many people do just that with JUnit I think that’s a bit of a leap.
      – I do not disagree; it is just that I do not prefer JUnit

      dependencies in tests
      – I agree with the principle and live by it; and will also agree that JUnit forces me to do so, especially when I have to write Unit tests. It is when I use the framework for something bigger like functional/integration testing is when I find testNG provides greater flexibility. Now, this is a debate I been into with many peers and everyone has their own POV

      having initialization methods as static
      – as I stated earlier “I can use best practices from OOP in my testing classes as well”. I have learned with tough situations that if I provide a way to my developers to write non-standard code, they love it. The problem is that the “un-intended habit” then gets into the core areas as well. In this case, I prefer to push my team to write good code in tests too which helps in bringing the mindset shift to completely move away from cutting corners.

      Finally, as you said and as I started this article with – these two are close (very). It is just like Eclipse and IntelliJ – one always have an edge over the other in some aspects while it would lack intent in other areas.

      And I certainly did not mean to criticize JUnit – it is a good framework – I just wanted to highlight some areas of the same when compared to TestNG and especially “how I perceive” them better. I guess, my summary was a bit too harsh; thanks for the inputs.

  3. 12 November, 2011 at 1:21 am

    I’m also a TestNG user and I would like to add one more feature which I like very in it – test groups. This allows me among other things to separate the fast real unit tests from the the slower unit(?) tests with embedded database (like described in my recent post).

    Btw, when I was in some projects using JUnit I missed data providers very much, but some time ago a frustrated guy from Pragmatists made a library which allows to write parametrized tests in Junit easier – junitparams. Nevertheless it’s not a core functionality and there is a problem when you need to use also another runner in the same test class.

    Marcin

    • 4 January, 2012 at 1:36 pm

      Hi Marcin – For the very need of parametrized testing, I have been frustrated enough that I have been spending my free time in writing a library that allows me to do just that. I am 80% done and hope that 2012 would be the year when I will make it public.

      This is one thing that can save a LOE in many projects and is true to both tools.

  1. 2 September, 2011 at 2:28 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: