TestNG

TestNG is a java testing framework. It was created by Cédric Beust out of frustration over JUnit's deficiencies. It was designed to cover all categories of tests such as unit, functional, integration, etc.

The idea is very simple. It is breakdown into the following steps:

  1. Create your class.
  2. Create test scenarios by using methods of your class and then you have to implement the logic to check whether the methods behave correctly or not. If they do, then tell TestNG that it is good. Otherwise, tell it it is bad.

At the end, TestNG will report whether methods behave correctly or not.

First TestNG test

Write TestNG test

Writing tests using TestNG is easy. There are only 2 steps:

  1. Add @Test annotation to your methods so that TestNG can recognize that the methods are in fact test methods.
  2. In your test methods, use TestNG's assertion methods(e.g. assertEquals(), assertTrue(), assertNotNull(), etc) to validate the expected results.

The code below demontrates how to use TestNG to verify the correctness of the methods. In this case, it will check whether add() method is correctly returning the sum of 2 integers. There are 2 test methods implemented below. The first test method, testAddPositive(), checks whether it correctly adds 2 positive values. The second test method, testAddNegative(), checks whether it correctly adds 2 negative values. Obviously, you can continue to add as much test methods with different test cases as you want.

/**
 * AddTest.java
 * Show how to use TestNG.
 * @author Xuan Ngo
 */
 
import org.testng.annotations.Test;           // Import @Test annotattion.
import static org.testng.Assert.assertEquals; // Import assertEquals() method.
 
public class AddTest
{
  @Test(description="Testing add() method with positive values.")
  public void testAddPositive()
  {
    int iActual = this.add(2, 3);
    int iExpected = 5;
    assertEquals(iActual, iExpected, "add() doesn't return expected positive value.");
  }
 
  @Test(description="Testing add() method with negative values.")
  public void testAddNegative()
  {
    int iActual = this.add(-2, -3);
    int iExpected = -5;
    assertEquals(iActual, iExpected, "add() doesn't return expected negative value.");
  }
 
  /**
   * I put the implementation of add() here so that the whole test class is self-complete.
   * In real life, this method should come from another class so that there is 
   * a complete separation between the test class and the actual class.
   */
  public int add(int a, int b)
  {
    return a+b;
  }
}

Compile TestNG test class

Execute the following command to compile code above. Note: Change the paths according to the locations of your files.

javac -classpath c:\testng-5.8-jdk15.jar AddTest.java

Execute TestNG test class

Execute the following command to run test. Note: Change the paths according to the locations of your files.

java -classpath c:\testng-5.8-jdk15.jar;. org.testng.TestNG -testclass AddTest

TestNG's results

The result on the Command Prompt will look like the following:

[Parser] Running:
  Command line suite
===============================================
Command line suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================

The result shows that there were 2 tests ran: 0 test failed and 0 test skipped. Therefore, all tests were ran successfully. By default, a detailed report will be created by TestNG in test-output/ directory. Open test-output/index.html with your browser to see the complete report.

Execution order of TestNG's annotations

Codes below show the execution order of commonly used annotations of TestNG.

package net.xngo.tutorial.java.testng;
 
/**
 * AnnotationsExecutionOrder.java
 * Show execution order of commonly used annotations of TestNG
 * Author: Xuan Ngo
 */
 
/*
 // OUTPUT:
Ran Constructor.(Work Time = 477 ms)
Ran @BeforeTest method.(Work Time = 1976 ms)
Ran @BeforeClass method.(Work Time = 1651 ms)
Ran @BeforeMethod method.(Work Time = 671 ms)
Ran @Test method.(Work Time = 236 ms)
Ran @AfterMethod method.(Work Time = 1403 ms)
Ran @AfterClass method.(Work Time = 979 ms)
Ran @AfterTest method.(Work Time = 964 ms)
 */
import org.testng.annotations.BeforeTest;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterTest;
 
import java.util.Random;
 
public class AnnotationsExecutionOrder
{
  // Variables
  Random oRandom = new Random();
  final int MAX = 2000;
 
  // Constructor
  public AnnotationsExecutionOrder()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran Constructor.(Work Time = " + iRnd + " ms)");
  }
 
  @BeforeClass
  public void BeforeClass()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @BeforeClass method.(Work Time = " + iRnd + " ms)");
  }
 
  @BeforeTest
  public void BeforeTest()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @BeforeTest method.(Work Time = " + iRnd + " ms)");
  }
 
  @BeforeMethod
  public void BeforeMethod()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @BeforeMethod method.(Work Time = " + iRnd + " ms)");
  }
 
  @Test
  public void Test()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @Test method.(Work Time = " + iRnd + " ms)");
  }
 
  @AfterMethod
  public void AfterMethod()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @AfterMethod method.(Work Time = " + iRnd + " ms)");
  }
 
  @AfterTest
  public void AfterTest()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @AfterTest method.(Work Time = " + iRnd + " ms)");
  }
 
  @AfterClass
  public void AfterClass()
  {
    final int iRnd = oRandom.nextInt(MAX);
    this.pause(iRnd);
    System.out.println("Ran @AfterClass method.(Work Time = " + iRnd + " ms)");
  }
 
  // Simulate some processing time by pausing.
  private void pause(long lPauseInMillisSec)
  {
    try
    {
      Thread.sleep(lPauseInMillisSec);
    }
    catch (Exception ex)
    {
      System.out.println("Can't sleep.");
    }
 
  }
}

https://github.com/xuanngo/Tutorial/blob/master/src/net/xngo/tutorial/java/testng/AnnotationsExecutionOrder.java

http://testng.org/doc/documentation-main.html#annotations

Run specific groups of tests

One of the nice features of TestNG is that it allows you to run specific groups of tests among all the tests that you have written. In my case, this feature is handy when I want to run read only tests on the production environment. The idea is to associate each test method to a group. When you want to run each group of tests separately, you only invoke your desired groups through the configuration file of TestNG. The following shows you how to do it.

  1. To associate a test to a group, simply specify the group name in the groups attribute of the @Test annotation.
    @Test(groups={"readonly"})
    public void myTest1(){ /*...*/ }
    You can also associate a test to multiple groups by using commas as shown below:
    @Test(groups={"readonly", "fast"})
    public void myTest2(){ /*...*/ }
  2. To choose which groups you would like to run, simply specify the name of the groups in the configuration file(testng.xml) of TestNG. In my case, I would like to run readonly test methods. My configuration file looks like this:
    <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
    <suite>
      <test>
        <groups>
          <run>
            <!-- List which groups to run from all the tests included.-->
            <include name="readonly"/>
          </run>
        </groups>
     
        <!-- Included all tests that you would like to run. -->
        <classes>
          <class name="my.classes.to.run" />
        </classes>
     
      </test>
    </suite>
    myTest1() and myTest2() will run. Note: In the configuration file, besides defining which groups to run, you must include all the methods/classes/packages that you would like to run. Otherwise, no test will be run if you are only specifying the groups and nothing else.

Testing private methods

Sometimes you need to test the private method of a class. I use reflection to access the private method. Here are the codes:

/**
 * Class with a private method that you want to test.
 * @author Xuan Ngo
 *
 */
public class ClassExample
{
  private String privateFoo(int a, String b)
  {
    return a+b;
  }
}
/**
 * Here is the code showing how to access the private method
 *  using reflection.
 * @author Xuan Ngo
 */
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
 
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
 
@Test(sequential=true)
public class ClassExampleTest
{
  @Test
  public void privateFooTest() throws NoSuchMethodException, IllegalAccessException, InvocationTargetException
  {
    // Get the class of the private method.
    ClassExample oClassExample = new ClassExample();
    Class<?> cNewClassExample = oClassExample.getClass();
 
    // Change the property of the private method to be accessible.
    Method newPrivateFoo = cNewClassExample.getDeclaredMethod("privateFoo", int.class, String.class);
    newPrivateFoo.setAccessible(true);
 
    // Run the private method.
    Object oActual = newPrivateFoo.invoke(oClassExample, new Integer(169), new String("_ABC"));
 
    // Test the private method
    String sActual = oActual.toString();
    String sExpected = "169_ABC";
    assertEquals(sActual, sExpected);
 
  }
}
 
/*
Note: When calling, watch out for the exceptions, especially InvocationTargetException:
 
IllegalAccessException: You're not allowed to call this method
 
IllegalArgumentException: You can call this method, but not with the supplied arguments
 
InvocationTargetException(Anything the invoked method throws is wrapped by InvocationTargetException.): You called the method just fine, but it threw an exception
 
*/
AttachmentSize
File ClassExample.java196 bytes
File ClassExampleTest.java1.13 KB

Ant build file for TestNG

Sample Ant build file to run TestNG

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Filename: build.xml
  Note: You have to change the followings according to your environment:
          -<pathelement location="lib/testng/testng-5.14.7.jar"/>
          -<pathelement location="bin"/>
-->
<project basedir="." default="runTestNG" name="Sample of Ant file for TestNG">
 
  <!-- Define <testng> task -->
  <taskdef name="testng" classname="org.testng.TestNGAntTask">
    <classpath>
      <pathelement location="lib/testng/testng-5.14.7.jar"/>
    </classpath>
  </taskdef>
 
  <!-- Directory name where the TestNG report will be saved. -->
  <property name="testng.output.dir" value="testng_output"/>
 
  <!-- Directory path of compiled classes(i.e *.class) -->
  <path id="classes">
     <pathelement location="bin"/>
  </path>
 
  <!--
  Target to run TestNG. It will run according to what are defined in testng.xml.
  The report will be saved at .../testng_output/index.html.
  -->
  <target name="runTestNG">
 
    <mkdir dir="${testng.output.dir}"/><!-- Create the output directory. -->
 
    <testng outputdir="${testng.output.dir}" classpathref="classes" failureProperty="test.failure">
      <jvmarg value="-DMy.App.System.Property=123" />
      <xmlfileset dir="." includes="testng.xml"/> 
    </testng>
    <fail if="test.failure" message="Not all tests passed!" />
  </target>
 
</project>



Sample TestNG configuration file

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Filename: testng.xml
  Note: You have to change the following according to your environment:
          -<class name="com.packageName.MyTestClassName" />  
-->
<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="My Test Suite">
 
  <test name="My Test">
 
    <!--  Add all classes you would like TestNG to run. -->  
    <classes>
      <class name="com.packageName.MyTestClassName" />
    </classes>
 
  </test>
 
</suite>



Sample Structure of my files and directories created by Eclipse

C:.
¦   .classpath
¦   .project
¦   build.xml
¦   testng.xml
¦
+---bin
¦   +---com
¦       +---packageName
¦               MyTestClassName.class
¦
+---lib
¦   +---testng
¦           testng-5.14.7.jar
¦
+---src
¦   +---com
¦       +---packageName
¦               MyTestClassName.java
¦
+---testng_output
    ¦   emailable-report.html
    ¦   index.html
    ¦   testng-results.xml
    ¦   testng.css
    ¦
    +---junitreports
    ¦       TEST-com.packageName.MyTestClassName.xml
    ¦
    +---My Test Suite
            classes.html
            groups.html
            index.html
            main.html
            methods-alphabetical.html
            methods-not-run.html
            methods.html
            My Test.html
            My Test.properties
            My Test.xml
            reporter-output.html
            testng.xml.html
            toc.html

Ant build using TestNG-XSLT

Ant build using TestNG-XSLT

<!-- 
  Example for TestNG-XSLT
    Run this target only after TestNG had generated the xml resutl file(i.e. testng_output/testng-results.xml).
    Change TestNG-XSLT stylesheet path accordingly(i.e. lib/testng-xslt-0.6.2/src/main/resources/testng-results.xsl).
-->
<target name="testng_nice_report">
  <mkdir dir="testng_nice_report"/>
  <xslt in="testng_output/testng-results.xml" 
        style="lib/testng-xslt-0.6.2/src/main/resources/testng-results.xsl"
        out="testng_nice_report/index.html">
    <param name="testNgXslt.outputDir" expression="${basedir}/testng_nice_report"/>
    <param name="testNgXslt.showRuntimeTotals" expression="true"/>
    <classpath refid="MyClasspath"/>
  </xslt>
</target>