p-unit - An open source framework for performance benchmark and unit test

p-unit An open source framework for unit test and performance benchmark, which was initiated by Andrew Zhang, under Apache License v2.0. p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time consumption, and generates the result in the form of plain text, image or pdf file.
Sourceforge homepage: https://sourceforge.net/projects/p-unit
Download: p-unit release
Contact info: zhanghuangzhu AT gmail DOT com

Features

Version 0.15

Improved compatibility with JUnit4 test cases.
p-unit does not support multiple VM runner any more.

Version 0.14

Refactored the core code. p-unit core is now based on J2SE5.
Some minor bug fixes.

p-unit 0.13 for Android

A new runner AndroidRunner for Android platform.
The usage of p-unit keeps all the same, except to use AndroidRunner(runner) to execute your test!
Example:
  The follwoing code executes MyTest with SoloRunner on PC:
  public static void main(String[] args) {
     SoloRunner runner = new SoloRunner();
     runner.run(MyTest.class);
   }

  The follwoing code executes MyTest with AndroidRunner on Android:
   AndroidRunner runner = new AndroidRunner(new SoloRunner());
   runner.run(MyTest.class);

Version 0.13

p-unit changes to adopt Apache License 2.0
Apache Harmony community is evaluting p-unit as the framework for performance test:

Added a customized watcher infrastructure to make it possible to report results from test cases.
Added a new EventListener which sets public test fields on start up.

Version 0.12

Sample Code Available in the Distribution
JUnit 4.x Compatabile
Better Annotation Support

Version 0.11

Annotation support
Check method

Version 0.10

Multiple executor pool

Version 0.10 dev

The differences between p-unit and junit
Test Suite and Test Class
Time/Memory Record
Concurrent
Parameterizable
Runner
Image/PDF reporter
Benchmark against different virtual machines
Some other nice features

Sample Code Available in the Distribution

p-unit is very simple, but quite powerful. For better understanding the functionality of p-unit, a sample project is included in the distribution. You can also check out the sample project from the svn respo(https://p-unit.svn.sourceforge.net/svnroot/p-unit).

The expected learning curve for each feature is less than five minutes.

JUnit 4.x compatible

p-unit can run JUnit 4.x test cases without any modification of the source code! A new convention "JUnitAnnotationConvention" is introduced in version 0.12. To run existing JUnit 4.x test case, you only need to set the convention as JUnitAnnotationConvention:

public static void main(String[] args) {
   SoloRunner runner = new SoloRunner();
   runner.setConvention(new JUnitAnnotationConvention());
   runner.run(JUnitAnnotationSample.class);
}

Supported JUnit annotation includes:

@Test(expected = SomeException.class)
@Before
@After
@BeforeClass
@AfterClass

For more details, please refer to the sample.JUnitAnnotationSample in the sample project.

p-unit Annotation

p-unit has its own defined annotation. It's very similar as JUnit 4.x convention, in order to provide better compatibility with your existing code. Developer should not suffer the pain to remember different annotation for different test tools. p-unit annotation includes:

@Test(expected = SomeException.class, checkMethod = "someMethod")
@Before
@After
@BeforeClass
@AfterClass

All of them are defined in org.punit.annotation package. As you see, you don't need to pay extra effort to learn about p-unit annotation except that check method property is p-unit unique.

Annotation Support

By default, p-unit detects the test methods by name convention, say, starts with "test". p-unit also supports to use annotation to identify the test method, and expected exception. For example, You can use following annotation:

@Test(expected = NullPointerException.class, checkMethod = "checkAdd")

To enable annotation support, you need to write one more line code in the main method:

runner.setConvention(new AnnotationConvention());

Check Method

It's optional, but very useful sometimes. After executing a test method, p-unit invokes the corresponding check method if such a method is available. The name convention is "check_" prefix. Of course, p-unit also supports to use annotation to mark the check method, i.e.

@Test(checkMethod="checkTestAdd")

To understand why this feature is useful, especially for concurrent test case, consider: how to test Vector#add is thread-safe? I hope following code will give you the answer:

private Vector _vector = new Vector();

@Test(checkMethod="checkAdd")
public void testAdd() {
_vector.add(new Object());
}

public void checkAdd() {
Assert.areEquals(10, _vector.size());
}

Multiple Executor Pool

p-unit supports to use an executor pool to execute the test suite. It fully takes advantages of multi-core machine, and even benefits single core machine too. The granulity is test class, which means every test class should be independent of each other. A simple example to explain this concept:

Given that there are two test classes, each of them has four methods, and every method sleeps five seconds. If you can this test suite in JUnit, the execution would be around 2 * 4 * 5 = 40 seconds. About in p-unit, by writing following code, the execution time would be reduced to around 20s.

/* -------------Executor Pool Sample--------------- */

public static void main(String[] args) {
   SoloRunner runner = new SoloRunner();
   runner.setExecutorPool(new ExecutorPoolImpl(2));
   runner.run(LongTimeExecutionPUnitTestSuite.class);
}

The differences between p-unit and junit

What are the differences between p-unit and junit? junit is a well known unit test harness which focuses on the correctness, while p-unit focuses on the infrastructure for performance benchmark. But p-unit can also used as the unit test harness. p-unit supports to run junit format test, and it supports some more special functionalities.

Test Suite and Test Class

Test suite and test class are two important concepts in p-unit. p-unit does not require any special type for a test class, so every class can be a test class. Of course, it includes junit test cases. There is one special interface for special p-unit only test class - p-unitTest. p-unit executes a normal test as following procedure:

invoke setUp if there is.

invoke the test method.

invoke tearDown method.

Noticed that setUp and tearDown will also be regarded as a part of performance test. If you don't want to count them into performance data, you need to implement p-unitTest interface, which will be executed as:

invoke setUpBeforeWatchers.

invoke setAfterWatchers.

invoke the test method.

invoke tearDownBeforeWatchers.

invoke tearDownAfterWatchers.

As the name suggests, you can put the setUp and tearDown code into the setUpBeforeWatchers and tearDownAfterWatchers.

Time/Memory Record

p-unit records the memory and time consumption of running a test method. There is a "watcher" concept in p-unit, which supervises the status during running the test method. By default, memory watcher and time watcher are installed. p-unit supports user-defined watcher too. The user only needs to implement the Watcher interface and register it to p-unit method runner. By default, time watcher is enabled. If you want to watch memory consumption, you need to add one line code:

runner.methodRunner().addWatcher(new MemoryWatcher());

Concurrent

p-unit supports to run the test concurrently. How to write it? You DON'T need to write anything about concurrent in your test - just the same as a normal test. You only need to use ConcurrentRunner to run your test concurrently. For more detail, please refer to Runner.

Parameterizable

Performance test usually requires to run the same test against different parameters. For example, you may be interested in the performance of inserting 1000/10000/100000 objects to java.util.ArrayList. p-unit supports to write parameterizable test method. Following sample will be executed three times with different user-defined parameters.

/* -------------Parameterizable sample--------------- */

public class ArrayListTest implements Parameterizable {
  public void testInsert(ArrayListParameter param) {
    int count = param.getArrayListCount();
    // do some stuff
  }

  public void parameters(ArrayListParameter param) {
    int count = param.getArrayListCount();
  }

  public Parameter[] parameters() {
     return new Parameter[] { new ArrayListParameter(1000), new ArrayListParameter(10000), new ArrayListParameter(100000) };
  }
}

class ArrayListParameter implements Parameter {
   private int _count;
   ArrayListParameter(int count) {
     _count = count;
   }
   public int getArrayListCount() {
     return _count;
   }
   public String toString() {
     return ""+_count;
   }
}

Runner

Runenr is the core concept of p-unit. You need a runner to run your test. By default, there are two different runners - solo runner and concurrent runner, so that one test can be executed solo and concurrently. Is it complex to launch a runner? ONE line code:

new SoloRunner().run(MyTest.class); or new ConcurrentRunner().run(MyTest.class);

By default, ConcurrentRunner starts up 10 threads to execute the test method. You can configure it in the constructor of ConcurrentRunner(int threadCount), and you can even do fine control in your test class. If you need to execute different test class with different thread count, you only need to implement Concurrent interface for your test, which requires to return concurrentCount.

Image/PDF Reporter

p-unit core is event based architecture, so that it accepts external reporter. Currently p-unit supports console, plain text file (p-unit core), Image and PDF reporter(p-unit extension). By default, only console logger is turned on. You can use following code to enable some build-in reporters:

/* -------------Reporter sample--------------- */

Runner runner = new SoloRunner();
// enable file output
runner.addEventListener(new FileLogger());
// enable image reporter which is based on TestSuite
runner.addEventListener(new TestSuiteReporter(new ImageRender()));
// enable image reporter which is Overview
runner.addEventListener(new OverviewReporter(new PDFRender()));

There are three grains for Image/PDF reporters - Overview/TestSuite/TestClass. The reporter chart includes the data of:

Overview - All test data in one chart

TestSuite - One test suite data in one chart

TestClass - One test class data in one chart

The result files are stored at ./result folder. The following reports are the performance charts for jdk ArrayList/Vector#add against DRLVM and SUN jre (see the code in the next chapter):

Benchmark against different virtual machines

Yes, it is an exciting feature of p-unit! You can run the same test against different virtual machines only by writing following code:

/* -------------Race against diffrent virtual machine sample--------------- */

SoloRunner runner = new SoloRunner();
runner.addp-unitEventListener(new OverviewReporter(new ImageRender()));
runner.runVMs(ListTestClass.class, new VM[] { VMConfig.HARMONY, VMConfig.SUN });

public class VMConfig {
   private static String CLASSPATH = " -cp somedir"; //$NON-NLS-1$
   private static String HARMONY_PATH = "harmony jre\\java" + CLASSPATH; //$NON-NLS-1$
   private static String SUN_PATH = "sun jre\\java" + CLASSPATH; //$NON-NLS-1$
   public static HARMONY = new VM(HARMONY_PATH, "Harmony"); //$NON-NLS-1$
   public static SUN = new VM(SUN_PATH, "SUN"); //$NON-NLS-1$
}

Some other nice features

Alphabetical interface

It ensures that the test methods will be executed alphabetically.