Ptest

Introduction

Ptest (package test) is a concept for building, installing and running the test suites that are included in many packages, and producing a consistent output format.

Adding ptest to your build

Ptest is enabled in your build by adding "ptest" to the DISTRO_FEATURES variable and "ptest-pkgs" to IMAGE_FEATURES of your image (or EXTRA_IMAGE_FEATURES in your local.conf). This will cause ptest-enabled packages to build and install the test suite in /usr/lib/<package>/ptest.

Running ptest

The "ptest-runner" package installs a "ptest-runner" shell script which loops through all installed ptest test suites and runs them in sequence.

Adding ptest support to a package

What constitutes a ptest?

A ptest must at minimum contain two things: run-ptest and the actual test.

run-ptest is a minimal shell script that starts the test suite. Note: It must not contain the test suite, only start it!

The test can be anything, from a simple shell script running a binary and checking its output to an elaborate system of test binaries and data files.

One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:

  result: testname

Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string. (The same format used by Automake.)

General recipe preparations

First, add "ptest" to the "inherit" line in the package recipe.

If a test adds build-time or run-time dependencies to the package which are not there normally (such as requiring "make" to run the test suite), add those with a -ptest suffix, like this:

  RDEPENDS_${PN}-ptest += "make"

Building the test suite

Few packages support cross-compiling their test suites, so this is something we typically have to add.

Many automake-based packages compile and run the test suite in a single command: "make check". This doesn't work when cross-compiling (since we need to build on host and run on target), so we need to split that into two targets: One for building the test and one for running it. Our build of automake comes with a patch which does this, so packages using the plain "make check" arrangement from automake get this automatically.

Now add a do_compile_append function to build the test suite:

  do_compile_append() {
     if [ "${PN}" = "${BPN}" -a ${PTEST_ENABLED} = "1" ]; then
        oe_runmake buildtest-TESTS
     fi
  }

Installing the test suite

The ptest.bbclass contains a ptest_do_install function which copies the required "run-ptest" file and runs "make install-ptest" if there is such a target in the top-level Makefile. This provides a standardized install method to use in the package do_install_append function:

  do_install_append() {
     ptest_do_install
  }