View source for Ptest

=Introduction=

Ptest (''package test'') is a concept for building, installing and running the test suites that are included in many packages, and producing a consistent output format.

=Adding ptest to your build=

To add package testing to your build, add the DISTRO_FEATURES and EXTRA_IMAGE_FEATURES variables to your local.conf file, which is found in the Build Directory:

     DISTRO_FEATURES_append = " ptest"
     * Adding "ptest" to your DISTRO_FEATURES variable causes -ptest packages to be built.

     EXTRA_IMAGE_FEATURES += "ptest-pkgs"
     * Adding "ptest-pkgs" to IMAGE_FEATURES variable causes -ptest packages to be installed in your image.

All ptest files are installed in /usr/lib/<package>/ptest.

=Running ptest=

The "ptest-runner" package installs a "ptest-runner" which loops through all installed ptest test suites and runs them in sequence. You may therefore want to add "ptest-runner" to your image.

==Ptest-runner Usage==

The usage for the ptest-runner is as follows:

<pre>Usage: ptest-runner [-d directory] [-l list] [-t timeout] [-h] [ptest1 ptest2 ...] </pre>

==Executing Ptest-runner==

To execute a full ptest run you can do the following command line anywhere in the target:

<pre> $ ptest-runner </pre>

Remember to redirect to a file if you want to have a log

=Implementing ptest in a package recipe=

==What constitutes a ptest?==

A ptest must at minimum contain two things: run-ptest and the actual test.

run-ptest is a minimal shell script that ''starts'' the test suite. Note: It must not ''contain'' the test suite, only start it!

The test can be anything, from a simple shell script running a binary and checking its output to an elaborate system of test binaries and data files.

One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:

   result: testname

Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string. (The same format [http://www.gnu.org/software/automake/manual/automake.html#Simple-Tests used by Automake].)

==General recipe preparations==

First, add "ptest" to the "inherit" line in the package recipe.

If a test adds build-time or run-time dependencies to the package which are not there normally (such as requiring "make" to run the test suite), add those with a -ptest suffix, like this:

   RDEPENDS_${PN}-ptest += "make"

==Building the test suite==

Few packages support cross-compiling their test suites, so this is something we typically have to add.

Many automake-based packages compile and run the test suite in a single command: "make check". This doesn't work when cross-compiling, since we need to build on host and run on target. So we need to split that into two targets: One for building the test and one for running it. Our build of automake comes with a [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/automake/automake/buildtest.patch patch] which does this automatically, so packages using the plain "make check" arrangement from automake get this automatically.

Now add a do_compile_ptest function to build the test suite:

   do_compile_ptest() {
      oe_runmake buildtest-TESTS
   }

If the package requires special configuration actions prior to compiling the test code, create a do_configure_ptest function to do that.

==Installing the test suite==

The ptest.bbclass will automatically copy the required "run-ptest" file and run "make install-ptest" if there is such a target in the top-level Makefile. For packages where this is enough, you don't need to do anything else.

If you need to do something more than the automatic "make install-ptest", create a do_install_ptest function and put the required actions there. This function will be called after "make install-ptest" has completed.

==Recipes with ptest Enabled==

The following is a list of recipes already have ptest enabled:
*acl
*attr
*apr
*apr-util
*bash
*beecrypt
*diffstat
*busybox
*bzip
*dbus
*ethtool
*flex
*gawk
*glib
*kbd
*kmod
*libxml2
*libpcre
*lttng-tools
*parted
*perl
*python
*quilt
*strace
*systemd
*sed
*tcl
*openssh
*openssl
*udev
*gdbm
*mdadm
*rt-tests
*sed
*lzo
*zlib
*diffutils
*e2fsprogs
*libevent

==Recipes in Progress==

The following list is being worked on by various people, to avoid duplication, please add any recipes you might be working on here, once they are merged we can move them to the list above.

*dhcp
*dropbear
*gcc-runtime
*gdb
*gettext
*guile
*libuser
*tar
*xz


Please consider when updating a recipe whether you can also incorporate a ptest package to it as well.