Posix result
Introduction
The Open POSIX* Test Suite is an open source test suite with the goal of performing conformance, functional, stress, and performance testing of the functions described in the IEEE Std 1003.1-2001 System Interfaces specification. Eventual testing of the full specification is desired (http://posixtest.sourceforge.net/)
Similar to [LTP result], embedded linux distribution is highly customized for a specific purpose, and thus both the kernel and base environment may not satisfy the posix requirement. So the major purposes of this work are:
- to understand the reasons for existing posix failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
- to track poix test results cross Yocto releases, and ensure a declining error number for same profile
- to be the reference if Yocto users want to get a base picture
There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=172
Overall trend
Overall trend tends to track the change of posix test results along with Yocto's progress.
In below table we list total POSIX failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)
| Milestone | qemux86 | qemu-x86_64 | qemuppc | qemuarm | qemumips | atom-pc | sugarbay | blacksand | beagleboard | routerstation | mpc8315e-rdb | huronriver |
| 1.3 M1 RC1 20120613 (****) | n/a | n/a | n/a | n/a | n/a | n/a | 125 | 125 | 126 | 129 | 126 | 127 |
| 1.3 20120606 build(****) | n/a | n/a | n/a | n/a | n/a | n/a | 126 | 125 | 125 | 126 | 126 | n/a |
| Denzil 1.2 RC5(****) | n/a | n/a | n/a | n/a | n/a | n/a | 128 | 125 | 126 | 129 | 126 | n/a |
| Denzil 1.2 RC4(****) | n/a | n/a | n/a | n/a | n/a | n/a | 128 | 126 | 126 | 128 | 127 | n/a |
| 1.2 M4 RC3(****) | n/a | n/a | n/a | n/a | n/a | n/a | 127 | 128 | 125 | n/a | 126 | n/a |
| 1.2 M4 RC2(****) | n/a | n/a | n/a | n/a | n/a | n/a | 127 | 125 | 127 | 130 | 126 | n/a |
| 1.2 M4 RC1(****) | n/a | n/a | n/a | n/a | n/a | n/a | 129 | 126 | 127 | 130 | 126 | n/a |
| 1.2 M3 RC1(****) | n/a | n/a | n/a | n/a | n/a | n/a | 130 | 127 | 129 | 132 | 126 | n/a |
| 1.1.1 | n/a | n/a | n/a | n/a | n/a | n/a | 10 | 9 | 31 | 33 | 32 | n/a |
| 1.2 M2 RC1(****) | n/a | n/a | n/a | n/a | n/a | n/a | 130 | 128 | 131 | 132 | 128 | n/a |
| 1.2 M1 RC2(***) | n/a | n/a | 90 | 92 | 92 | n/a | 98 | 98 | 91 | 92 | 89 | n/a |
| 1.0.2 | n/a | n/a | 35 | 32 | 33 | n/a | 30 | 71 | 34 | n/a | ||
| 1.1 M4 RC4 | n/a | n/a | 31 | 33 | 32 | n/a | 25 | 25 | 30 | 31 | 32 | n/a |
| 1.1 M4 RC3 | n/a | n/a | 32 | 35 | 33 | n/a | 24 | 23 | 37 | 33 | 31 | n/a |
| 1.1 M4 RC2 | n/a | n/a | 31 | 34 | 34 | n/a | 25 | 24 | 31 | 33 | 32 | n/a |
| 1.1 M3 | n/a | n/a | 32 | 34 | 34 | n/a | 23 | 24 | 30 | 32 | 31 | n/a |
| 1.1 M2 | n/a | n/a | 30 | 34 | 32 | n/a | 31 | 26 | 32 | 69 | 33 | n/a |
| 1.1 M1(**) | n/a | n/a | 35 | 49 | 43 | n/a | 50 | 48 | 46 | 84 | 49 | n/a |
| 1.0-M2 (*) | 1744 | 1744 | 1745 | 1750 | 1753 | 1753 | n/a | 1744 | 1755 | 1754 | n/a | n/a |
*: 1.0-M2 misses some link flags, and thus many unexpected link failures observed **: 1.1 M1 has gcc upgrade from 4.5.1 to 4.6.1, which has two new options, -Wunused-but-set-variable and -Waddress enabled by default. They casused many cases failed. We need add -Wno-unused-but-set-variable and -Wno-address to fix the problems. ***: For 1.2, We using POSIX test suite in LTP release 20110915 instead of POSIX test suite 1.5.2. ****: For 1.2 M2, We also test AIO.
general tips:
- use "-m 512" to launch the qemu targets
- make sure below line is uncommented in LDFLAGS file:
#-D_XOPEN_SOURCE=600 -lpthread -lrt -lm
- don't test AIO as the test cases themselves have compilation warnings which are treated as errors (use 'exit' while doesn't include stdlib.h)
Result analysis
Current results are base on Yocto 1.1 and POSIX test suite in LTP release 20110915. After analysis every test case except PASS will be list here, whether a patch will be gave to fix it or put the case to blacklist with a explanation.
POSIX test results: https://wiki.yoctoproject.org/wiki/POSIX-results
Current Milestone (Yocto 1.3 M1 RC1 20120613 build)
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 125 | 125 | 126 | 126 | 129 |
Previous Milestone
Yocto 1.3 M1 RC1 20120613 build
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation | huronriver |
| failures | 125 | 125 | 126 | 126 | 129 | 127 |
Yocto 1.3 20120606 build
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 126 | 125 | 126 | 125 | 126 |
Yocto 1.2 Denzil RC5
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 128 | 125 | 126 | 126 | 129 |
Yocto 1.2 Denzil RC4
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 128 | 126 | 127 | 126 | 128 |
Yocto 1.2 M4 RC3
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 127 | 128 | 125 | n/a | 126 |
Yocto 1.2 M4 RC2
environment OpenPOSIX version: posixtestsuite in LTP release 20120401
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 127 | 125 | 126 | 127 | 130 |
Yocto 1.2 M4 RC1
environment OpenPOSIX version: posixtestsuite in LTP release 20120104
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 129 | 126 | 126 | 127 | 130 |
Yocto 1.2 M3 RC1
environment OpenPOSIX version: posixtestsuite in LTP release 20120104
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 130 | 127 | 126 | 129 | 132 |
Yocto 1.1.1
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 10 | 9 | 32 | 31 | 33 |
Yocto 1.2 M2 RC1
environment OpenPOSIX version: posixtestsuite in LTP release 20110915
profile: core-image-sato-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | mpc8315e-rdb | beagleboard | routerstation |
| failures | 128 | 130 | 128 | 131 | 132 |
Yocto 1.2 M1
environment OpenPOSIX version: posixtestsuite in LTP release 20110915
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 98 | 98 | 90 | 92 | 92 | 89 | 91 | 92 |
Yocto 1.0.2
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: poky-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 35 | 32 | 33 | 34 | 30 | 71 |
Yocto 1.1 M4 RC4
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 25 | 25 | 31 | 33 | 32 | 32 | 30 | 31 |
Yocto 1.1 M4 RC3
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 24 | 23 | 32 | 35 | 33 | 31 | 37 | 33 |
Yocto 1.1 M4 RC2
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 25 | 24 | 31 | 34 | 34 | 32 | 31 | 33 |
Yocto 1.1 M3
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 32 | 34 | 34 | 31 | 30 | 32 |
Yocto 1.1 M2
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 31 | 26 | 30 | 34 | 32 | 33 | 32 | 69 |
Yocto 1.1 M1
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: core-image-sato-sdk
run command: ./run_tests [MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | sugarbay | blacksand | qemuppc | qemuarm | qemumips | mpc8315e-rdb | beagleboard | routerstation |
| failures | 50 | 48 | 35 | 49 | 43 | 49 | 46 | 84 |
Yocto 1.0 M2
environment OpenPOSIX version: posixtestsuite-1.5.2.tar
profile: poky-image-sdk
run command: ./run_tests [AIO|MEM|MSG|SEM|SIG|THR|TMR|TPS]
result
| Status | qemux86 | qemu-x86_64 | qemuppc | qemuarm | qemumips | atom-pc | blacksand | beagleboard | routerstation |
| failures | 1744 | 1744 | 1745 | 1750 | 1753 | 1753 | 1744 | 1755 | 1754 |
The major problem for high failure numbers here is due to one environmental issue. It's suggested to uncomment below line from LDFLAGS file:
#-D_XOPEN_SOURCE=600 -lpthread -lrt -lm
libpthread and librt are critical for posix interface test. Without doing this the majority of the failures are link failures being failed to find necessary libraries.
My quick test on qemux86 shows that the failure could be reduced a lot to <200 in a rough run
Based on above fact, no more analysis is done on 1.0-M2 posix results. The thorough analysis will be postponed to 1.0-M3 which will use the right test steps and newer 2.6.37 kernel
