LTP result: Difference between revisions
m (→Overall trend) |
(→result) |
||
| Line 51: | Line 51: | ||
|| total || 1072 || n/a || 1071 || 1071 || 1081 || n/a || n/a || 1174 || 1182 | || total || 1072 || n/a || 1071 || 1071 || 1081 || n/a || n/a || 1174 || 1182 | ||
|- | |- | ||
|| failures || 34 || 32 || 34 || 37 || 37 || 33 || 29 || | || failures || 34 || 32 || 34 || 37 || 37 || 33 || 29 || 35 || 145 (*) | ||
|- | |- | ||
|| '''similarity''' || '''100%''' || '''90%''' || '''88%''' || '''78%''' || '''78%''' || '''90%''' || '''96%''' || ''' | || '''similarity''' || '''100%''' || '''90%''' || '''88%''' || '''78%''' || '''78%''' || '''90%''' || '''96%''' || '''85%''' || '''19%''' | ||
|- | |- | ||
|| audit || 18(53%) || 0(0%) || 0(0%) || 0(0%) || 0(0%) || 0(0%) ||0(0%) || 0(0%) || 0(0%) | || audit || 18(53%) || 0(0%) || 0(0%) || 0(0%) || 0(0%) || 0(0%) ||0(0%) || 0(0%) || 0(0%) | ||
| Line 63: | Line 63: | ||
|| LTP || 3 || 0 || 0 || 0 || 0 || 0 || 0 || 0 || 0 | || LTP || 3 || 0 || 0 || 0 || 0 || 0 || 0 || 0 || 0 | ||
|} | |} | ||
* on routerstation kernel may not be configured correctly. Lots of new failures are related to IPC, which may come from lacking of some kernel options | |||
===configuration=== | ===configuration=== | ||
Revision as of 07:01, 18 January 2011
Introduction
Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)
Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:
- to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
- to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
- to be the reference if Yocto users want to get a base picture
There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=169
Overall trend
Overall trend tends to track the change of LTP test results along with Yocto's progress. Previous data in 0.9 is not complete and not consistent configured on all targets. So we start from 1.0-M2 as the starting point.
In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)
| Milestone | qemux86 | qemu-x86_64 | qemuppc | qemuarm | qemumips | atom-pc | blacksand | beagleboard | routerstation |
| 1.0-M2 | 34 | 32 | 34 | 37 | 37 | 33 | 29 | 35 | 172 (*) |
*: routerstation has kernel config issues. See later for detail.
general tips:
- Yocto links /tmp to /var/volatiles/tmp which is dynamically created at each boot. This causes one issue to LTP test. /var is normally not writable to non-root users, while LTP does run some test cases under non-root accounts. This causes considerable failures (~60 in qemux86) when creating temporary contents under /tmp. The workaround to this issue is to use "-d newtmpdir" when running LTP, where newtmpdir is created with full write permissions to all users.
- There's a puzzle which profile should be chosen as the test target. Now SDK is chosen because same image can work on both qemu and real hardware. LSB profile provides better result which is more desktop like but now this profile doesn't run on real hardware and lack of some development packages like gcc. For now we just keep this choice, and introduce a Not-A-Bug type to mark failures which are caused simply because related packages are missing
Current Milestone (1.0/M2)
environment
LTP version: ltp-full-20101031.bz2
Profile: poky-image-sdk
run command: ../runltp -p -q -l result-M2-20101218.log -C result-M2-20101218.fail -d /opt/tmp
result
Below table shows overall progress in LTP failures analysis for current milestone:
- total: the number of total test cases
- similarity: most failures are common to all targets. Qemux86 is chosen as the base, which is the target spent with most time
- failures: the number of failures
- audit: the number of audited failures, which status could be "fixed", "LTP" and "Not a Bug"
- fixed: the number of fixed failures
- LTP: native failures
- NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition
(click the title like "qemux86" to check detail data for that target
| Status | Qemux86 | Qemux86_64 | Qemuppc | Qemuarm | Qemumips | netbook | blacksand | beagleboard | routerstation |
| total | 1072 | n/a | 1071 | 1071 | 1081 | n/a | n/a | 1174 | 1182 |
| failures | 34 | 32 | 34 | 37 | 37 | 33 | 29 | 35 | 145 (*) |
| similarity | 100% | 90% | 88% | 78% | 78% | 90% | 96% | 85% | 19% |
| audit | 18(53%) | 0(0%) | 0(0%) | 0(0%) | 0(0%) | 0(0%) | 0(0%) | 0(0%) | 0(0%) |
| fixed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| NAB | 15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| LTP | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
* on routerstation kernel may not be configured correctly. Lots of new failures are related to IPC, which may come from lacking of some kernel options
configuration
Below table includes which LTP categories are included in the measurement:
| Milestone | qemu targets | real hardware |
| 1.0-M2 |
syscalls mm ipc sched math nptl pty admin_tools command |
syscalls fs fsx dio io mm ipc sched math nptl pty admin_tools timers commands |
Previous milestone
a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.
