LTP result
Introduction
Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)
Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:
- to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
- to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
- to be the reference if Yocto users want to get a base picture
There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=169
Overall matrix
Overall matrix tends to track the change of LTP test results along with Yocto's progress.
In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)
| Milestone | qemux86 | qemu-x86_64 | qemuppc | qemuarm | qemumips | atom-pc | blacksand | beagleboard | routerstation |
| 1.0-M2 | 36 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
Current Milestone (1.0/M2)
Below table shows overall progress in LTP failures analysis for current milestone:
- total: the number of total failures
- audit: the number of audited failures, which status could be "pending", "fixed", and "Not a Bug"
- fixed: the number of fixed failures
- NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition
(click the title like "qemux86" to check detail data for that target
| Status | Qemux86 | Qemux86_64 | Qemuppc | Qemuarm | Qemumips | netbook | blacksand | beagleboard | routerstation |
| total | 36 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| audit | 36 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| fixed | 36 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| NAB | 36 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
Below table includes which LTP categories are included in the measurement:
| Milestone | qemu targets | real hardware |
| 1.0-M2 |
syscalls mm ipc sched math nptl pty admin_tools command |
syscalls fs fsx dio io mm ipc sched math nptl pty admin_tools timers commands |
Current Milestone (1.0-M2)
(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)
This show details info about each failure in a milestone, merged from the results from all machines:
LTP version: ltp-full-20100430.tgz
Profile: poky-image-sdk
run command: ./runltp -p -l M2-sdk-20101216.log 1>&M2-sdk-20101216-full.log
| Name | Machines | Profile | Status | Owner | Comments / Bugzilla Links |
| lchown02 | Qemux86 | poky-image-sdk | LTP | Kevin Tian | also fail on Ubuntu 9.04 |
| lchown02_16 | Qemux86 | poky-image-sdk | LTP | Kevin Tian | also fail on Ubuntu 9.04 |
| su01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' and 'userdel |
| cron02 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' and 'userdel' |
| cron_deny01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' and 'userdel' |
| cron_allow01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' and 'userdel' |
| at_deny01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' available |
| at_allow01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | no 'useradd' available |
| ldd | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | No 'file' and 'ldd' in sdk image |
| tar | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | 'file' is not available in sdk image |
| logrotate | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | No 'logrotate' |
| cpio | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | busybox supports less option (no "-o") |
| gzip01 | Qemux86 | poky-image-sdk | Not A Bug | Kevin Tian | busybox supports less option (no "-r") |
| cron_dirs_checks01 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | miss many cron specific dirs |
| chdir01A | Qemux86 | poky-image-sdk | Pending | Kevin Tian | symlink error |
| ioctl03 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | No /dev/net/tun |
| splice02 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | splice failed - errno = 28 : No space left on device |
| sysctl03 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | Permission denied |
| syslog01 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; *.crit not found in /etc/syslog.conf |
| syslog02 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no mail.emerg in /etc/syslog.conf |
| syslog03 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no daemon.info in /etc/syslog.conf |
| syslog04 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no user.info in /etc/syslog.conf |
| syslog05 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no user.info in /etc/syslog.conf |
| syslog07 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no user.debug in /etc/syslog.conf |
| syslog08 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no user.info |
| syslog09 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages; no user.debug |
| syslog10 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no /var/log/messages |
| utimensat01 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | chattr: Inappropriate ioctl for device; permission denied |
| mtest06_2 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | mmap(): Cannot allocate memory |
| mmapstress06 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | errno = 12; large mmap failed |
| clock_gettime03 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | get unexpected error number |
| timer_create04 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | get unexpected error number |
| ld | Qemux86 | poky-image-sdk | Pending | Kevin Tian | No 'file' available |
| file | Qemux86 | poky-image-sdk | Pending | Kevin Tian | mixed error message. need further check |
| cron | Qemux86 | poky-image-sdk | Pending | Kevin Tian | no crontab; no /var/log/messages |
| unzip01 | Qemux86 | poky-image-sdk | Pending | Kevin Tian | output message differs. 'inflating' while 'extracting' is expected. Perhaps a LTP issue |
Previous milestone
a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.
