LTP result

Introduction

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)

Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:

• to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
• to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
• to be the reference if Yocto users want to get a base picture

Overall matrix

(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)

Overall matrix tends to track the change of LTP test results along with Yocto's progress.

For each LTP failure, we define below status:

• Not start: not start investigation yet
• Pending: in investigation
• LTP: fail on other distribution too
• Fixed: fixed, which is expected to disappear in next milestone (bug link may be attached in comment)
• Not A Bug: if the failure is simply caused by missing some packages in current profile (the alternative profile may be listed in the comment if it works)

In below table we list total LTP failures for each architectures in each milestone, with the number in parenthesis indicating "Native"/"Fixed"/"Not A Bug number". All log files on a machine is compressed and linked here too. This way the owner of each machine is able to update the column independently.

Milestone	qemux86	qemuppc	qemuarm	qemumips	atom-pc
1.0-M2	36(2/11/0)	n/a	n/a	n/a	n/a

Below table includes which LTP categories are included in the measurement:

Milestone	qemux86	qemuppc	qemuarm	qemumips	atom-pc
1.0-M2	syscalls mm ipc sched math nptl pty admin_tools timers command	n/a	n/a	n/a	n/a

Current Milestone (1.0-M2)

(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)

This show details info about each failure in a milestone, merged from the results from all machines:

LTP version: ltp-full-20100430.tgz

Profile: poky-image-sdk

run command: ./runltp -p -l M2-sdk-20101216.log 1>&M2-sdk-20101216-full.log

Name	Machines	Profile	Status	Owner	Comments / Bugzilla Links
lchown02	Qemux86	poky-image-sdk	LTP	Kevin Tian	also fail on Ubuntu 9.04
lchown02_16	Qemux86	poky-image-sdk	LTP	Kevin Tian	also fail on Ubuntu 9.04
su01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel
cron02	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
cron_deny01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
cron_allow01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
at_deny01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' available
at_allow01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' available
ldd	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	No 'file' and 'ldd' in sdk image
tar	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	'file' is not available in sdk image
logrotate	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	No 'logrotate'
cpio	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	busybox supports less option (no "-o")
gzip01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	busybox supports less option (no "-r")
cron_dirs_checks01	Qemux86	poky-image-sdk	Pending	Kevin Tian	miss many cron specific dirs
chdir01A	Qemux86	poky-image-sdk	Pending	Kevin Tian	symlink error
ioctl03	Qemux86	poky-image-sdk	Pending	Kevin Tian	No /dev/net/tun
splice02	Qemux86	poky-image-sdk	Pending	Kevin Tian	splice failed - errno = 28 : No space left on device
sysctl03	Qemux86	poky-image-sdk	Pending	Kevin Tian	Permission denied
syslog01	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; *.crit not found in /etc/syslog.conf
syslog02	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no mail.emerg in /etc/syslog.conf
syslog03	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no daemon.info in /etc/syslog.conf
syslog04	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info in /etc/syslog.conf
syslog05	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info in /etc/syslog.conf
syslog07	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.debug in /etc/syslog.conf
syslog08	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info
syslog09	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.debug
syslog10	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages
utimensat01	Qemux86	poky-image-sdk	Pending	Kevin Tian	chattr: Inappropriate ioctl for device; permission denied
mtest06_2	Qemux86	poky-image-sdk	Pending	Kevin Tian	mmap(): Cannot allocate memory
mmapstress06	Qemux86	poky-image-sdk	Pending	Kevin Tian	errno = 12; large mmap failed
clock_gettime03	Qemux86	poky-image-sdk	Pending	Kevin Tian	get unexpected error number
timer_create04	Qemux86	poky-image-sdk	Pending	Kevin Tian	get unexpected error number
ld	Qemux86	poky-image-sdk	Pending	Kevin Tian	No 'file' available
file	Qemux86	poky-image-sdk	Pending	Kevin Tian	mixed error message. need further check
cron	Qemux86	poky-image-sdk	Pending	Kevin Tian	no crontab; no /var/log/messages
unzip01	Qemux86	poky-image-sdk	Pending	Kevin Tian	output message differs. 'inflating' while 'extracting' is expected. Perhaps a LTP issue

Previous milestone

a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.