LTP result

Introduction

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)

Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:

• to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
• to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
• to be the reference if Yocto users want to get a base picture

There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=169

Overall matrix

Overall matrix tends to track the change of LTP test results along with Yocto's progress.

In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)

Milestone	qemux86	qemu-x86_64	qemuppc	qemuarm	qemumips	atom-pc	blacksand	beagleboard	routerstation
1.0-M2	36	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a

Current Milestone (1.0/M2)

Below table shows overall progress in LTP failures analysis for current milestone:

• total: the number of total failures
• audit: the number of audited failures, which status could be "pending", "fixed", and "Not a Bug"
• fixed: the number of fixed failures
• NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition

(click the title like "qemux86" to check detail data for that target

Status	Qemux86	Qemux86_64	Qemuppc	Qemuarm	Qemumips	netbook	blacksand	beagleboard	routerstation
total	36	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
audit	36	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
fixed	36	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
NAB	36	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a

Below table includes which LTP categories are included in the measurement:

Milestone	qemu targets	real hardware
1.0-M2	syscalls mm ipc sched math nptl pty admin_tools command	syscalls fs fsx dio io mm ipc sched math nptl pty admin_tools timers commands

Current Milestone (1.0-M2)

(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)

This show details info about each failure in a milestone, merged from the results from all machines:

LTP version: ltp-full-20100430.tgz

Profile: poky-image-sdk

run command: ./runltp -p -l M2-sdk-20101216.log 1>&M2-sdk-20101216-full.log

Name	Machines	Profile	Status	Owner	Comments / Bugzilla Links
lchown02	Qemux86	poky-image-sdk	LTP	Kevin Tian	also fail on Ubuntu 9.04
lchown02_16	Qemux86	poky-image-sdk	LTP	Kevin Tian	also fail on Ubuntu 9.04
su01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel
cron02	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
cron_deny01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
cron_allow01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' and 'userdel'
at_deny01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' available
at_allow01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	no 'useradd' available
ldd	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	No 'file' and 'ldd' in sdk image
tar	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	'file' is not available in sdk image
logrotate	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	No 'logrotate'
cpio	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	busybox supports less option (no "-o")
gzip01	Qemux86	poky-image-sdk	Not A Bug	Kevin Tian	busybox supports less option (no "-r")
cron_dirs_checks01	Qemux86	poky-image-sdk	Pending	Kevin Tian	miss many cron specific dirs
chdir01A	Qemux86	poky-image-sdk	Pending	Kevin Tian	symlink error
ioctl03	Qemux86	poky-image-sdk	Pending	Kevin Tian	No /dev/net/tun
splice02	Qemux86	poky-image-sdk	Pending	Kevin Tian	splice failed - errno = 28 : No space left on device
sysctl03	Qemux86	poky-image-sdk	Pending	Kevin Tian	Permission denied
syslog01	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; *.crit not found in /etc/syslog.conf
syslog02	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no mail.emerg in /etc/syslog.conf
syslog03	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no daemon.info in /etc/syslog.conf
syslog04	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info in /etc/syslog.conf
syslog05	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info in /etc/syslog.conf
syslog07	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.debug in /etc/syslog.conf
syslog08	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.info
syslog09	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages; no user.debug
syslog10	Qemux86	poky-image-sdk	Pending	Kevin Tian	no /var/log/messages
utimensat01	Qemux86	poky-image-sdk	Pending	Kevin Tian	chattr: Inappropriate ioctl for device; permission denied
mtest06_2	Qemux86	poky-image-sdk	Pending	Kevin Tian	mmap(): Cannot allocate memory
mmapstress06	Qemux86	poky-image-sdk	Pending	Kevin Tian	errno = 12; large mmap failed
clock_gettime03	Qemux86	poky-image-sdk	Pending	Kevin Tian	get unexpected error number
timer_create04	Qemux86	poky-image-sdk	Pending	Kevin Tian	get unexpected error number
ld	Qemux86	poky-image-sdk	Pending	Kevin Tian	No 'file' available
file	Qemux86	poky-image-sdk	Pending	Kevin Tian	mixed error message. need further check
cron	Qemux86	poky-image-sdk	Pending	Kevin Tian	no crontab; no /var/log/messages
unzip01	Qemux86	poky-image-sdk	Pending	Kevin Tian	output message differs. 'inflating' while 'extracting' is expected. Perhaps a LTP issue

Previous milestone

a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.