LTP result: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 12: Line 12:
==Overall matrix==
==Overall matrix==


(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)
Overall matrix tends to track the change of LTP test results along with Yocto's progress.


Overall matrix tends to track the change of LTP test results along with Yocto's progress.
In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)
 
{|border="1"
|| '''Milestone''' || '''qemux86''' || '''qemu-x86_64''' || '''qemuppc''' || '''qemuarm''' || '''qemumips''' || '''atom-pc''' || '''blacksand''' || beagleboard || routerstation
|-
|| 1.0-M2 || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|}


For each LTP failure, we define below status:
==Current Milestone (1.0/M2)==
* '''Not start''': not start investigation yet
* '''Pending''': in investigation
* '''LTP''': fail on other distribution too
* '''Fixed''': fixed, which is expected to disappear in next milestone (bug link may be attached in comment)
* '''Not A Bug''': if the failure is simply caused by missing some packages in current profile (the alternative profile may be listed in the comment if it works)


In below table we list total LTP failures for each architectures in each milestone, with the number in parenthesis indicating "Native"/"Fixed"/"Not A Bug number". All log files on a machine is compressed and linked here too. This way the owner of each machine is able to update the column independently.
Below table shows overall progress in LTP failures analysis for current milestone:
* total: the number of total failures
* audit: the number of audited failures, which status could be "pending", "fixed", and "Not a Bug"
* fixed: the number of fixed failures
* NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition


{|border="1"
{|border="1"
|| '''Milestone''' || '''qemux86''' || '''qemuppc''' || '''qemuarm''' || '''qemumips''' || '''atom-pc'''
|| '''Status''' || '''qemux86''' || '''qemu-x86_64''' || '''qemuppc''' || '''qemuarm''' || '''qemumips''' || '''atom-pc''' || '''blacksand''' || beagleboard || routerstation
|-
|| total || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|-
|| audit || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|-
|-
|| 1.0-M2 || 36(2/11/0)|| n/a || n/a || n/a || n/a
|| fixed || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|-
|| NAB || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|}
|}


Below table includes which LTP categories are included in the measurement:
Below table includes which LTP categories are included in the measurement:
{|border="1"
{|border="1"
|| '''Milestone''' || '''qemux86''' || '''qemuppc''' || '''qemuarm''' || '''qemumips''' || '''atom-pc'''
|| '''Milestone''' || '''qemu targets''' || '''real hardware'''
|-
|-
|| 1.0-M2  
|| 1.0-M2  
||  
||  
syscalls
syscalls
mm
ipc
sched
math
nptl
pty
admin_tools
command
||
syscalls
fs
fsx
dio
io


mm
mm
Line 55: Line 93:
timers
timers


command
commands
|| n/a || n/a || n/a || n/a
|}
|}


Revision as of 08:25, 11 January 2011

Introduction

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)

Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:

  • to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
  • to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
  • to be the reference if Yocto users want to get a base picture

There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=169

Overall matrix

Overall matrix tends to track the change of LTP test results along with Yocto's progress.

In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)

Milestone qemux86 qemu-x86_64 qemuppc qemuarm qemumips atom-pc blacksand beagleboard routerstation
1.0-M2 36 n/a n/a n/a n/a n/a n/a n/a n/a

Current Milestone (1.0/M2)

Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total failures
  • audit: the number of audited failures, which status could be "pending", "fixed", and "Not a Bug"
  • fixed: the number of fixed failures
  • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition
Status qemux86 qemu-x86_64 qemuppc qemuarm qemumips atom-pc blacksand beagleboard routerstation
total 36 n/a n/a n/a n/a n/a n/a n/a n/a
audit 36 n/a n/a n/a n/a n/a n/a n/a n/a
fixed 36 n/a n/a n/a n/a n/a n/a n/a n/a
NAB 36 n/a n/a n/a n/a n/a n/a n/a n/a

Below table includes which LTP categories are included in the measurement:

Milestone qemu targets real hardware
1.0-M2

syscalls

mm

ipc

sched

math

nptl

pty

admin_tools

command

syscalls

fs

fsx

dio

io

mm

ipc

sched

math

nptl

pty

admin_tools

timers

commands

Current Milestone (1.0-M2)

(current data is an example taken from 12/11 nightly test, which will be updated after M2 test is finished)

This show details info about each failure in a milestone, merged from the results from all machines:

LTP version: ltp-full-20100430.tgz

Profile: poky-image-sdk

run command: ./runltp -p -l M2-sdk-20101216.log 1>&M2-sdk-20101216-full.log


Name Machines Profile Status Owner Comments / Bugzilla Links
lchown02 Qemux86 poky-image-sdk LTP Kevin Tian also fail on Ubuntu 9.04
lchown02_16 Qemux86 poky-image-sdk LTP Kevin Tian also fail on Ubuntu 9.04
su01 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' and 'userdel
cron02 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' and 'userdel'
cron_deny01 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' and 'userdel'
cron_allow01 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' and 'userdel'
at_deny01 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' available
at_allow01 Qemux86 poky-image-sdk Not A Bug Kevin Tian no 'useradd' available
ldd Qemux86 poky-image-sdk Not A Bug Kevin Tian No 'file' and 'ldd' in sdk image
tar Qemux86 poky-image-sdk Not A Bug Kevin Tian 'file' is not available in sdk image
logrotate Qemux86 poky-image-sdk Not A Bug Kevin Tian No 'logrotate'
cpio Qemux86 poky-image-sdk Not A Bug Kevin Tian busybox supports less option (no "-o")
gzip01 Qemux86 poky-image-sdk Not A Bug Kevin Tian busybox supports less option (no "-r")
cron_dirs_checks01 Qemux86 poky-image-sdk Pending Kevin Tian miss many cron specific dirs
chdir01A Qemux86 poky-image-sdk Pending Kevin Tian symlink error
ioctl03 Qemux86 poky-image-sdk Pending Kevin Tian No /dev/net/tun
splice02 Qemux86 poky-image-sdk Pending Kevin Tian splice failed - errno = 28 : No space left on device
sysctl03 Qemux86 poky-image-sdk Pending Kevin Tian Permission denied
syslog01 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; *.crit not found in /etc/syslog.conf
syslog02 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no mail.emerg in /etc/syslog.conf
syslog03 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no daemon.info in /etc/syslog.conf
syslog04 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no user.info in /etc/syslog.conf
syslog05 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no user.info in /etc/syslog.conf
syslog07 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no user.debug in /etc/syslog.conf
syslog08 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no user.info
syslog09 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages; no user.debug
syslog10 Qemux86 poky-image-sdk Pending Kevin Tian no /var/log/messages
utimensat01 Qemux86 poky-image-sdk Pending Kevin Tian chattr: Inappropriate ioctl for device; permission denied
mtest06_2 Qemux86 poky-image-sdk Pending Kevin Tian mmap(): Cannot allocate memory
mmapstress06 Qemux86 poky-image-sdk Pending Kevin Tian errno = 12; large mmap failed
clock_gettime03 Qemux86 poky-image-sdk Pending Kevin Tian get unexpected error number
timer_create04 Qemux86 poky-image-sdk Pending Kevin Tian get unexpected error number
ld Qemux86 poky-image-sdk Pending Kevin Tian No 'file' available
file Qemux86 poky-image-sdk Pending Kevin Tian mixed error message. need further check
cron Qemux86 poky-image-sdk Pending Kevin Tian no crontab; no /var/log/messages
unzip01 Qemux86 poky-image-sdk Pending Kevin Tian output message differs. 'inflating' while 'extracting' is expected. Perhaps a LTP issue

Previous milestone

a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=LTP_result&oldid=457"