LTP result: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 31: Line 31:


Below table shows overall progress in LTP failures analysis for current milestone:
Below table shows overall progress in LTP failures analysis for current milestone:
* total: the number of total failures
* total: the number of total test cases
* audit: the number of audited failures, which status could be "pending", "fixed", and "Not a Bug"
* failures: the number of failures
* fixed: the number of fixed failures
* audit: the number of audited failures, which status could be "fixed", "LTP" and "Not a Bug"
* NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition
** fixed: the number of fixed failures
** LTP: native failures
** NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition


(click the title like "qemux86" to check detail data for that target
(click the title like "qemux86" to check detail data for that target
Line 40: Line 42:
|| '''Status''' || '''[[qemux86-ltp|Qemux86]]''' || '''[[qemux86_64-ltp|Qemux86_64]]''' || '''[[qemuppc-ltp|Qemuppc]]''' || '''[[qemuarm-ltp|Qemuarm]]''' || '''[[qemumips-ltp|Qemumips]]''' || '''[[netbook-ltp|netbook]]''' || '''[[blacksand-ltp|blacksand]]''' || '''[[beagleboard-ltp|beagleboard]]''' || [[routerstation-ltp|routerstation]]
|| '''Status''' || '''[[qemux86-ltp|Qemux86]]''' || '''[[qemux86_64-ltp|Qemux86_64]]''' || '''[[qemuppc-ltp|Qemuppc]]''' || '''[[qemuarm-ltp|Qemuarm]]''' || '''[[qemumips-ltp|Qemumips]]''' || '''[[netbook-ltp|netbook]]''' || '''[[blacksand-ltp|blacksand]]''' || '''[[beagleboard-ltp|beagleboard]]''' || [[routerstation-ltp|routerstation]]
|-
|-
|| total || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|| total || 1071 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|-
|| failures || 36 || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|-
|-
|| audit || 36(100%) || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a
|| audit || 36(100%) || n/a || n/a || n/a || n/a || n/a || n/a || n/a || n/a

Revision as of 02:30, 12 January 2011

Introduction

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)

Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:

  • to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
  • to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
  • to be the reference if Yocto users want to get a base picture

There is one bugzilla entry related: http://bugzilla.pokylinux.org/show_bug.cgi?id=169

Overall trend

Overall trend tends to track the change of LTP test results along with Yocto's progress. Previous data in 0.9 is not complete and not consistent configured on all targets. So we start from 1.0-M2 as the starting point.

In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)

Milestone qemux86 qemu-x86_64 qemuppc qemuarm qemumips atom-pc blacksand beagleboard routerstation
1.0-M2 36 n/a n/a n/a n/a n/a n/a n/a n/a

Current Milestone (1.0/M2)

LTP version: ltp-full-20100430.tgz

Profile: poky-image-sdk

run command: ../runltp -p -q -l result-M2-20101218.log -C result-M2-20101218.fail -d /opt/tmp

Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be "fixed", "LTP" and "Not a Bug"
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition

(click the title like "qemux86" to check detail data for that target

Status Qemux86 Qemux86_64 Qemuppc Qemuarm Qemumips netbook blacksand beagleboard routerstation
total 1071 n/a n/a n/a n/a n/a n/a n/a n/a
failures 36 n/a n/a n/a n/a n/a n/a n/a n/a
audit 36(100%) n/a n/a n/a n/a n/a n/a n/a n/a
fixed 36 n/a n/a n/a n/a n/a n/a n/a n/a
NAB 36 n/a n/a n/a n/a n/a n/a n/a n/a
LTP 36 n/a n/a n/a n/a n/a n/a n/a n/a

Below table includes which LTP categories are included in the measurement:

Milestone qemu targets real hardware
1.0-M2

syscalls

mm

ipc

sched

math

nptl

pty

admin_tools

command

syscalls

fs

fsx

dio

io

mm

ipc

sched

math

nptl

pty

admin_tools

timers

commands

Previous milestone

a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.