LTP result: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 72: Line 72:
|| fixed || 0 || 0 || 0 ||  || 0 || 0 || 0 ||
|| fixed || 0 || 0 || 0 ||  || 0 || 0 || 0 ||
|-
|-
|| NAB || '''34''' || '''35''' || '''35''' ||  || '''36''' || '''35''' || '''36''' ||  
|| NAB || '''34''' || '''35''' || '''35''' ||  || '''36''' || '''37''' || '''36''' ||  
|}
|}



Revision as of 06:28, 8 December 2011

Introduction

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. (http://ltp.sourceforge.net)

Unlike typical desktop/server distributions, embedded linux distribution is highly customized for a specific purpose. It's not unusual to observe more failures in Yocto than in a typical desktop Linux distro like Ubuntu, etc. So the major purposes of this work are:

  • to understand the reasons for existing LTP failures. If the cause is from a specific package itself, go to fix it. Or else if the error simply come from the fact that a given package doesn't exist in current profile, add the explanation here
  • to track LTP test results cross Yocto releases, and ensure a declining error number for same profile
  • to be the reference if Yocto users want to get a base picture

Overall trend

Overall trend tends to track the change of LTP test results along with Yocto's progress. Previous data in 0.9 is not complete and not consistent configured on all targets. So we start from 1.0-M2 as the starting point.

In below table we list total LTP failures for each architectures in each milestone: (now this wiki doesn't allow upload files. So for detail logs please refer to earlier bug link)

Milestone sugarbay qemux86 qemu-x86_64 qemuppc qemuarm qemumips atom-pc blacksand beagleboard routerstation mpc8315e-rdb
1.0.2 n/t n/t 35 35 35 n/t 36 137 (*) 36
1.1-M4 RC4 35 n/t n/t 38 39 37 n/t 36 41 43 39
1.1-M4 RC3 37 n/t n/t 37 38 37 n/t 37 41 44 39
1.1-M4 RC2 26 n/t n/t 38 39 37 n/t 26 43 59 39
1.1-M3 27 n/t n/t 35 37 36 n/t 28 41 57 42
1.1-M2 30 n/t n/t 25 29 26 n/t 28 33 146 (*) 31
1.1-M1 33 n/t n/t 26 29 26 n/t 30 34 146 (*) 32
1.0-M2 34 32 34 37 37 33 29 35 145 (*)
*: routerstation has kernel config issues. See later for detail.

general tips:

  • Yocto links /tmp to /var/volatiles/tmp which is dynamically created at each boot. This causes one issue to LTP test. /var is normally not writable to non-root users, while LTP does run some test cases under non-root accounts. This causes considerable failures (~60 in qemux86) when creating temporary contents under /tmp. The workaround to this issue is to use "-d newtmpdir" when running LTP, where newtmpdir is created with full write permissions to all users.
  • There's a puzzle which profile should be chosen as the test target. Now SDK is chosen because same image can work on both qemu and real hardware. LSB profile provides better result which is more desktop like but now this profile doesn't run on real hardware and lack of some development packages like gcc. For now we just keep this choice, and introduce a Not-A-Bug type to mark failures which are caused simply because related packages are missing
    • It's better to use a tmp dir from the 2nd disk which contains ltp test suite. This way it avoids some noisy disk space limitation errors
  • suggest to use "-m 512" as some tests are memory stress
  • suggest to disable hackbench from runtest/sched, which is not suitable to run in emulator, and simply hang
  • don't use "-q" option when running ltp, which hides many useful output info

Current Milestone (1.0.2)

environment LTP version: ltp-full-20110606.bz2

profile: poky-image-sato-sdk

run command: ./runltp -p -l result-20111201.log -C result-20111201.fail -d /opt/ltp/tmp &> result-20111201.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1104 1105 1104 1206 1206 1206
failures 35 35 35 36 137 36
fixed 0 0 0 0 0 0
NAB 34 35 35 36 37 36

Configuration

Below table includes which LTP categories are included in the measurement:

Milestone qemu targets real hardware
1.0.2

syscalls

mm

ipc

sched

math

nptl

pty

admin_tools

command

syscalls

fs

fsx

dio

io

mm

ipc

sched

math

nptl

pty

admin_tools

timers

commands

Previous Milestone

a placeholder. Once we reach a new milestone, the data under "current milestone" will be moved here.

1.1 M4 RC4

environment LTP version: ltp-full-20110606.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M4-20110914.log -C result-M4-20110914.fail -d /opt/ltp/tmp &> result-M4-20110914.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1104 1105 1104 1208 1206 1206 1206 1208
failures 38 39 37 36 41 43 39 35
fixed 0 0 0 0 0 0 0 0
NAB 37 38 36 36 36 37 37 35

1.1 M4 RC3

environment LTP version: ltp-full-20110606.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M4-20110914.log -C result-M4-20110914.fail -d /opt/ltp/tmp &> result-M4-20110914.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1104 1105 1104 1208 1206 1206 1206 1203
failures 37 38 37 37 41 44 39 37
fixed 1 1 0 0 2 16 0 0
NAB 37 38 36 37 36 37 37 37

1.1 M4 RC2

environment LTP version: ltp-full-20110606.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M4-20110914.log -C result-M4-20110914.fail -d /opt/ltp/tmp &> result-M4-20110914.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1104 1105 1104 1180 1206 1206 1206 1180
failures 38 39 37 26 43 59 39 26
fixed 0 0 0 5 0 0 6 1
NAB 37 38 36 26 38 43 37 26

1.1 M3

environment LTP version: ltp-full-20110606.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M3-20110806.log -C result-M3-20110806.fail -d /opt/ltp/tmp &> result-M3-20110806.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1104 1105 1104 1180 1207 1206 1206 1180
failures 35 37 36 28 41 57 42 27
fixed 0 3 3 0 3 104 0 9
NAB 34 36 32 23 35 51 36 27

1.1 M2

environment LTP version: ltp-full-20101031.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M2-20110711.log -C result-M2-20110711.fail -d /opt/ltp/tmp &> result-M2-20110711.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1078 1078 1080 1182 1180 1179 1180 1180
failures 25 29 26 28 33 146 31 30
fixed 1 0 0 2 1 2 1 3
NAB 23 25 23 23 24 23 24 24

1.1 M1

environment LTP version: ltp-full-20101031.bz2

profile: core-image-sato-sdk

run command: ./runltp -p -l result-M1-20110528.log -C result-M1-20110528.fail -d /opt/ltp/tmp &> result-M2-20110528.fulllog

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemuarm" to check detail data for that target

Status Qemuppc Qemuarm Qemumips blacksand beagleboard routerstation mpc8315e-rdb sugarbay
total 1078 1078 1078 1182 1180 1179 1180 1180
failures 26 29 26 30 34 146 32 33
fixed 0 0 0 0 0 0 0 0
NAB 23 25 23 4 25 23 25 0
LTP 0 0 0 0 0 0 0 0
Duplicated 0 0 0 29 5 5 5 35
pending 0 3 3 3 3 3 0 0

1.0 M4

environment LTP version: ltp-full-20101031.bz2

profile: poky-image-sdk

run command: ../runltp -p -l result-M4-20110318.log -C result-M4-20110318.fail -d /opt/tmp

result Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemux86" to check detail data for that target

Status Qemux86 Qemux86_64 Qemuppc Qemuarm Qemumips netbook blacksand beagleboard routerstation Jasperforest (lsb-sdk) mpc8315e-rdb
total 1072 n/a 1071 1071 1081 n/a n/a 1174 1182 1198
failures 34 32 34 37 37 33 29 35 145 (*) 20
audit 100% 100% 100% 100% 100% 100% 100% 100% 0(0%) 0(0%)
fixed 6 0 0 1 0 0 0 0 0 0
NAB 26 1 1 4 1 2 1 0 0 0
LTP 2 2 0 0 1 0 0 0 0
Duplicated 0 29 33 29 29 31 28 35 0 13
pending 0 0 0 3 0 0 0 0 0
similarity 100% 90% 88% 78% 78% 90% 96% 85% 19%
* on routerstation kernel is not configured correctly. Lots of new failures 
  are related to IPC, which may come from lacking of some kernel options. This
  has been verified with newer 2.6.37 kernel recently, that only ~60 failures
  remaining. So the analysis is skipped for it in this cycle.

1.0 M2

environment LTP version: ltp-full-20101031.bz2

profile: poky-image-sdk

run command: ../runltp -p -q -l result-M2-20101218.log -C result-M2-20101218.fail -d /opt/tmp

Below table shows overall progress in LTP failures analysis for current milestone:

  • total: the number of total test cases
  • failures: the number of failures
  • audit: the number of audited failures, which status could be:
    • fixed: the number of fixed failures
    • LTP: native failures
    • NAB: Not-A-Bug, the number of the failures which come from miss of some packages due to the profile definition, or can't reproduce locally
    • pending: the analysis is done with bug reported, but pending for final solution
    • duplicated: the number of failures which are same as on other platforms which have been analyzed already
  • similarity: similarity to Qemux86 which is chosen as the base. most cases are common on all targets

(click the title like "qemux86" to check detail data for that target

Status Qemux86 Qemux86_64 Qemuppc Qemuarm Qemumips netbook blacksand beagleboard routerstation
total 1072 n/a 1071 1071 1081 n/a n/a 1174 1182
failures 34 32 34 37 37 33 29 35 145 (*)
audit 100% 100% 100% 100% 100% 100% 100% 100% 0(0%)
fixed 6 0 0 1 0 0 0 0 0
NAB 26 1 1 4 1 2 1 0 0
LTP 2 2 0 0 1 0 0 0 0
Duplicated 0 29 33 29 29 31 28 35 0
pending 0 0 0 3 0 0 0 0 0
similarity 100% 90% 88% 78% 78% 90% 96% 85% 19%
* on routerstation kernel is not configured correctly. Lots of new failures 
  are related to IPC, which may come from lacking of some kernel options. This
  has been verified with newer 2.6.37 kernel recently, that only ~60 failures
  remaining. So the analysis is skipped for it in this cycle.