View source for LSB Result

==Introduction==
The Linux Standard Base(LSB) was created to lower the overall costs of supporting the Linux platform. By reducing the differences between individual Linux distributions, the LSB greatly reduces the costs involved with porting applications to different distributions, as well as lowers the cost and effort involved in after-market support of those applications. (http://www.linuxfoundation.org/collaborate/workgroups/lsb)

LSB Test Suite support the following architecture.
*i486
*ia64
*ppc32
*ppc64
*s390
*s390x
*x86_64
So LSB don't support arm and mips at present time and it will add test suite to support arm and mips possibly in the future.

==Test Environment==
* Image: core-image-lsb-qt3
* Platform: x86-32(emenlow), x86-64(jasperforest), ppc(mpc8315e-rdb)
* Version of LSB Test Suite: 4.1

==result==
Below table shows the result of LSB Test on different platforms.
X86 result is updated in 2011-06-16. Ppc32 result is update in 2011-06-16
{|border="1"
|| '''Test''' || '''Version''' || '''X86-32(emenlow)''' || '''X86-64(jasperforest)''' || '''ppc(mpc8315e-rdb)''' 
|-
|| '''Command Check''' || v 4.1.0-1 || Failures: 1 ||  || Failures: 1 
|-
|| '''Library Check''' || v 4.1.0-1 || Failures: 8 ||  || Failures: 10 
|-
|| '''Alsa-T2C Tests''' || v 4.1.0-1 || Test suite error ||  || Test suite error 
|-
|| '''Core Tests''' || v 4.1.0-1 || Failures: 13 ||  || Failures: 66 
|-
|| '''Core-T2C Tests''' || v 4.1.0-1 || Success ||  || Success 
|-
|| '''Cpp-T2C Tests''' || v 4.1.0-1 || Success ||  || Success 
|-
|| '''Desktop Tests''' || v 4.1.0-2 || Test suite error ||   || Test suite error 
|-
|| '''Desktop-T2C Tests''' || v 4.1.0-1 || Test suite error ||   || Test suite error
|-
|| '''Libstdc++ Tests''' || v 4.1.0-13 || Failures: 6 ||   || Failures: 2 
|-
|| '''OLVER Core Tests''' || v 4.1.0-1 || Failures: 10 ||   || Block 
|-
|| '''Perl Tests''' || v 4.1.0-1 || Failures: 1 ||   || Failures: 1 
|-
|| '''Printing Tests''' || v 4.1.0-1 || Failures: 1 ||   || Failures: 3 
|-
|| '''Python Runtime Tests''' || v 4.1.1-2 || Success ||   || Success 
|-
|| '''Qt3-Azov Tests''' || v 4.1.0-1 || Success ||   || Failures: 106 
|-
|| '''Qt4-Azov Tests''' || v 4.1.0-1 ||Failures: 5 ||   || Failures: 69 
|-
|| '''Xml2-Azov Tests''' || v 4.1.0-1 ||Success ||    || Success 
|-
|| '''Xts5 Tests''' || v 5.1.5-38 || Warnings: 8||   || Warnings: 8
|-
|| '''Apache Tests''' || v 2.2.14-3 || Success ||  ||  Success
|-
|| '''Expect Tests''' || v 5.43.0-11 || Success ||  || Success 
|-
|| '''Groff Tests''' || v 1.20.1-5 || Success ||   || Success 
|-
|| '''Raptor Tests''' || v 1.4.19-3 || Success ||   || Success 
|-
|| '''Rsync Tests''' || v 3.0.6-3 || Success ||   || Success 
|-
|| '''Samba Tests''' || v 3.4.3-5 || Success ||  || Success 
|-
|| '''Tcl Tests''' || v 8.5.7-6 || Success ||  || Success 
|-
|| '''Xpdf Tests''' || v 1.01-10 || Success ||   || Failures: 1 
|}

==Analysis of result==
Below table shows the failed test cases.
* '''Total failures''': The number of total failures after running test. It includes known problems and real failures which should be fixed.
* '''[[LSB_Bug_tracker | Known problem]]''': Known problem means it is a bug of LSB itself and we can find trace these bugs on http://bugs.linuxbase.org. On link page we give a summary of known problems.
* '''Failures(need to fix)''' : Real failures we should fix them
* '''Test suite error''' : Caused by two reasons: one is for absent library needed by test suite. one is that test suite could lose some function which should be included in it.
===x86-32 (emenlow,update(2011-06-16))===
 
* '''New problem''': It is a bug without arising on previous version. It could be caused by some patches updated every day.
{|border="1"
|| '''Test''' || '''Total failures''' || '''Known problem''' || '''Failures(need to fix)''' || '''Note''' 
|-
|| '''Command Check''' || 1 ||  || 1 ||New problem,Absent command
|-
|| '''Alsa-T2C Tests''' || ||  ||  || New problem,Test suite error
|-
|| '''Core Tests''' || 13 || 13 ||  ||      
|-
|| '''Libstdc++ Tests''' || 6 || 6 ||  || 
|-
|| ''' Library Check''' || 8 ||  || 10 ||  New problem,Absent libraries
|-
|| '''Desktop Tests''' ||  ||  ||    ||  New problem,Test suite error
|-
|| '''Desktop-T2C Tests''' || ||  ||  ||  New problem,Test suite error
|-
|| '''OLVER Core Tests''' || 10 || 10 ||   ||  
|-
|| '''Perl Tests''' || 1 || 1 ||  ||   
|-
|| '''Printing Tests''' || 1 || 1 ||  ||  
|-
|| '''Qt4-Azov Tests''' || 5 || 5 ||  ||  
|-
|}

* Some new problems arose for latest lsb test
** Command Check
*** No command "strip" which existed in the previous lsb image. 
** Library Check
*** The following libraries which existed in the previous lsb image are absent
**** libatk-1.0.so.0 ,libgdk-x11-2.0.so ,libcairo.so.2,library libpango-1.0.so,libpangoft2-1.0.so.0,libpangoxft-1.0.so.0 ,library libpangocairo-1.0.so  
** Desktop Tests,Desktop-T2C Tests(Test suite error)
*** It is caused by the above absent libraries because LSB Test Suite can't find the library it want to test.
*  Absent libQtopenGL

===ppc(mpc8315e,update(2011-06-16))===
We have the following failures(including "unresolved" and "failed") to fix. 
{|border="1"
|| '''Test''' || '''Total failures''' || '''Known problem''' || '''Failures(need to fix)''' || '''Note''' 
|-
|| '''Command Check''' || 1 ||  || 1 ||New problem,Absent command
|-
|| '''Alsa-T2C Tests''' || ||  ||  || New problem,Test suite error
|-
|| '''Core Tests''' || 66 || 10 || 47(unresolved), 8(failed), 1(unreported) ||    
|-
|| '''Desktop Tests''' || ||  ||  ||  New problem, test suite error
|-
|| '''Desktop-T2C Tests''' ||  ||  ||  ||  New problem,  test suite error
|-
|| '''Library Check''' || 10 ||  10 ||  || New problem, Absent libraries
|-
|| '''Libstdc++ Tests''' || 2 ||   ||  ||  
|-
|| '''OLVER Core Tests''' ||  ||  ||   || Blocked 
|-
|| '''Perl Tests''' || 1 || 1 ||  ||   
|-
|| '''Printing Tests''' || 3 || 1 || 1(unresolved), 1(failure), 1(unreported)  ||  
|-
|| '''Qt3-Azov''' || 106 ||  ||  ||  No enough memory
|-
|| '''Qt4-Azov''' || 69 || 5 || 64(unresolved) ||  New problem
|-
|| '''Xpdf Tests''' || 1 ||  || 1 ||  
|}
*For new problem, it has others besides the problems emenlow has
** Qt4-Azov Tests
*** "Child process timed out" caused 64 unresolved itmes  
*Illegal instruction(Known problem http://bugs.linuxbase.org/show_bug.cgi?id=3253)
** 65 SIGILL error 
*** Core Tests
**** /tset/ANSI.os/math/... : SIGILL cause 47 unresolved failures.
**** /tset/LSB.os/libm/fpeInvalid/T.fpeInvalid (1st--2nd)
*** Desktop-t2c Tests
**** /glib-t2c/tests/glib_random/glib_random (9th--23rd)
*** Xpdf Tests
**** pdftotext
** It was caused by the difference of compilation tool between LSB Test Suite and mpc8315e. In fact, instruction "fsqrt" doesn't exist on e300core serial and CPU of mpc8315e belongs to e300core type. So this problem can't be solved for mpc8315e  
* OLVER Core Tests:
** OLVER Core Tests can't be finished on mpc8315e-rdb
** We think the reason which make test blocked is the low memory(128M on mpc8315e). Because we re-run the test case which is blocked during OLVER core tests and it can pass successfully. That is to say, if we run the single test case, it will be passed. But we run all OLVER core tests and the whole tests will be blocked. We have ever run OLVER Core Tests for nearly 50 hours and it can't finish.
** lsb-test-olver-core may require about 512Mb of system memory (with an X server and a contemporary desktop running), as well as 2Gb of free disk space to complete a test run. The reference linkage: http://ldn.linuxfoundation.org/lsb/preparing-lsb-distribution-testing
* Qt3-Azov Tests, Printing Tests 
** Out of memory error cause all of the failures, the information like the following arise on screen when running Qt3 and Printing test suite
          
    Killed process 22453 (Xvfb) total-vm:84064kB, anon-rss:73720kB, file-rss:8kB
    run-parts invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0,
    oom_score_adj=0
    Call Trace:
    [c0cadc90] [c0009f3c] show_stack+0x108/0x1c4 (unreliable)
    [c0cadce0] [c0431098] dump_stack+0x2c/0x44
    [c0cadcf0] [c00a1880] dump_header.clone.14+0x70/0x184
    [c0cadd30] [c00a1b80] oom_kill_process.clone.16+0x94/0x294
    [c0cadd60] [c00a222c] out_of_memory+0x270/0x33c
    [c0caddb0] [c00a6628] __alloc_pages_nodemask+0x624/0x65c
    [c0cade50] [c00a6694] __get_free_pages+0x34/0x6c
    [c0cade60] [c00269f8] copy_process+0xe4/0xd84   
    [c0cadec0] [c0027764] do_fork+0x80/0x39c
    [c0cadf20] [c0009cb4] sys_vfork+0x68/0x80
    [c0cadf40] [c0011d0c] ret_from_syscall+0x0/0x38       
      --- Exception: c00 at 0xfecc358
      LR = 0x10084518

** I think this should be  limited memory problem
* Libstdc++ Tests
** /26_numerics/complex/13450.cc
*** This test case check whether data type "long double" can't be supported or not on ppc. Mp8315e deviates from what the test expects. It is Known problem.
** /27_io/basic_stringbuf/overflow/wchar_t/1.cc
*** No enough memory for this test case. It can pass on qemuppc with 256M and fail with 128M.

==Failures tracker(update 2011-06-16)==
We should fix the following problems. We also filed bugs for them and try to fix them.
{|border="1"
||  '''Tests''' || '''Sub Tests''' || '''Test Number''' || '''Arch''' || '''Bug ID of yocto''' || '''Proposal suggestion''' 
|-
|| '''Core Tests''' || /tset/LI18NUX2K.L1/base/iswcntrl/T.iswcntrl 1st  || 1 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1115 ||
|-
|| '''Core Tests''' || /tset/LI18NUX2K.L1/base/iswdigit/T.iswdigit 1st  || 1 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1115 ||
|-
|| '''Core Tests''' || /tset/LI18NUX2K.L1/base/iswlower/T.iswlower 1st  || 1 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1115 ||
|-
|| '''Core Tests''' || /tset/LI18NUX2K.L1/base/iswupper/T.iswupper  1st  || 1 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1115 ||
|-
|| '''Core Tests''' || /tset/LI18NUX2K.L1/base/iswxdigit/T.iswxdigit 1st  || 1 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1115 ||
|-
|| '''Desktop Tests''' || /tests/functions/SpawnProc/SpawnProc 2nd (unresolved) || 2 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1117 ||

|-
|| '''Printing Tests''' || /convenience/cupsConvenience 37th (unresolved) || 37 || ppc || http://bugzilla.pokylinux.org/show_bug.cgi?id=1123 ||
|- 
|| '''Library Check''' || No library libQtOpenGL||  || x86,x86-64,ppc|| ||http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=xiaofeng/opengl&id=1f612db123f367778de2a7a6fafd31b79538f075
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=xiaofeng/opengl&id=f654a8321554458d47d1cab6ae1201c1f60467e3
|-
|| '''Command Check''' ||No command strip ||  || x86,x86-64,ppc ||  || upstream recipes should plus command "strip"  linking to command ${TARGET}-poky-linux-strip
|-
|| '''Library Check''' || No libraries:libatk-1.0.so.0 ,libgdk-x11-2.0.so ,libcairo.so.2,library libpango-1.0.so,libpangoft2-1.0.so.0,libpangoxft-1.0.so.0 ,library libpangocairo-1.0.so||   || x86,x86-64,ppc ||  ||Add packages including these libraries to task-core-lsb.bb or upstream recipes or task list
|-
|| '''Desktop Tests''' || Test suite error||  || x86,x86-64,ppc ||  ||After plussing the above packages, this test item can be test.
|-
|| '''Desktop-T2c Tests''' ||Test suite error ||  || x86,x86-64,ppc ||  ||After plussing the above packages, this test item can be test.
|-
|| '''Alsa-T2C Tests''' ||Test suite error ||  ||x86,x86-64 ppc ||  ||Add package including library "libasound.so.2.0.0 "to task-core-lsb.bb or upstream recipes or task list
|-
|| '''Qt4-Azov Tests''' ||Child process timed out ||  || ppc ||  ||
|- 
|}

==Known problems tracker==
See details on [[LSB_Bug_tracker | Known problems tracker]]