LSB Result

Introduction

The Linux Standard Base(LSB) was created to lower the overall costs of supporting the Linux platform. By reducing the differences between individual Linux distributions, the LSB greatly reduces the costs involved with porting applications to different distributions, as well as lowers the cost and effort involved in after-market support of those applications. (http://www.linuxfoundation.org/collaborate/workgroups/lsb)

LSB Test Suite support the following architecture.

• i486
• ia64
• ppc32
• ppc64
• s390
• s390x
• x86_64

So LSB don't support arm and mips at present time and it will add test suite to support arm and mips possibly in the future.

Test Environment

• Image: core-image-lsb-qt3
• Platform: x86-32(emenlow), x86-64(jasperforest), ppc(mpc8315e-rdb)

result

Below table shows the result of LSB Test on different platforms. X86 result is updated in 2011-06-03.

Test	Version	X86-32(emenlow)	X86-64(jasperforest)	ppc(mpc8315e-rdb)
Command Check	v 4.1.0-1	Success		Success
Library Check	v 4.1.0-1	Failures: 1		Warning
Alsa-T2C Tests	v 4.1.0-1	Success		Success
Core Tests	v 4.1.0-1	Failures: 10		Failures: 66
Core-T2C Tests	v 4.1.0-1	Success		Success
Cpp-T2C Tests	v 4.1.0-1	Success		Success
Desktop Tests	v 4.1.0-2	Failures: 18		Failures: 24
Desktop-T2C Tests	v 4.1.0-1	Failures: 3		Failures: 17
Libstdc++ Tests	v 4.1.0-13	Success		Failures: 2
OLVER Core Tests	v 4.1.0-1	Failures: 10		Block
Perl Tests	v 4.1.0-1	Failures: 1		Failures: 1
Printing Tests	v 4.1.0-1	Failures: 1		Failures: 3
Python Runtime Tests	v 4.1.1-2	Success		Success
Qt3-Azov Tests	v 4.1.0-1	Success		Failures: 106
Qt4-Azov Tests	v 4.1.0-1	Failures: 5		Failures: 46
Xml2-Azov Tests	v 4.1.0-1	Success		Success
Xts5 Tests	v 5.1.5-38	Warnings: 8		Failures: 2
Apache Tests	v 2.2.14-3	Success
Expect Tests	v 5.43.0-11	Success		Success
Groff Tests	v 1.20.1-5	Success		Success
Raptor Tests	v 1.4.19-3	Success		Success
Rsync Tests	v 3.0.6-3	Success		Success
Samba Tests	v 3.4.3-5	Success		Success
Tcl Tests	v 8.5.7-6	Success		Success
Xpdf Tests	v 1.01-10	Success		Failures: 1

Analysis of result

Below table shows the failed test cases.

• Total failures: The number of total failures after running test. It includes known problems and real failures which should be fixed.
• Known problem: Known problem means it is a bug of LSB itself and we can find trace these bugs on http://bugs.linuxbase.org. On link page we give a summary of known problems.
• Failures(need to fix) : Real failures we should fix them

x86-32 (emenlow,update(2011-06-03))

We have 9 failures to fix.

• Library Check 1
• Libstdc++ Tests 6
• Desktop Tests 1
• Desktop-T2C Tests 1

Test	Total failures	Known problem	Failures(need to fix)	Note
Core Tests	10	10
Libstdc++ Tests	6	0	6	New problem
Library Check	1	0	1	New problem
Desktop Tests	18	17	1	New problem
Desktop-T2C Tests	3	2	1	New problem
OLVER Core Tests	10	10
Perl Tests	1	1
Printing Tests	1	1
Qt4-Azov Tests	5	5

• New problem: It is a new bug without arising at previous version. It could be caused by some patches updated every day.

ppc(mpc8315e)

We have the following failures(including "unresolved" and "failed") to fix.

Test	Total failures	Known problem	Failures(need to fix)	Note
Core Tests	66	10	47(unresolved), 8(failed), 1(unreported)
Desktop Tests	24	20	2(unresolved), 1(failed), 1 unreported
Desktop-T2C Tests	17	2	15(unresolved)
Libstdc++ Tests	2		2
OLVER Core Tests				Blocked
Perl Tests	1	1
Printing Tests	3	1	1(unresolved), 1(failure), 1(unreported)
Qt3-Azov	106		106(unresolved)
Qt4-Azov	46	1	45(unresolved)
Xts5 Tests	2		2
Xpdf Tests	1		1

• Left memory：
  • There are only 20M left when system start up without running any other program except for command "free -m".
• Illegal instruction
  • 65 SIGILL error
    • Core Tests
      • /tset/ANSI.os/math/... : SIGILL cause 47 unresolved failures.
      • /tset/LSB.os/libm/fpeInvalid/T.fpeInvalid (1st--2nd)
    • Desktop-t2c Tests
      • /glib-t2c/tests/glib_random/glib_random (9th--23rd)
    • Xpdf Tests
      • pdftotext
  • It was caused by the difference of compilation tool between LSB Test Suite and mpc8315e. In fact, instruction "fsqrt" doesn't exist on e300core serial and CPU of mpc8315e belongs to e300core type. So this problem can't be solved for mpc8315e
• OLVER Core Tests:
  • OLVER Core Tests can't be finished on mpc8315e-rdb
  • We think the reason which make test blocked is the low memory(128M on mpc8315e). Because we re-run the test case which is blocked during OLVER core tests and it can pass successfully. That is to say, if we run the single test case, it will be passed. But we run all OLVER core tests and the whole tests will be blocked. We have ever run OLVER Core Tests for nearly 50 hours and it can't finish.
  • lsb-test-olver-core may require about 512Mb of system memory (with an X server and a contemporary desktop running), as well as 2Gb of free disk space to complete a test run. The reference linkage: http://ldn.linuxfoundation.org/lsb/preparing-lsb-distribution-testing
• Qt3-Azov Tests
  • Out of memory error cause all of the failures, the following information arise on screen when running Qt3 test suite

   Killed process 22453 (Xvfb) total-vm:84064kB, anon-rss:73720kB, file-rss:8kB
   run-parts invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0,
   oom_score_adj=0
   Call Trace:
   [c0cadc90] [c0009f3c] show_stack+0x108/0x1c4 (unreliable)
   [c0cadce0] [c0431098] dump_stack+0x2c/0x44
   [c0cadcf0] [c00a1880] dump_header.clone.14+0x70/0x184
   [c0cadd30] [c00a1b80] oom_kill_process.clone.16+0x94/0x294
   [c0cadd60] [c00a222c] out_of_memory+0x270/0x33c
   [c0caddb0] [c00a6628] __alloc_pages_nodemask+0x624/0x65c
   [c0cade50] [c00a6694] __get_free_pages+0x34/0x6c
   [c0cade60] [c00269f8] copy_process+0xe4/0xd84   
   [c0cadec0] [c0027764] do_fork+0x80/0x39c
   [c0cadf20] [c0009cb4] sys_vfork+0x68/0x80
   [c0cadf40] [c0011d0c] ret_from_syscall+0x0/0x38       
     --- Exception: c00 at 0xfecc358
     LR = 0x10084518

• • I think this should be limited memory problem
• Libstdc++ Tests
  • /26_numerics/complex/13450.cc
    • This test case check whether data type "long double" can't be supported or not on ppc. Mp8315e deviates from what the test expects. It is Known problem.
  • /27_io/basic_stringbuf/overflow/wchar_t/1.cc
    • No enough memory for this test case. It can pass on qemuppc with 256M and fail with 128M.

Failures tracker(update 2011-06-03)

We should fix the following problems. We also filed bugs for them and try to fix them.

Tests	Sub Tests	Test Number	Arch	Bug ID of yocto	Proposal suggestion
Core Tests	/tset/ANSI.os/maths/*	47	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1114	No instruction "fsqrt" for e300core
Core Tests	/tset/LSB.os/libm/fpeInvalid/T.fpeInvalid	2	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1114	No instruction "fsqrt" for e300core
Core Tests	/tset/LI18NUX2K.L1/base/iswcntrl/T.iswcntrl 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1115
Core Tests	/tset/LI18NUX2K.L1/base/iswdigit/T.iswdigit 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1115
Core Tests	/tset/LI18NUX2K.L1/base/iswlower/T.iswlower 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1115
Core Tests	/tset/LI18NUX2K.L1/base/iswupper/T.iswupper 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1115
Core Tests	/tset/LI18NUX2K.L1/base/iswxdigit/T.iswxdigit 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1115
Desktop	/tests/functions/GHashTable/GHashTable 5th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1116
Desktop	/tests/functions/SpawnProc/SpawnProc 2 (unresolved)	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1117
Desktop-t2c	/glib-t2c/tests/glib_threads/glib_threads 36th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1119	http://bugs.linuxbase.org/show_bug.cgi?id=3042
Desktop-t2c	/glib-t2c/tests/glib_random/glib_random 9th-23th	14	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1114	No instruction "fsqrt" for e300core
Desktop-t2c	/gobject-t2c/tests/gobject_gtype_instance/gobject_gtype_instance 39th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1120	http://bugs.linuxbase.org/show_bug.cgi?id=3184
Libstdc++ Tests	/27_io/basic_stringbuf/overflow/wchar_t/1.cc 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1121	No enough memory for this test case. It can pass on qemuppc with 256M and fail with 128M
Libstdc++ Tests	/26_numerics/complex/13450.cc 1th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1122	http://bugs.linuxbase.org/show_bug.cgi?id=2447
Printing Tests	/convenience/cupsConvenience 37th (unresolved)	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1123
Xts5 Tests	/tset/Xlib10/kllclnt/Test 2th	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1124
Xts5 Tests	/tset/Xopen/strtkysym/Test	7	x86-32, ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1095	http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=jingdonglu/distro&id=c6dde065b9fafabeda6904c23118a11e99d23e08
Perl Tests	../ext/IPC/SysV/t/ipcsysv.t	294	x86-32, ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1078	Maybe a LSB bug(we have filed it as a bug on LSB bugzilla
Xpdf Tests	/opt/lsb/appbat/bin/pdftotext test1.pdf	1	ppc	http://bugzilla.pokylinux.org/show_bug.cgi?id=1114	No instruction "fsqrt" for e300core

Known problems tracker

See details on Known problems tracker