LSB Result

Introduction

The Linux Standard Base(LSB) was created to lower the overall costs of supporting the Linux platform. By reducing the differences between individual Linux distributions, the LSB greatly reduces the costs involved with porting applications to different distributions, as well as lowers the cost and effort involved in after-market support of those applications. (http://www.linuxfoundation.org/collaborate/workgroups/lsb)

LSB Test Suite support the following architecture.

• i486
• ia64
• ppc32
• ppc64
• s390
• s390x
• x86_64

LSB doesn't support arm and mips at present and maybe support in the future. YOCTO will make lsb test before releasing a new version. This test will be finished at M3 of YOCTO 1.2.

Test Environment

• Image: core-image-lsb-qt3
• Platform: x86-32(emenlow), x86-64(jasperforest), ppc(mpc8315e-rdb)
• Version of LSB Test Suite: 4.1

Setup test environment

We setup LSB test environment on emenlow as an example.

• Build the image

1. Download the poky source

$ git clone git://git.pokylinux.org/poky.git

2. Download the intel BSP meta source

$ cd poky
$ git clone git://git.pokylinux.org/meta-intel.git

3. Downloand the qt3 meta source

$ git clone git://git.pokylinux.org/meta-qt3.git

4. Prepare the build environment

$ source oe-init-build-env

5. Edit the conf/bblayers.conf and conf/local.conf

$ vi conf/bblayers.conf 
…...
BBLAYERS = " \ 
  ##COREBASE##/meta \ 
  ##COREBASE##/meta-yocto \ 
  ##COREBASE##/meta-intel \
  ##COREBASE##/meta-intel/meta-emenlow \ 
  ##COREBASE##/meta-qt3 \
“
$ vi conf/local.conf
…...
MACHINE ?= "emenlow"
…... 
DISTRO ?= "poky-lsb 
…...

6. Build

$ bitbake core-image-lsb-qt3

When build successfully, you can find the kernel, rootfs image, rootfs cpio tarball, and modules tarball in build/tmp/deploy/images direcotry.

• Install system

We assume that there is already a default OS on target(You can install the system by following the README.hardware in poky source). In order to install the new system, you need to prepare a free disk partition. If not, you need to create it by running 'fdisk' command. We assume the free disk partition is /dev/sda5.

There are 2 ways to install the rootfs to the target: Using rootfs image or using rootfs cpio tarball.

Using rootfs image:

1. Boot up the emenlow hardware by default mode.

2. Copy core-image-lsb-qt3-emenlow.ext3 to emenlow from host:

$ scp core-image-lsb-qt3-emenlow.ext3 root@emenlow-ip:/

3. Install the core-image-lsb-qt3-emenlow.ext3 to a free disk partition.

emenlow$ mount /dev/sda5 /mnt 
emenlow$ dd if=core-image-lsb-qt3-emenlow.ext3 of=/dev/sda5 
emenlow$ umount /mnt 
emenlow$ e2fsck -f /dev/sda5 
emenlow$ resize2fs /dev/sda5 

Using rootfs cpio tarball:

1. Boot up the emenlow hardware by default mode.

2. Copy core-image-lsb-qt3-emenlow.cpio.gz to emenlow from host:

$ scp core-image-lsb-qt3-emenlow.cpio.gz root@emenlow-ip:/

3. Install the core-image-lsb-qt3-emenlow.cpio.gz to a free disk partition.

emenlow$ gzip -d core-image-lsb-qt3-emenlow.cpio.gz
emenlow$ mount /dev/sda5 /mnt 
emenlow$ cd /mnt; cpio -iudm < ../ core-image-lsb-qt3-emenlow.cpio 
emenlow$ sync; umount /mnt  

Then install the kernel and modified the grub.

1. Copy kernel to emenlow from host. We assume your boot disk partition is /dev/sda1:

$ scp bzImage-emenlow.bin root@emenlow-ip:/
$ mount /dev/sda1 /mnt
$ cp bzImage-emenlow.bin /mnt/boot/bzImage

2. Modify grub menu.lst

$ vi /mnt/boot/grub/menu.lst

add new lines as below:

title LSB 
root  (hd0,0) 
kernel /boot/bzImage root=/dev/sda5 rw

You can also change the default boot sequence.

3. Restart the emenlow

• Install LSB testsuite

You can follow the File:LSB Test.pdf to install and run LSB test.

Result

Below table shows the result of LSB Test on different platforms.

• X86(emenlow) result is updated in 2011-12-30.
• X86-64(sugarbay) result is updated in 2012-01-08.
• Ppc32(mpc8315e-rdb) result is updated in 2012-01-09

Test	Version	X86-32(emenlow)	X86-64(sugarbay)	ppc(mpc8315e-rdb)
Command Check	v 4.1.0-1	Success	Success	Success
Library Check	v 4.1.0-1	Warnings: 49	Warnings: 54	Warnings: 49
Alsa-T2C Tests	v 4.1.0-1	Success	Success	Success
Core Tests	v 4.1.0-1	Failures: 14	Failures: 15	Failures: 68
Core-T2C Tests	v 4.1.0-1	Success	Success	Success
Cpp-T2C Tests	v 4.1.0-1	Success	Success	Success
Desktop Tests	v 4.1.0-2	Failures: 28	Failures: 27	Failures: 27
Desktop-T2C Tests	v 4.1.0-1	Failures: 6	Failures: 24	Failures: 20
Libstdc++ Tests	v 4.1.0-13	Failures: 132	Failures: 121	Failures: 152
OLVER Core Tests	v 4.1.0-1	Failures: 9	Failures: 10	Block
Perl Tests	v 4.1.0-1	Failures: 19	Failures: 19	Failures: 19
Printing Tests	v 4.1.0-1	Success	Success	Failures: 3
Python Runtime Tests	v 4.1.1-2	Success	Success	Success
Qt3-Azov Tests	v 4.1.0-1	Success	Failures: 2	Failures: 199
Qt4-Azov Tests	v 4.1.0-1	Failures: 5	Failures: 6	Failures: 2
Xml2-Azov Tests	v 4.1.0-1	Success	Success	Success
Xts5 Tests	v 5.1.5-38	Failures: 1	Success	Warnings: 8
Apache Tests	v 2.2.14-3	Success	Success	Success
Expect Tests	v 5.43.0-11	Success	Success	Success
Groff Tests	v 1.20.1-5	Success	Success	Success
Raptor Tests	v 1.4.19-3	Success	Success	Success
Rsync Tests	v 3.0.6-3	Success	Success	Success
Samba Tests	v 3.4.3-5	Success	Success	Success
Tcl Tests	v 8.5.7-6	Failures:1	Success	Failures: 25
Xpdf Tests	v 1.01-10	Success	Success	Failures: 1

Analysis of result

Below table shows the failed test cases.

• Total failures: The number of total failures after running test. It includes known problems and real failures which should be fixed.
• Known problem: Known problem means it is a bug of LSB itself and we can find trace these bugs on http://bugs.linuxbase.org. On link page we give a summary of known problems.
• Failures(need to fix) : Real failures we should fix them
• Test suite error : Caused by two reasons: one is for absent library needed by test suite. one is that test suite could lose some function which should be included in it.

x86-32 (emenlow,update(2011-01-08))

• New problem: It is a bug without arising on previous version. It could be caused by some patches updated every day.

Test	Total failures	Known problem	Failures(need to fix)	Note
Core Tests	14	13	1
Libstdc++ Tests	132	?	132
Desktop Tests	28	21	7
Desktop-T2C Tests	6	1	5
OLVER Core Tests	9	9
Perl Tests	19		19
Qt4-Azov Tests	5	5
Tcl Tests	1		1

x86-64 (sugarbay,update(2012-01-08))

• New problem: It is a bug without arising on previous version. It could be caused by some patches updated every day.

Test	Total failures	Known problem	Failures(need to fix)	Note
Core Tests	15	14	1
Libstdc++ Tests	121	?	121
Desktop Tests	27	20	7
Desktop-T2C Tests	24	19	5
OLVER Core Tests	10	9	1
Perl Tests	19		19
Qt4-Azov Tests	5	5
Qt3 Tests	2		2

ppc(mpc8315e,update partly(2012-01-14))

We have the following failures(including "unresolved" and "failed") to fix.

Test	Total failures	Known problem	Failures(need to fix)	Note
Core Tests	70	66	4
Desktop Tests	27	20	7
Desktop-T2C Tests	20	15	5
Libstdc++ Tests	152		152
OLVER Core Tests				Blocked
Perl Tests	19			19
Printing Tests	3	1	1(unresolved), 1(failure), 1(unreported)
Qt3-Azov	199			No enough memory
Qt4-Azov	2		2

• For new problem, it has others besides the problems emenlow has
  • Qt4-Azov Tests
    • "Child process timed out" caused many unresolved items
  • No locale and charmap in the latest lsb image, there were many failures because of the absent of locale resources.
• Illegal instruction(Known problem http://bugs.linuxbase.org/show_bug.cgi?id=3253)
  • 65 SIGILL error
    • Core Tests
      • /tset/ANSI.os/math/... : SIGILL cause 47 unresolved failures.
      • /tset/LSB.os/libm/fpeInvalid/T.fpeInvalid (1st--2nd)
    • Desktop-t2c Tests
      • /glib-t2c/tests/glib_random/glib_random (9th--23rd)
    • Xpdf Tests
      • pdftotext
  • It was caused by the difference of compilation tool between LSB Test Suite and mpc8315e. In fact, instruction "fsqrt" doesn't exist on e300core serial and CPU of mpc8315e belongs to e300core type. So this problem can't be solved for mpc8315e
• OLVER Core Tests:
  • OLVER Core Tests can't be finished on mpc8315e-rdb
  • We think the reason which make test blocked is the low memory(128M on mpc8315e). Because we re-run the test case which is blocked during OLVER core tests and it can pass successfully. That is to say, if we run the single test case, it will be passed. But we run all OLVER core tests and the whole tests will be blocked. We have ever run OLVER Core Tests for nearly 50 hours and it can't finish.
  • lsb-test-olver-core may require about 512Mb of system memory (with an X server and a contemporary desktop running), as well as 2Gb of free disk space to complete a test run. The reference linkage: http://ldn.linuxfoundation.org/lsb/preparing-lsb-distribution-testing
• Qt3-Azov Tests, Printing Tests
  • Out of memory error cause all of the failures, the information like the following arise on screen when running Qt3 and Printing test suite

   Killed process 22453 (Xvfb) total-vm:84064kB, anon-rss:73720kB, file-rss:8kB
   run-parts invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0,
   oom_score_adj=0
   Call Trace:
   [c0cadc90] [c0009f3c] show_stack+0x108/0x1c4 (unreliable)
   [c0cadce0] [c0431098] dump_stack+0x2c/0x44
   [c0cadcf0] [c00a1880] dump_header.clone.14+0x70/0x184
   [c0cadd30] [c00a1b80] oom_kill_process.clone.16+0x94/0x294
   [c0cadd60] [c00a222c] out_of_memory+0x270/0x33c
   [c0caddb0] [c00a6628] __alloc_pages_nodemask+0x624/0x65c
   [c0cade50] [c00a6694] __get_free_pages+0x34/0x6c
   [c0cade60] [c00269f8] copy_process+0xe4/0xd84   
   [c0cadec0] [c0027764] do_fork+0x80/0x39c
   [c0cadf20] [c0009cb4] sys_vfork+0x68/0x80
   [c0cadf40] [c0011d0c] ret_from_syscall+0x0/0x38       
     --- Exception: c00 at 0xfecc358
     LR = 0x10084518

• • I think this should be limited memory problem
• Libstdc++ Tests
  • /26_numerics/complex/13450.cc
    • This test case check whether data type "long double" can't be supported or not on ppc. Mp8315e deviates from what the test expects. It is Known problem.
  • /27_io/basic_stringbuf/overflow/wchar_

Failures tracker(update 2012-01-14)

We should fix the following problems. We also filed bugs for them and try to fix them.

Tests	Sub Tests	Test Number	Arch	Bug ID of yocto	Proposal suggestion
Qt4-Azov Tests	/Qt4_Internationalization-t2c/tests/Qt4_Internationalization/Qt4_Internationalization 2		ppc		Child process timed out
Qt4-Azov Tests	/Qt4_Widgets-t2c/tests/QDesktopServices/QDesktopServices 3		x86-64
Desktop	Desktop-gtkvts	/tests/functions/Keyboard_Accelerator_Groups/Keyboard_Accelerator_Groups 27-29	x86,x86-64,ppc		Function "gtk_accelerator_name" from gtk/gtkaccelgroup.c has been changed since version 2.24.8. So this is incorrect test case now.(Upstream, Don't need fix)
Desktop	Desktop-gtkvts	/tests/functions/GtkTreeSortable/GtkTreeSortable 15-18	x86,x86-64,ppc		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3238 (Upstream, Don't need fix)
Desktop-T2c		/atk-t2c/tests/AtkObject_relation/AtkObject_relation 2,5,6	x86,x86-64,ppc		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3189(Upstream, Don't need fix)
Desktop-T2c		/gobject-t2c/tests/gobject_value_arrays/gobject_value_arrays 24	x86,x86-64,ppc
Desktop-T2c		/gobject-t2c/tests/gobject_gtype_instance/gobject_gtype_instance 39	x86.x86-64,ppc		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3184(Upstream, Don't need fix)
Libstdc++			x86(132),x86-64(121),ppc(152)		missing locale resource
Perl test		all/tst_perlModPresent.pl 1	x86, x86-64,ppc		LSB 4.1 hasn't capability to test perl-5.14.2 because there are too many changes for new version perl than the old. So this is upstream problem. We can resolve this problem until the newer version for lsb test suite.(Upstream, Don't need fix)
Perl test		../lib/AutoSplit.t 402	x86, x86-64,ppc		as same as above
Perl test		../lib/autouse.t 403	x86, x86-64,ppc		as same as above
Perl test		../lib/Class/ISA/t/00_about_verbose.t 440	x86, x86-64,ppc		as same as above
Perl test		../lib/Class/ISA/t/01_old_junk.t 441	x86, x86-64,ppc		as same as above
Perl test		../lib/Dumpvalue.t 458	x86, x86-64,ppc		as same as above
Perl test		../lib/English.t 460	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/basic.t 466	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/FIRST_MAKEFILE.t 469	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/Liblist.t 473	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/PL_FILES.t 480	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/prereq_print.t 483	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/recurs.t 486	x86, x86-64,ppc		as same as above
Perl test		../lib/ExtUtils/t/writemakefile_args.t 490	x86,x86-64,ppc		as same as above
Perl test		../lib/File/CheckTree.t 493	x86, x86-64,ppc		as same as above
Perl test		../lib/overload.t 644	x86, x86-64,ppc		as same as above
Perl test		../lib/Pod/t/eol.t 651	x86, x86-64,ppc		as same as above
Perl test		../lib/Pod/t/Usage.t 659	x86, x86-64		as same as above
Perl test		../lib/strict.t 668	x86, x86-64,ppc		as same as above
Perl test		../lib/Test/Simple/t/overload.t 708	x86,ppc		as same as above
Perl test		../lib/Unicode/UCD.t 809	x86, x86-64,ppc		as same as above
Xts5 Tests		/tset/Xopen/strtkysym/Test 7	x86		host contamination，Add patch "makekeys_crosscompile.patch" to libx11-1.4.4
Tcl test		exec-9.7	x86
Tcl test		http-3.3(24)	ppc
Tcl test		socket-2.10	ppc
Core test		/tset/POSIX.os/procprim/fork/T.fork 5	x86-64		http://bugs.linuxbase.org/show_bug.cgi?id=2270(fixed after 4.1)
Core test		/tset/LSB.fhs/var/lib-hwclock/lib-hwclock-tc 1	x86,x86-64,ppc		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3116(Upstream, Don't need fix)
Core test		/tset/LSB.os/network/gethostbyaddr/T.gethostbyaddr 2	x86,x86-64,ppc		Absent startup program avahi-daemon.
Qt3-Azov Tests		/Qt3_Object_Model-t2c/tests/QMetaObject/QMetaObject 3	x86-64		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3231(Upstream, Don't need fix)
Qt3-Azov Tests		/Qt3_Object_Model-t2c/tests/QMetaObject/QMetaObject 15	x86-64		https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3231(Upstream, Don't need fix)

Known problems tracker

See details on Known problems tracker