BSP Test Plan
A Board Support Package (BSP) is the collection of software and data that allows an operating system to run in a specific hardware. It contains the drivers, routines and information that defines how to support a particular hardware device, set of devices, or hardware platform.
Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.
|Ee Peng Yeoh||Intel|
|Yi Zhao||Wind River|
Types of Tests
- Manual tests on different platforms
- Automated test on different platforms
Platforms, type of images and type of tests are showed detailed on BSP Test Matrix.
- Manual Tests
- Boot image
- Install Image
- Graphical interface components
- System reboot and shutdown recovery and standby
- X server tests
- rpm packages
- File system (reading/writing)
- Mont / Umont
complete test steps of these manual tests are on poky repo under meta/lib/oeqa/manual/ <>
- Automated Tests
full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>
As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.
Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image.
Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.
1. Prepare the image and device under test 1.1 Download Artifacts there are 2 links available from Public AutoBuilder to download images http://autobuilder.yoctoproject.org/pub/releases/ https://autobuilder.yocto.io/pub/releases/ Artifacts to be downloaded are: Real HW under folder machines/<MACHINE_UNDER_TEST> - <IMAGE_TO_BE_TESTED>.hddimg - <IMAGE_TO_BE_TESTED>.wic - <IMAGE_TO_BE_TESTED>.wic.bmap - <IMAGE_TO_BE_TESTED>.manifest - <IMAGE_TO_BE_TESTED>.testdata.json example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.* QEMU under folder machines/qemu/<MACHINE_UNDER_TEST> - <BZIMAGE>.bin - <IMAGE_TO_BE_TESTED>.ext4 - <IMAGE_TO_BE_TESTED>.qemuboot.conf - <IMAGE_TO_BE_TESTED>.manifest - <IMAGE_TO_BE_TESTED>.testdata.json example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.* 1.2 For real hardware devices: - Write the image to a bootable media (USB key, SD Card, etc.) using the dd command: $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=TARGET_DEVICE example with usb key connected: $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=/dev/sdb - Attach the bootable device to the DUT - Turn the DUT on, and use its boot options to select the correct media for booting - After the DUT boots to image in USB stick, the default username is "root". (This is because the yocto image by default uses "debug-tweaks" feature) - Ensure the device has a network connection and a valid IP address
2. For Manual Test Cases: 2.1 - Go to the proper Test Run assigned on the execution and start executing test cases
3. For automated Test Cases: 3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky 3.2 $cd poky 3.3 Checkout to the commit specified on the release information $git checkout <commit-id> 3.4 If testing meta-intel, clone the layer repo inside poky $git clone git://git.yoctoproject.org/meta-intel 3.4.1 $cd meta-intel 3.4.2 Checkout to the meta-intel commit specified on the release information $git checkout <commit-id> 3.4.3 Return to the poky root path $cd .. 3.5 $source oe-init-build-env 3.6 If testing meta-intel, add the layer to conf/bblayers.conf. It may be done with: $bitbake-layers add-layer "../meta-intel" 3.7 For real HW edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding: Install / Boot the image on the DUT and check the IP INHERIT += "testimage" TEST_TARGET = "simpleremote" TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests TEST_TARGET_IP = "DUT ip" --> IP of the device under test if image is LSB DISTRO ?= "poky-lsb" 3.8 For QEMU edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding: INHERIT += "testimage" if image is LSB DISTRO ?= "poky-lsb" 3.9 Make sure conf only set DISTRO once, if DISTRO was set more than once, during testimage you could face error: > ERROR: Nothing PROVIDES 'core-image-lsb-sdk' > core-image-lsb-sdk was skipped: missing required distro feature 'pam' > (not in DISTRO_FEATURES)
For releases before 2.3 also add TEST_SUITES = " auto" Remember to add the machine to local.conf, for example: If genericx86 is needed, MACHINE ?= "genericx86" If qemux86 is needed, MACHINE ?= "qemux86" If intel-corei7-64 is needed, MACHINE = "intel-corei7-64"
4. Prepare the environment to execute test suite 4.1 $bitbake rpm busybox curl run-postinsts 4.2 $bitbake package-index 4.3 Ensure that a folder is created, otherwise create it, under tmp/deploy/images/<MACHINE>, where <MACHINE> corresponds to the variable defined on local.conf on previous steps 4.4 Paste artifacts downloaded on previously created folder
5. Execute the test suite 5.1 $bitbake <IMAGE_TESTED> -c testimage example bitbake core-image-sato-sdk -c testimage -v
6. Collect the test result from <build>/tmp/log/oeqa/testresults.json Note: Collect test results after one test run completed.
7. Verify that all test cases PASSED if not raise bugs properly
[Deprecated] Update results to corresponding Testopia Test Run There is a script available to upload results form the log automatically on QA-Tools and follow this example Update TR by ID
For correctly report the bugs follow the already defined steps here
There are 2 special test that also applied as BSP Components those are:
For detailed information about behavior and characteristics check Ptest
This required 2 machine: a) host machine - to start/control poky/oeqa/runtime tests b) device under test (DUT) - to boot up the target ptest image and to run the ptest trigger by host machine
1. Download pTest image from autobuilder (core-image-sato-sdk-ptest-genericx86-64.hddimg)
2. On DUT, install or write the image on a media connected to the DUT Configure the network so it be able to work externally: - Export the proxy using "export http_proxy=<add your proxy link>" command eg $ export http_proxy=http://<my_proxy>:<port> - And check if you have internet connection, typing on terminal: $ wget www.google.com
3. On host machine Configure for running oeqa/runtime test (refer to above BSP runtime test instruction) Add to local.conf, INHERIT += "testimage" TEST_SUITES = "ping ssh ptest" Then, start the runtime test and collect the test result from <build>/tmp/log/oeqa/testresults.json
1. Generate the report of pTests as follows:
1.1 Collect the testresults.json files for both current release (base) and previous release (target). 1.2 Use "resulttool report" to compute the ptest summary result using current release (base). $ resulttool report <file_path_to_current_ptest_testresultsjson> 1.3 Use "resulttool regression" to compute the ptest regression as below $ resulttool regression <file_path_to_current_ptest_testresultsjson> <file_path_to_previous_ptest_testresultsjson> 1.4 Copy and paste the report and regression report to wiki page https://wiki.yoctoproject.org/wiki/Ptest/archive
In this section, 3 Test suites are executed on NUC HW with a core-image-lsb-sdk-genericx86-64 image.
- LSB tests
- POSIX tests
- LTP tests
1. Download the core-image-lsb-sdk-genericx86-64.hddimg image from Autobuilder (same image as in LSB automated testing for genericx86-64-lsb bsp)
2. Install the image on the DUT * We test compliance on a NUC
3. Configure the network so it be able to work externally: * edit /etc/resolv.conf and add the gateway ip_address * add the ip and netmask using "ifconfig" command * add the route using "route add default gw <ip_address>" * export the proxy using "export http_proxy=<add your proxy link>" command : there is a bug and if you make these steps in another order than above, it may be possible not work
4. Copy "compliance_test.py" script on DUT script can be found here
5. Make sure that your network connection is working
6. Run the script like this: * make the script executable: "chmod a+x compliance_local.py" * run in command line the following command "./compliance_test.py <milestone> <date>" 7. Get the logs from DUT: - result-<milestone>-data.fulllog - result-<milestone>-data.log - result-<milestone>-data.fail - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) - the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to email@example.com specifying the version and the type of image - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily) -- it has like 18 Mb -- upload this file on drive and send the link to firstname.lastname@example.org -- also I'll fwd an email to see how it looks
8. Put the tests from Testopia - Runtime test run on passed
<To Be Updated>
Add New Tests
<To be Updated>
Yocto QA Cycle
|Release||CPU Class||HW Platform||BSP Name||linux-yocto||Image-type||AUTO||MANUAL||Compliance||pTest|
|generic BSPs (x86)||Big Core||MinwMax 32bit||genericx86||4.x||core-image-sato-sdk||Y||Y|
|generic BSPs (n-x86)||MIPS||EdgeRouter||EdgeRouter||4.x||core-image-sato-sdk||Y||Y|
|Release||CPU Class||HW Platform||BSP Name||Image-type||Auto||Manual|
|meta-intel||Big Core||MinnowMax 64bit||corei7-64||core-image-sato-sdk||Y|