BSP Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
mNo edit summary
Line 101: Line 101:
= Test Strategy  =
= Test Strategy  =


As mentioned bellow there are 2 types of test Automated an Manual tests, below a description of how those test are applied.
As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.





Revision as of 17:25, 25 October 2017

About BSP

A Board Support Package (BSP) is a collection of information that defines how to support a particular hardware device, set of devices, or hardware platform.

Objectives

Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.


Team members

Member Team
José Pérez Carranza Intel
Oscar Lopez Arandas Intel
Yi Zaho Wind River


Scope

Types of Tests


  • Manual tests on different platforms
  • Automated test on different platforms


Platforms, type of images and type of tests are showed detailed on BSP Test Matrix.


Features tested

  • Manual Tests
    • Boot image
    • Install Image
    • Multimedia
    • Graphical interface components
    • System reboot and shutdown recovery and standby
    • X server tests
    • rpm packages
    • File system (reading/writing)
    • Mont / Umont
    • Network

All test cases can be found here <>

  • Automated Tests
    • buildcpio
    • buildgalculator
    • buildlzip
    • connman
    • date
    • df
    • dnf
    • gcc
    • kernelmodule
    • ldd
    • logrotate
    • multilib
    • oe_syslog
    • pam
    • parselogs
    • perl
    • ping
    • ptest
    • python
    • rpm
    • scanelf
    • scp
    • skeletoninit
    • ssh
    • systemd
    • x32lib
    • xorg

full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>


Test Strategy

As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.


Manual

Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image.


Automated Tests

Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.


Test Process

1. Download Artifacts 
   there are 2 links available from Public AutoBuilder to download images 
      http://autobuilder.yoctoproject.org/pub/releases/
      https://autobuilder.yocto.io/pub/releases/
    
   Artifacts to be downloaded are: 
   
   Real HW
   under folder machines/<MACHINE_UNDER_TEST>
   - <IMAGE_TO_BE_TESTED>.hhddimg
   - <IMAGE_TO_BE_TESTED>.wic
   - <IMAGE_TO_BE_TESTED>.wic.bmap
   - <IMAGE_TO_BE_TESTED>.manifest
   - <IMAGE_TO_BE_TESTED>.testdata.json
   example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*
   
   QEMU
   under folder machines/qemu/<MACHINE_UNDER_TEST>
   - <BZIMAGE>.bin
   - <IMAGE_TO_BE_TESTED>.ext4
   - <IMAGE_TO_BE_TESTED>.qemuboot.conf
   - <IMAGE_TO_BE_TESTED>.manifest
   - <IMAGE_TO_BE_TESTED>.testdata.json
   example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*
2. For Manual Test Cases:
  
  2.1 - Go to the proper Test Run assigned on the execution and start executing test cases  
3. For automated Test Cases
   
   3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky"
   3.2 $cd poky 
   3.3 Checkout to the commit specified on the release information $git checkout <commit-id>
   3.4 $source oe-init-build-env
   3.5 For real HW edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: 
       Install / Boot the image on the DUT and check the IP
       INHERIT += "testimage"
       TEST_TARGET = "simpleremote"
       TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
       TEST_TARGET_IP = "DUT ip"  --> IP of the device under test
       if image is LSB 
       DISTRO ?= "poky-lsb"
   3.6 For real QEMU edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:
       INHERIT += "testimage"
       if image is LSB 
       DISTRO ?= "poky-lsb"
   For releases before 2.3 also add
   TEST_SUITES = " auto" 
   
   Remember to add the machine to local.conf for example:
   if genericx86 is needed, MACHINE ?= "genericx86
   if qemux86 is needed, MACHINE ?= "qemux86
4. Prepare the environment to execute test suite 
   
   4.1 $bitbake rpm busybox run-postinsts
   4.2 $bitbake package-index
   4.3 Ensure that a folder is created, otherwise create it, under
       tmp/deploy/images/<MACHINE>, where <MACHINE> corresponds to the variable defined on
       local.conf on previous steps
   4.4 Paste artifacts downloaded on previously created folder 
5. Execute the test suite
   
   5.1 $bitbake <IMAGE_TESTED> -c testimage
      example bitbake core-image-sato-sdk -c testimage -v
6. Verify that all test cases PASSED if not raise bugs properly 
7. Update results to corresponding Testopia Test Run 
   There is a script available to upload results form the log automatically on QA-Tools and follow this example Update TR by ID

Bug Reporting

For correctly report the bugs follow the already defined steps here

Special Tests

There are 2 special test that also applied as BSP Components those are:

pTest

For detailed information about behavior and characteristics check Ptest

Test Process

1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)
   * Test are applied on NUC with genericx86-64 
2. Install the image on DUT (using legacy boot)
   Configure the network so it be able to work externally:
   - Export the proxy using "export http_proxy=<add your proxy link>" command eg
     $ export http_proxy=http://<my_proxy>:<port>
   - And check if you have internet connection, typing on terminal:
     $ wget www.google.com
3. In command line type
   $ ptest-runner > ptest.log
     and wait for it to finish ( about 5 hours)
     Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.
     and > ptest.log, in order to save the test results in that file.

Reporting Process

1. Generate the report of pTests as follows:

    1.1 Get the log from the DUT.
    1.2 Go to the Ptest/archive page and check the previous execution to this release.
    1.3 Go the the report of the execution and download the log.
    1.4 Use the ptest-parser script to generate the report, this script generates a file called <release>wiki.page.
    1.5 Create a wiki with the name Ptest <commit>_<release-tested>
    1.6 Copy the content of the content of <release>wiki.page and paste it on new wiki created.
    1.7 Upload the log of current execution and update the recent created wiki with the link to the log.
    1.8 Check the packages that had regression on pass rate Down rate.png and if there is a bug reported update the bug with comments that still present 
        on current commit, if there is no bug created raise a new one.
    1.9 Update the Ptest/archive wiki with the info of this new execution.

Compliance

In this section 3 Test suites are executed on a NUC HW with core-image-sato-sdk-generixc86-64 image booted.

  • LSB tests
  • POSIX tests
  • LTP tests

Test Process

1. Download lsb image from autobuilder( same image as in LSB automated testing for genericx86-64-lsb bsp)
   * we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk
2. Install the image on DUT
3. Configure the network so it be able to work externally:
   * edit /etc/resolv.conf and add the gateway ip_address
   * add the ip and netmask using "ifconfig" command
   * add the route using "route add default gw <ip_address>" 
   * export the proxy using "export http_proxy=<add your proxy link>" command
     : there is a bug and if you make these steps in another order than above, it may be possible not work
 4. Copy "compliance_test.py" script on DUT
 5. Make sure that your network connection is working
 6. Run the script like this:
   * make the script executable: "chmod a+x compliance_local.py"
   * run in command line the following command "./compliance_test.py <milestone> <date>"
   * wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)
 7. Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)
 8. Get the logs from DUT:
   - result-<milestone>-data.fulllog
   - result-<milestone>-data.log
   - result-<milestone>-data.fail
   - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)
     -  the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to yi.zhao@windriver.com specifying the v 
        version and the type of image
   - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)
     -- it has like 18 Mb
     -- upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com
     -- also I'll fwd an email to see how it looks
 9. Put the tests from Testopia - Runtime test run on passed

Reporting Process

<To Be Updated>

Add New Tests

<To be Updated>

Test Approach

Test Matrix

Yocto QA Cycle

Release CPU Class HW Platform BSP Name linux-yocto Image-type AUTO MANUAL Compliance pTest
generic BSPs (x86) Big Core MinwMax 32bit genericx86 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk Y
genericx86-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
MinwMax 64bit genericx86-64 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
genericx86-64-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
NUC genericx86-64 4.x core-image-sato-sdk Y Y Y
4,x-ltsi core-image-lsb-sdk Y Y
genericx86-64-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
VM QEMU qemux86 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
qemux86-64 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
generic BSPs (n-x86) MIPS EdgeRouter EdgeRouter 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
PPC MPC8315e-rdb MPC8315e-rdb 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x-ltsi core-image-sato-sdk WIC Y
ARM Beaglebone Black beaglebone 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x-ltsi core-image-sato-sdk WIC Y
VM QEMU qemuarm 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemuarm64 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
qemuppc 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemumips 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemumips64 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk


Meta-Intel

Release CPU Class HW Platform BSP Name Image-type Auto Manual
meta-intel Big Core MinnowMax 64bit corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
NUC corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
CherryHill corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
Joule corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk
core-image-sato-sdk WIC Y
MinnowMax 32bit core2-32 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
meta-intel-quark 32bit intel-quark core-image-sato-sdk Y
core-image-lsb-sdk
core-image-sato-sdk WIC Y

Not Tested

Requirements

Tools