BSP Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
 
(51 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= About BSP  =
= About BSP  =


A Board Support Package (BSP) is a collection of information that defines how to support a particular hardware device, set of devices, or hardware platform.
A Board Support Package (BSP) is the collection of software and data that allows an operating system to run in a specific hardware. It contains the drivers, routines and information that defines how to support a particular hardware device, set of devices, or hardware platform.


= Objectives =
= Objectives =
Line 19: Line 19:
{| style="border-spacing:0;margin:auto;width:6.925in;"
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | José Pérez Carranza
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Ee Peng Yeoh
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel  
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel  
|-
|-
Line 25: Line 25:
{| style="border-spacing:0;margin:auto;width:6.925in;"
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Oscar Lopez Arandas
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Sangeeta Jain
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel  
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel  
|-
|-
Line 31: Line 31:
{| style="border-spacing:0;margin:auto;width:6.925in;"
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Yi Zaho
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Yi Zhao
| style="border:0.05pt solid #000000;padding:0.0382in;" | Wind River  
| style="border:0.05pt solid #000000;padding:0.0382in;" | Wind River  
|-
|-
|}
|}


= Scope =
= Scope =
Line 65: Line 64:
** Network
** Network


All test cases can be found here <>
complete test steps of these manual tests are on poky repo under meta/lib/oeqa/manual/ <>


* Automated Tests  
* Automated Tests  
Line 97: Line 96:


full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>
full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>


= Test Strategy  =
= Test Strategy  =


As mentioned bellow there are 2 types of test Automated an Manual tests, below a description of how those test are applied.
As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.




Line 117: Line 115:
==Test Process==
==Test Process==


  1. Download Artifacts  
  1. Prepare the image and device under test
    there are 2 links available from Public AutoBuilder to download images  
      http://autobuilder.yoctoproject.org/pub/releases/
    1.1 Download Artifacts
      https://autobuilder.yocto.io/pub/releases/
      there are 2 links available from Public AutoBuilder to download images  
          http://autobuilder.yoctoproject.org/pub/releases/
          https://autobuilder.yocto.io/pub/releases/
      
      
    Artifacts to be downloaded are:  
      Artifacts to be downloaded are:  
      
      
    '''Real HW'''
      '''Real HW'''
    under folder machines/<MACHINE_UNDER_TEST>
      under folder machines/<MACHINE_UNDER_TEST>
    - <IMAGE_TO_BE_TESTED>.hhddimg
      - <IMAGE_TO_BE_TESTED>.hddimg
    - <IMAGE_TO_BE_TESTED>.wic
      - <IMAGE_TO_BE_TESTED>.wic
    - <IMAGE_TO_BE_TESTED>.wic.bmap
      - <IMAGE_TO_BE_TESTED>.wic.bmap
    - <IMAGE_TO_BE_TESTED>.manifest
      - <IMAGE_TO_BE_TESTED>.manifest
    - <IMAGE_TO_BE_TESTED>.testdata.json
      - <IMAGE_TO_BE_TESTED>.testdata.json
    '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*'''
      '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*'''
     
      '''QEMU'''
      under folder machines/qemu/<MACHINE_UNDER_TEST>
      - <BZIMAGE>.bin
      - <IMAGE_TO_BE_TESTED>.ext4
      - <IMAGE_TO_BE_TESTED>.qemuboot.conf
      - <IMAGE_TO_BE_TESTED>.manifest
      - <IMAGE_TO_BE_TESTED>.testdata.json
      '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*'''
      
      
     '''QEMU'''
     1.2 For real hardware devices:
    under folder machines/qemu/<MACHINE_UNDER_TEST>
      - Write the image to a bootable media (USB key, SD Card, etc.) using the dd command:
    - <BZIMAGE>.bin
          '''$ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=TARGET_DEVICE'''
    - <IMAGE_TO_BE_TESTED>.ext4
          '''example with usb key connected: $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=/dev/sdb'''
    - <IMAGE_TO_BE_TESTED>.qemuboot.conf
      - Attach the bootable device to the DUT
    - <IMAGE_TO_BE_TESTED>.manifest
      - Turn the DUT on, and use its boot options to select the correct media for booting
    - <IMAGE_TO_BE_TESTED>.testdata.json
      - After the DUT boots to image in USB stick, the default username is "root". (This is because the yocto image by default uses "debug-tweaks" feature)
    '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*'''
      - Ensure the device has a network connection and a valid IP address


  2. For Manual Test Cases:
  2. For Manual Test Cases:
    
    
  2.1 - Go to the proper Test Run assigned on the execution and start executing test cases   
    2.1 - Go to the proper Test Run assigned on the execution and start executing test cases   


  3. For automated Test Cases
  3. For automated Test Cases:
      
      
     3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky"
     3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky
     3.2 Checkout to the commit specified on the release information $git checkout <commit-id>
     3.2 $cd poky
     3.3 $cd poky  
    3.3 Checkout to the commit specified on the release information $git checkout <commit-id>
     3.4 $source oe-init-build-env
     3.4 '''If testing meta-intel''', clone the layer repo inside poky $git clone git://git.yoctoproject.org/meta-intel
     3.5 For real HW edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:  
        3.4.1 $cd meta-intel
        3.4.2 Checkout to the meta-intel commit specified on the release information $git checkout <commit-id>
        3.4.3 Return to the poky root path $cd ..
     3.5 $source oe-init-build-env
     3.6 '''If testing meta-intel''', add the layer to conf/bblayers.conf. It may be done with:
        $bitbake-layers add-layer "../meta-intel"
    3.7 For real HW edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding:  
        '''Install / Boot the image on the DUT and check the IP'''
         INHERIT += "testimage"
         INHERIT += "testimage"
         TEST_TARGET = "simpleremote"
         TEST_TARGET = "simpleremote"
         TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
         TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
         TEST_TARGET_IP = "DUT ip"  --> IP of the device under tes
         TEST_TARGET_IP = "DUT ip"  --> IP of the device under test
         if image is LSB  
         if image is LSB  
         DISTRO ?= "poky-lsb"
         DISTRO ?= "poky-lsb"
     3.6 For real QEMU edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:
     3.8 For QEMU edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding:
         INHERIT += "testimage"
         INHERIT += "testimage"
         if image is LSB  
         if image is LSB  
         DISTRO ?= "poky-lsb"
         DISTRO ?= "poky-lsb"
    3.9 Make sure conf only set DISTRO once, if DISTRO was set more than once, during testimage you could face error:
    > ERROR: Nothing PROVIDES 'core-image-lsb-sdk'
    > core-image-lsb-sdk was skipped: missing required distro feature 'pam'
    > (not in DISTRO_FEATURES)
     '''For releases before 2.3 also add'''
     '''For releases before 2.3 also add'''
     TEST_SUITES = " auto"  
     TEST_SUITES = " auto"  
      
      
     '''Remember to add the machine to local.conf for example:'''
     '''Remember to add the machine to local.conf, for example:'''
     if genericx86 is needed, MACHINE ?= "genericx86
     If genericx86 is needed, MACHINE ?= "genericx86"
     if qemux86 is needed, MACHINE ?= "qemux86
     If qemux86 is needed, MACHINE ?= "qemux86"
    If intel-corei7-64 is needed, MACHINE = "intel-corei7-64"


  4. Prepare the environment to execute test suite  
  4. Prepare the environment to execute test suite  
      
      
     4.1 $bitbake rpm busybox run-postinsts
     4.1 $bitbake rpm busybox curl run-postinsts
     4.2 $bitbake package-index
     4.2 $bitbake package-index
     4.3 Ensure that a folder is create under tmp/deploy/image/<MACHINE>
     4.3 Ensure that a folder is created, otherwise create it, under
        where <MACHINE> corresponds to variable defined on local.conf on previous steps
        tmp/deploy/images/<MACHINE>, where <MACHINE> corresponds to the variable defined on
        local.conf on previous steps
     4.4 Paste artifacts downloaded on previously created folder  
     4.4 Paste artifacts downloaded on previously created folder  


Line 181: Line 204:
      
      
     5.1 $bitbake <IMAGE_TESTED> -c testimage
     5.1 $bitbake <IMAGE_TESTED> -c testimage
       '''example bitbake core-image-sato-sdk -c testimage'''
       '''example bitbake core-image-sato-sdk -c testimage -v'''


  6. Verify that all test cases PASSED if not raise bugs properly
  6. Collect the test result from <build>/tmp/log/oeqa/testresults.json
    Note: Collect test results after one test run completed.


  7. Update results to corresponding Testopia Test Run
  7. Verify that all test cases PASSED if not raise bugs properly
    '''There is a script available to upload results form the log automatically on [here]'''


    [Deprecated]
    Update results to corresponding Testopia Test Run
    '''There is a script available to upload results form the log automatically on [[Qa-tools#Testopia_Update|QA-Tools]] and follow this example [[Qa-tools#Update_a_Test_Run_by_ID|Update TR by ID]]'''


==Bug Reporting==
==Bug Reporting==


For correctly report the bugs follow the already defined steps here
For correctly report the bugs follow the already defined steps [[Bug_reporting_and_Information_levels | here]]


= Special Tests =
= Special Tests =
Line 203: Line 229:
=== Test Process ===
=== Test Process ===


This required 2 machine:
a) host machine - to start/control poky/oeqa/runtime tests
b) device under test (DUT) - to boot up the target ptest image and to run the ptest trigger by host machine


  1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)
  1. Download pTest image from autobuilder (core-image-sato-sdk-ptest-genericx86-64.hddimg)
    * Test are applied on NUC with genericx86-64  


  2. Install the image on DUT (using legacy boot)
  2. On DUT, install or write the image on a media connected to the DUT  
     Configure the network so it be able to work externally:</p>
     Configure the network so it be able to work externally:
     - Export the proxy using "export http_proxy=<add your proxy link>" command eg</p>
     - Export the proxy using "export http_proxy=<add your proxy link>" command eg
       $ export http_proxy=http://proxy-chain.intel.com:911</pre>
       $ export http_proxy=http://<my_proxy>:<port>
     - And check if you have internet connection, typing on terminal:
     - And check if you have internet connection, typing on terminal:
       $ wget www.google.com</pre>
       $ wget www.google.com


  3. In command line type</p>
  3. On host machine
     $ ptest-runner > ptest.log
    Configure for running oeqa/runtime test (refer to above BSP runtime test instruction)
      and wait for it to finish ( about 5 hours)
     Add to local.conf,
      '''Note:''' The "script" is already placed on the distro, you just have to type '''ptest'''...TAB and the terminal should autocomplete the command.
    INHERIT += "testimage"
      and > '''ptest.log''', in order to save the test results in that file.
    TEST_SUITES = "ping ssh ptest"
    Then, start the runtime test and collect the test result from <build>/tmp/log/oeqa/testresults.json


=== Reporting Process ===
=== Reporting Process ===


<update>
1. Generate the report of pTests as follows:
    1.1 Collect the testresults.json files for both current release (base) and previous release (target).
    1.2 Use "resulttool report" to compute the ptest summary result using current release (base).
        $ resulttool report <file_path_to_current_ptest_testresultsjson>
    1.3 Use "resulttool regression" to compute the ptest regression as below
        $ resulttool regression <file_path_to_current_ptest_testresultsjson> <file_path_to_previous_ptest_testresultsjson>
    1.4 Copy and paste the report and regression report to wiki page https://wiki.yoctoproject.org/wiki/Ptest/archive


== Compliance ==
== Compliance ==


In this section 3 Test suites are executed on a NUC HW with core-image-sato-sdk-generixc86-64 image booted.
In this section, 3 Test suites are executed on NUC HW with a core-image-lsb-sdk-genericx86-64 image.


*LSB tests
*LSB tests
Line 233: Line 268:


=== Test Process  ===
=== Test Process  ===
  1. Download lsb image from autobuilder( same image as in LSB automated testing for genericx86-64-lsb bsp)
  1. Download the core-image-lsb-sdk-genericx86-64.hddimg image from Autobuilder (same image as in LSB automated testing for genericx86-64-lsb bsp)
    * we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk


  2. Install the image on DUT
  2. Install the image on the DUT
    * We test compliance on a NUC


  3. Configure the network so it be able to work externally:
  3. Configure the network so it be able to work externally:
Line 246: Line 281:


   4. Copy "compliance_test.py" script on DUT
   4. Copy "compliance_test.py" script on DUT
    script can be found [http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/tree/compliance/compliance_test.py here]


   5. Make sure that your network connection is working
   5. Make sure that your network connection is working
Line 252: Line 288:
     * make the script executable: "chmod a+x compliance_local.py"
     * make the script executable: "chmod a+x compliance_local.py"
     * run in command line the following command "./compliance_test.py <milestone> <date>"
     * run in command line the following command "./compliance_test.py <milestone> <date>"
    * wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)
 
 
   7. Get the logs from DUT:
   7. Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)
 
  8. Get the logs from DUT:
     - result-<milestone>-data.fulllog
     - result-<milestone>-data.fulllog
     - result-<milestone>-data.log
     - result-<milestone>-data.log
     - result-<milestone>-data.fail
     - result-<milestone>-data.fail
     - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)
     - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)
       -  the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to [[yi.zhao@windriver.com]] specifying the v
       -  the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to [[yi.zhao@windriver.com]] specifying the
         version and the type of image
         version and the type of image
     - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)
     - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)
       -- it has like 18 Mb
       -- it has like 18 Mb
       -- upload this file on drive and send the link to [[hongxu.jia@windriver.com]] and [[mark.hatle@windriver.com]]
       -- upload this file on drive and send the link to [[yi.zhao@windriver.com]]
       -- also I'll fwd an email to see how it looks
       -- also I'll fwd an email to see how it looks


   9. Put the tests from Testopia - Runtime test run on passed
   8. Put the tests from Testopia - Runtime test run on passed


=== Reporting Process  ===
=== Reporting Process  ===
<To Be Updated>
= Add New Tests =
<To be Updated>


=Test Approach=
=Test Approach=


=Test Matrix=
=Test Matrix=
'''Yocto QA Cycle'''
<!-- The table can be edited with this tool : http://www.tablesgenerator.com/mediawiki_tables -->
{| class="wikitable"
! Release
! CPU Class
! HW Platform
! BSP Name
! linux-yocto
! Image-type
! AUTO
! MANUAL
! Compliance
! pTest
|-
| '''generic BSPs (x86)'''
| Big Core
| MinwMax 32bit
| genericx86
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
| genericx86-wic
| 4.x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
| MinwMax 64bit
| genericx86-64
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
| genericx86-64-wic
| 4.x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
| NUC
| genericx86-64
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
| <span style="background:#00ff00"> Y </span>
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
| <span style="background:#00ff00"> Y </span>
|
|-
|
|
|
| genericx86-64-wic
| 4.x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
| VM
| QEMU
| qemux86
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
|
| 4,x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
| qemux86-64
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
|
| 4,x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
| '''generic BSPs (n-x86)'''
| MIPS
| EdgeRouter
| EdgeRouter
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
|
| 4,x
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
| PPC
| MPC8315e-rdb
| MPC8315e-rdb
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
| ARM
| Beaglebone Black
| beaglebone
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
| VM
| QEMU
| qemuarm
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
| qemuarm64
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
| qemuppc
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
| qemumips
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|-
|
|
|
| qemumips64
| 4.x
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|
|
|-
|
|
|
|
| 4,x-ltsi
| core-image-lsb-sdk
|
|
|
|
|}
'''Meta-Intel'''
{| class="wikitable"
! Release
! CPU Class
! HW Platform
! BSP Name
! Image-type
! Auto
! Manual
|-
| rowspan="18" style="text-align: center;" | meta-intel
| rowspan="18" | Big Core
| rowspan="3" | MinnowMax 64bit
| rowspan="3" | corei7-64
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|-
| rowspan="3" | NUC
| rowspan="3" | corei7-64
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|-
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|-
| rowspan="3" | CherryHill
| rowspan="3" | corei7-64
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|-
| rowspan="3" | Joule
| rowspan="3" | corei7-64
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-lsb-sdk
| <span style="background:#ff0000">  </span>
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|-
| rowspan="3" | MinnowMax 32bit
| rowspan="3" | core2-32
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
| <span style="background:#00ff00"> Y </span>
|-
| core-image-lsb-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|-
| rowspan="3" | meta-intel-quark 32bit
| rowspan="3" | intel-quark
| core-image-sato-sdk
| <span style="background:#00ff00"> Y </span>
|
|-
| core-image-lsb-sdk
|
|
|-
| core-image-sato-sdk WIC
| <span style="background:#00ff00"> Y </span>
|
|}


=Not Tested=
=Not Tested=

Latest revision as of 18:52, 13 February 2020

About BSP

A Board Support Package (BSP) is the collection of software and data that allows an operating system to run in a specific hardware. It contains the drivers, routines and information that defines how to support a particular hardware device, set of devices, or hardware platform.

Objectives

Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.


Team members

Member Team
Ee Peng Yeoh Intel
Sangeeta Jain Intel
Yi Zhao Wind River

Scope

Types of Tests


  • Manual tests on different platforms
  • Automated test on different platforms


Platforms, type of images and type of tests are showed detailed on BSP Test Matrix.


Features tested

  • Manual Tests
    • Boot image
    • Install Image
    • Multimedia
    • Graphical interface components
    • System reboot and shutdown recovery and standby
    • X server tests
    • rpm packages
    • File system (reading/writing)
    • Mont / Umont
    • Network

complete test steps of these manual tests are on poky repo under meta/lib/oeqa/manual/ <>

  • Automated Tests
    • buildcpio
    • buildgalculator
    • buildlzip
    • connman
    • date
    • df
    • dnf
    • gcc
    • kernelmodule
    • ldd
    • logrotate
    • multilib
    • oe_syslog
    • pam
    • parselogs
    • perl
    • ping
    • ptest
    • python
    • rpm
    • scanelf
    • scp
    • skeletoninit
    • ssh
    • systemd
    • x32lib
    • xorg

full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>

Test Strategy

As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.


Manual

Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image.


Automated Tests

Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.


Test Process

1. Prepare the image and device under test

   1.1 Download Artifacts
      there are 2 links available from Public AutoBuilder to download images 
         http://autobuilder.yoctoproject.org/pub/releases/
         https://autobuilder.yocto.io/pub/releases/
    
      Artifacts to be downloaded are: 
   
      Real HW
      under folder machines/<MACHINE_UNDER_TEST>
      - <IMAGE_TO_BE_TESTED>.hddimg
      - <IMAGE_TO_BE_TESTED>.wic
      - <IMAGE_TO_BE_TESTED>.wic.bmap
      - <IMAGE_TO_BE_TESTED>.manifest
      - <IMAGE_TO_BE_TESTED>.testdata.json
      example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*
      
      QEMU
      under folder machines/qemu/<MACHINE_UNDER_TEST>
      - <BZIMAGE>.bin
      - <IMAGE_TO_BE_TESTED>.ext4
      - <IMAGE_TO_BE_TESTED>.qemuboot.conf
      - <IMAGE_TO_BE_TESTED>.manifest
      - <IMAGE_TO_BE_TESTED>.testdata.json
      example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*
   
   1.2 For real hardware devices:
      - Write the image to a bootable media (USB key, SD Card, etc.) using the dd command:
          $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=TARGET_DEVICE
          example with usb key connected: $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=/dev/sdb
      - Attach the bootable device to the DUT
      - Turn the DUT on, and use its boot options to select the correct media for booting
      - After the DUT boots to image in USB stick, the default username is "root". (This is because the yocto image by default uses "debug-tweaks" feature)
      - Ensure the device has a network connection and a valid IP address

2. For Manual Test Cases:
  
   2.1 - Go to the proper Test Run assigned on the execution and start executing test cases  

3. For automated Test Cases:
   
   3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky
   3.2 $cd poky 
   3.3 Checkout to the commit specified on the release information $git checkout <commit-id>
   3.4 If testing meta-intel, clone the layer repo inside poky $git clone git://git.yoctoproject.org/meta-intel
       3.4.1 $cd meta-intel
       3.4.2 Checkout to the meta-intel commit specified on the release information $git checkout <commit-id>
       3.4.3 Return to the poky root path $cd ..
   3.5 $source oe-init-build-env
   3.6 If testing meta-intel, add the layer to conf/bblayers.conf. It may be done with:
       $bitbake-layers add-layer "../meta-intel"
   3.7 For real HW edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding: 
       Install / Boot the image on the DUT and check the IP
       INHERIT += "testimage"
       TEST_TARGET = "simpleremote"
       TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
       TEST_TARGET_IP = "DUT ip"  --> IP of the device under test
       if image is LSB 
       DISTRO ?= "poky-lsb"
   3.8 For QEMU edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding:
       INHERIT += "testimage"
       if image is LSB 
       DISTRO ?= "poky-lsb"
   3.9 Make sure conf only set DISTRO once, if DISTRO was set more than once, during testimage you could face error: 
   > ERROR: Nothing PROVIDES 'core-image-lsb-sdk'
   > core-image-lsb-sdk was skipped: missing required distro feature 'pam'
   > (not in DISTRO_FEATURES)

   For releases before 2.3 also add
   TEST_SUITES = " auto" 
   
   Remember to add the machine to local.conf, for example:
   If genericx86 is needed, MACHINE ?= "genericx86"
   If qemux86 is needed, MACHINE ?= "qemux86"
   If intel-corei7-64 is needed, MACHINE = "intel-corei7-64"

4. Prepare the environment to execute test suite 
   
   4.1 $bitbake rpm busybox curl run-postinsts
   4.2 $bitbake package-index
   4.3 Ensure that a folder is created, otherwise create it, under
       tmp/deploy/images/<MACHINE>, where <MACHINE> corresponds to the variable defined on
       local.conf on previous steps
   4.4 Paste artifacts downloaded on previously created folder 

5. Execute the test suite
   
   5.1 $bitbake <IMAGE_TESTED> -c testimage
      example bitbake core-image-sato-sdk -c testimage -v

6. Collect the test result from <build>/tmp/log/oeqa/testresults.json
   Note: Collect test results after one test run completed.

7. Verify that all test cases PASSED if not raise bugs properly 

   [Deprecated] 
    Update results to corresponding Testopia Test Run 
   There is a script available to upload results form the log automatically on QA-Tools and follow this example Update TR by ID

Bug Reporting

For correctly report the bugs follow the already defined steps here

Special Tests

There are 2 special test that also applied as BSP Components those are:

pTest

For detailed information about behavior and characteristics check Ptest

Test Process

This required 2 machine: a) host machine - to start/control poky/oeqa/runtime tests b) device under test (DUT) - to boot up the target ptest image and to run the ptest trigger by host machine 

1. Download pTest image from autobuilder (core-image-sato-sdk-ptest-genericx86-64.hddimg)

2. On DUT, install or write the image on a media connected to the DUT 
   Configure the network so it be able to work externally:
   - Export the proxy using "export http_proxy=<add your proxy link>" command eg
     $ export http_proxy=http://<my_proxy>:<port>
   - And check if you have internet connection, typing on terminal:
     $ wget www.google.com

3. On host machine
   Configure for running oeqa/runtime test (refer to above BSP runtime test instruction)
   Add to local.conf, 
   INHERIT += "testimage"
   TEST_SUITES = "ping ssh ptest"
   Then, start the runtime test and collect the test result from <build>/tmp/log/oeqa/testresults.json

Reporting Process

1. Generate the report of pTests as follows: 

    1.1 Collect the testresults.json files for both current release (base) and previous release (target). 
    1.2 Use "resulttool report" to compute the ptest summary result using current release (base). 
        $ resulttool report <file_path_to_current_ptest_testresultsjson>
    1.3 Use "resulttool regression" to compute the ptest regression as below
        $ resulttool regression <file_path_to_current_ptest_testresultsjson> <file_path_to_previous_ptest_testresultsjson>
    1.4 Copy and paste the report and regression report to wiki page https://wiki.yoctoproject.org/wiki/Ptest/archive

Compliance

In this section, 3 Test suites are executed on NUC HW with a core-image-lsb-sdk-genericx86-64 image.

  • LSB tests
  • POSIX tests
  • LTP tests

Test Process

1. Download the core-image-lsb-sdk-genericx86-64.hddimg image from Autobuilder (same image as in LSB automated testing for genericx86-64-lsb bsp)

2. Install the image on the DUT
   * We test compliance on a NUC

3. Configure the network so it be able to work externally:
   * edit /etc/resolv.conf and add the gateway ip_address
   * add the ip and netmask using "ifconfig" command
   * add the route using "route add default gw <ip_address>" 
   * export the proxy using "export http_proxy=<add your proxy link>" command
     : there is a bug and if you make these steps in another order than above, it may be possible not work

 4. Copy "compliance_test.py" script on DUT
    script can be found here

 5. Make sure that your network connection is working

 6. Run the script like this:
   * make the script executable: "chmod a+x compliance_local.py"
   * run in command line the following command "./compliance_test.py <milestone> <date>"
 
 7. Get the logs from DUT:
   - result-<milestone>-data.fulllog
   - result-<milestone>-data.log
   - result-<milestone>-data.fail
   - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)
     -  the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to yi.zhao@windriver.com specifying the
        version and the type of image
   - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)
     -- it has like 18 Mb
     -- upload this file on drive and send the link to yi.zhao@windriver.com
     -- also I'll fwd an email to see how it looks

 8. Put the tests from Testopia - Runtime test run on passed

Reporting Process

<To Be Updated>

Add New Tests

<To be Updated>

Test Approach

Test Matrix

Yocto QA Cycle

Release CPU Class HW Platform BSP Name linux-yocto Image-type AUTO MANUAL Compliance pTest
generic BSPs (x86) Big Core MinwMax 32bit genericx86 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk Y
genericx86-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
MinwMax 64bit genericx86-64 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
genericx86-64-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
NUC genericx86-64 4.x core-image-sato-sdk Y Y Y
4,x-ltsi core-image-lsb-sdk Y Y
genericx86-64-wic 4.x core-image-sato-sdk WIC Y
4,x-ltsi core-image-lsb-sdk
VM QEMU qemux86 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
qemux86-64 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
generic BSPs (n-x86) MIPS EdgeRouter EdgeRouter 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x core-image-sato-sdk WIC Y
PPC MPC8315e-rdb MPC8315e-rdb 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x-ltsi core-image-sato-sdk WIC Y
ARM Beaglebone Black beaglebone 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
4,x-ltsi core-image-sato-sdk WIC Y
VM QEMU qemuarm 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemuarm64 4.x core-image-sato-sdk Y Y
4,x-ltsi core-image-lsb-sdk
qemuppc 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemumips 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk
qemumips64 4.x core-image-sato-sdk Y
4,x-ltsi core-image-lsb-sdk


Meta-Intel

Release CPU Class HW Platform BSP Name Image-type Auto Manual
meta-intel Big Core MinnowMax 64bit corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
NUC corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
CherryHill corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
Joule corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk
core-image-sato-sdk WIC Y
MinnowMax 32bit core2-32 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
core-image-sato-sdk WIC Y
meta-intel-quark 32bit intel-quark core-image-sato-sdk Y
core-image-lsb-sdk
core-image-sato-sdk WIC Y

Not Tested

Requirements

Tools

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=BSP_Test_Plan&oldid=68995"