BSP Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with "c")
 
No edit summary
Line 1: Line 1:
c
= About BSP  =
 
A Board Support Package (BSP) is a collection of information that defines how to support a particular hardware device, set of devices, or hardware platform.
 
= Objectives =
 
Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.
 
 
= Team members =
 
 
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
| align=center style="background-color:#0000cc;border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;color:#ffffff;" | '''Member'''
| align=center style="background-color:#0000cc;border:0.05pt solid #000000;padding:0.0382in;color:#ffffff;" | '''Team'''
|-
|}
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | José Pérez Carranza
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel
|-
|}
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Oscar Lopez Arandas
| style="border:0.05pt solid #000000;padding:0.0382in;" | Intel
|-
|}
{| style="border-spacing:0;margin:auto;width:6.925in;"
|-
| style="border-top:0.05pt solid #000000;border-bottom:0.05pt solid #000000;border-left:0.05pt solid #000000;border-right:none;padding:0.0382in;" | Yi Zaho
| style="border:0.05pt solid #000000;padding:0.0382in;" | Wind River
|-
|}
 
 
= Scope =
 
 
 
'''Types of Tests '''
 
 
* Manual tests on different platforms
* Automated test on different platforms
 
 
Platforms, type of images and type of tests are showed detailed on BSP Test Matrix.
 
 
'''Features tested'''
 
* Manual Tests
** Boot image
** Install Image
** Multimedia
** Graphical interface components
** System reboot and shutdown recovery and standby
** X server tests
** rpm packages
** File system (reading/writing)
** Mont / Umont
** Network
 
All test cases can be found here <>
 
* Automated Tests
** buildcpio
** buildgalculator
** buildlzip
** connman
** date
** df
** dnf
** gcc
** kernelmodule
** ldd
** logrotate
** multilib
** oe_syslog
** pam
** parselogs
** perl
** ping
** ptest
** python
** rpm
** scanelf
** scp
** skeletoninit
** ssh
** systemd
** x32lib
** xorg
 
full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>
 
 
= Test Strategy  =
 
As mentioned bellow there are 2 types of test Automated an Manual tests, below a description of how those test are applied.
 
 
==Manual==
 
Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image.
 
 
==Automated Tests==
 
 
Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.
 
 
==Test Process==
 
1. Download Artifacts
    there are 2 links available from Public AutoBuilder to download images
      http://autobuilder.yoctoproject.org/pub/releases/
      https://autobuilder.yocto.io/pub/releases/
   
    Artifacts to be downloaded are:
   
    '''Real HW'''
    under folder machines/<MACHINE_UNDER_TEST>
    - <IMAGE_TO_BE_TESTED>.hhddimg
    - <IMAGE_TO_BE_TESTED>.wic
    - <IMAGE_TO_BE_TESTED>.wic.bmap
    - <IMAGE_TO_BE_TESTED>.manifest
    - <IMAGE_TO_BE_TESTED>.testdata.json
    '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*'''
   
    '''QEMU'''
    under folder machines/qemu/<MACHINE_UNDER_TEST>
    - <BZIMAGE>.bin
    - <IMAGE_TO_BE_TESTED>.ext4
    - <IMAGE_TO_BE_TESTED>.qemuboot.conf
    - <IMAGE_TO_BE_TESTED>.manifest
    - <IMAGE_TO_BE_TESTED>.testdata.json
    '''example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*'''
 
2. For Manual Test Cases:
 
  2.1 - Go to the proper Test Run assigned on the execution and start executing test cases 
 
3. For automated Test Cases
   
    3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky"
    3.2 Checkout to the commit specified on the release information $git checkout <commit-id>
    3.3 $cd poky
    3.4 $source oe-init-build-env
    3.5 For real HW edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:
        INHERIT += "testimage"
        TEST_TARGET = "simpleremote"
        TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
        TEST_TARGET_IP = "DUT ip"  --> IP of the device under tes
        if image is LSB
        DISTRO ?= "poky-lsb"
    3.6 For real QEMU edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:
        INHERIT += "testimage"
        if image is LSB
        DISTRO ?= "poky-lsb"
    '''For releases before 2.3 also add'''
    TEST_SUITES = " auto"
   
    '''Remember to add the machine to local.conf for example:'''
    if genericx86 is needed, MACHINE ?= "genericx86
    if qemux86 is needed, MACHINE ?= "qemux86
 
4. Prepare the environment to execute test suite
   
    4.1 $bitbake rpm busybox run-postinsts
    4.2 $bitbake package-index
    4.3 Ensure that a folder is create under tmp/deploy/image/<MACHINE>
        where <MACHINE> corresponds to variable defined on local.conf on previous steps
    4.4 Paste artifacts downloaded on previously created folder
 
5. Execute the test suite
   
    5.1 $bitbake <IMAGE_TESTED> -c testimage
      '''example bitbake core-image-sato-sdk -c testimage'''
 
6. Verify that all test cases PASSED if not raise bugs properly
 
7. Update results to corresponding Testopia Test Run
    '''There is a script available to upload results form the log automatically on [here]'''
 
 
==Bug Reporting==
 
For correctly report the bugs follow the already defined steps here
 
= Special Tests =
 
There are 2 special test that also applied as BSP Components those are:
 
== pTest ==
 
<add link to the process>
 
== Compliance ==
 
<add link to the process>
 
=Test Approach=
 
=Test Matrix=
 
=Not Tested=
 
=Requirements=
 
=Tools=

Revision as of 19:10, 20 September 2017

About BSP

A Board Support Package (BSP) is a collection of information that defines how to support a particular hardware device, set of devices, or hardware platform.

Objectives

Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.


Team members

Member Team
José Pérez Carranza Intel
Oscar Lopez Arandas Intel
Yi Zaho Wind River


Scope

Types of Tests


  • Manual tests on different platforms
  • Automated test on different platforms


Platforms, type of images and type of tests are showed detailed on BSP Test Matrix.


Features tested

  • Manual Tests
    • Boot image
    • Install Image
    • Multimedia
    • Graphical interface components
    • System reboot and shutdown recovery and standby
    • X server tests
    • rpm packages
    • File system (reading/writing)
    • Mont / Umont
    • Network

All test cases can be found here <>

  • Automated Tests
    • buildcpio
    • buildgalculator
    • buildlzip
    • connman
    • date
    • df
    • dnf
    • gcc
    • kernelmodule
    • ldd
    • logrotate
    • multilib
    • oe_syslog
    • pam
    • parselogs
    • perl
    • ping
    • ptest
    • python
    • rpm
    • scanelf
    • scp
    • skeletoninit
    • ssh
    • systemd
    • x32lib
    • xorg

full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>


Test Strategy

As mentioned bellow there are 2 types of test Automated an Manual tests, below a description of how those test are applied.


Manual

Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image.


Automated Tests

Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.


Test Process

1. Download Artifacts 
   there are 2 links available from Public AutoBuilder to download images 
      http://autobuilder.yoctoproject.org/pub/releases/
      https://autobuilder.yocto.io/pub/releases/
    
   Artifacts to be downloaded are: 
   
   Real HW
   under folder machines/<MACHINE_UNDER_TEST>
   - <IMAGE_TO_BE_TESTED>.hhddimg
   - <IMAGE_TO_BE_TESTED>.wic
   - <IMAGE_TO_BE_TESTED>.wic.bmap
   - <IMAGE_TO_BE_TESTED>.manifest
   - <IMAGE_TO_BE_TESTED>.testdata.json
   example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*
   
   QEMU
   under folder machines/qemu/<MACHINE_UNDER_TEST>
   - <BZIMAGE>.bin
   - <IMAGE_TO_BE_TESTED>.ext4
   - <IMAGE_TO_BE_TESTED>.qemuboot.conf
   - <IMAGE_TO_BE_TESTED>.manifest
   - <IMAGE_TO_BE_TESTED>.testdata.json
   example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*

2. For Manual Test Cases:
  
  2.1 - Go to the proper Test Run assigned on the execution and start executing test cases  

3. For automated Test Cases
   
   3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky"
   3.2 Checkout to the commit specified on the release information $git checkout <commit-id>
   3.3 $cd poky 
   3.4 $source oe-init-build-env
   3.5 For real HW edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: 
       INHERIT += "testimage"
       TEST_TARGET = "simpleremote"
       TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
       TEST_TARGET_IP = "DUT ip"  --> IP of the device under tes
       if image is LSB 
       DISTRO ?= "poky-lsb"
   3.6 For real QEMU edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:
       INHERIT += "testimage"
       if image is LSB 
       DISTRO ?= "poky-lsb"
   For releases before 2.3 also add
   TEST_SUITES = " auto" 
   
   Remember to add the machine to local.conf for example:
   if genericx86 is needed, MACHINE ?= "genericx86
   if qemux86 is needed, MACHINE ?= "qemux86

4. Prepare the environment to execute test suite 
   
   4.1 $bitbake rpm busybox run-postinsts
   4.2 $bitbake package-index
   4.3 Ensure that a folder is create under tmp/deploy/image/<MACHINE>
       where <MACHINE> corresponds to variable defined on local.conf on previous steps
   4.4 Paste artifacts downloaded on previously created folder 

5. Execute the test suite
   
   5.1 $bitbake <IMAGE_TESTED> -c testimage
      example bitbake core-image-sato-sdk -c testimage

6. Verify that all test cases PASSED if not raise bugs properly 

7. Update results to corresponding Testopia Test Run 
   There is a script available to upload results form the log automatically on [here]


Bug Reporting

For correctly report the bugs follow the already defined steps here

Special Tests

There are 2 special test that also applied as BSP Components those are:

pTest

<add link to the process>

Compliance

<add link to the process>

Test Approach

Test Matrix

Not Tested

Requirements

Tools

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=BSP_Test_Plan&oldid=31752"