Quality Assurance yocto project

Steps for BSP Automated Testing

1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)

2. Make checkout on the poky commit from the mail (in command line):

a) Clone a clean version of poky:

$ git clone git://git.yoctoproject.org/poky"

b)In poky directory checkout on the desired commit:

$ git checkout <commit-id>

3. Source the environment, from poky directory

 $ cd ~/poky 

$ source oe-init-build-env

4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
TEST_TARGET_IP = "DUT ip"  --> IP of the device under test

For releases before 2.3 also add 

TEST_SUITES = " auto" 

Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"

5. In ~poky/build directory, in command line, type the following commands:

$ bitbake rpm busybox run-postinsts
$ bitbake package-index

NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) and the testdata.json (core-image-sato-sdk-genericx86.testdata.json) from autobuilder at the following path: "~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)

$ bitbake core-image-sato-sdk -c testimage

If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip

Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

After a while the testing will be finished

NOTE: For QEMU images there are additional steps that need to be followed. They are detailed on

https://wiki.yoctoproject.org/wiki/Steps_for_Automated_Testing_on_QEMU

Steps for LSB Automated Testing

1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.

2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):

• The #1 steps are followed for lsb images

• download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)

NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

3. The checkout step is the same as BSP.

4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro

• DISTRO ?= "poky-lsb"

From command line type the following commands

$ bitbake rpm busybox run-postinsts (wait for tasks to be executed)

$ bitbake package-index (wait for tasks to be executed)

NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)

Type the following command from terminal:

 $ bitbake core-image-lsb-sdk -c testimage 

Defect/Bug life Cycle

---

If you need more information about bug information and levels, please go to https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels

Weekly Testing (automated)

1. Genericx86 weekly tests:

a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)

b) make checkout on the poky commit from the mail (in command line):

- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"

- in poky directory checkout on the desired commit: "git checkout <commit-id>"

c) source the environment:

- in poky directory ( "cd ~/poky"): " source oe-init-build-env"

d) add in local. conf :

* local.conf path: ~/poky/build/conf

- at the end of file add:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "DUT ip"
TEST_SERVER_IP = "workstation ip"

- change the machine: MACHINE ??= "genericx86"

e) in ~poky/build directory, in command line, type the following commands:

- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)

- "bitbake package-index" (wait for tasks to be executed)

- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:

"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)

- "bitbake core-image-sato-sdk -c testimage"

If having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip

Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.

3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):

- the #1 steps are followed for lsb images

a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)

* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

b) the checkout step is the same as #1

c) same as #1

d) add in local. conf :

* local.conf path: ~/poky/build/conf

- at the end of file add:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "DUT ip"
TEST_SERVER_IP = "workstation ip"

- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")

- change the distro : DISTRO ?= "poky-lsb"

e) in command line (~poky/build directory) type the following commands:

- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)

- "bitbake package-index" (wait for tasks to be executed)

- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:

"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)

- "bitbake core-image-lsb-sdk -c testimage"

Compliance Testing

Runtime compliance steps:

1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)

* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk

2) Install the image on DUT

3) Configure the network so it be able to work externally:

- Export the proxy using "export http_proxy=<add your proxy link>" command eg

$ export http_proxy=http://proxy-chain.intel.com:911

And check if you have internet connection, typing on terminal:

$ wget www.google.com

* There is a bug and if you make these steps in another order than above, it may be possible not work

4) Copy "compliance_test.py" script on DUT

- Make the script executable: "chmod a+x compliance_local.py"

- Run in command line the following command:

 ./compliance_test.py <milestone> <date>" 

- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)

8) Get the logs from DUT:

- Results fulllog: /opt/ltp

- results.log: /opt/ltp/results

- results.fail: /opt/ltp/output

- Results posix: /opt/ltp/testcases/open_posix_testsuites

* 18 MB File: /var/opt/lsb/test/manager/results/x86.../x86....targz

* send these log to Yi Zhao specifying the version and the type of image

- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)

* It has like 18 Mb

* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com

9) Put the tests from Testopia - Runtime test run on passed

pTest Run

pTest run steps:

1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)

* we test it on NUC with genericx86-64

2. Install the image on DUT (using legacy boot)

3) Configure the network so it be able to work externally:

- Export the proxy using "export http_proxy=<add your proxy link>" command eg

$ export http_proxy=http://proxy-chain.intel.com:911

And check if you have internet connection, typing on terminal:

$ wget www.google.com

4. In command line type

$ ptest-runner > ptest.log 

and wait for it to finish ( about 5 hours)

Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.

"> ptest.log", in order to save the test results in that file.