Quality Assurance yocto project

Steps for BSP Automated Testing

1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)

2. Make checkout on the poky commit from the mail (in command line):

a) Clone a clean version of poky:

$ git clone git://git.yoctoproject.org/poky"

b)In poky directory checkout on the desired commit:

$ git checkout <commit-id>

3. Source the environment, from poky directory

 $ cd ~/poky 

$ source oe-init-build-env

4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
TEST_TARGET_IP = "DUT ip"  --> IP of the device under test 
TEST_SUITES = " auto"
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"

5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf

• in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above

6. In ~poky/build directory, in command line, type the following commands:

$ bitbake rpm psplash
$ bitbake package-index

NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)

$ bitbake core-image-sato-sdk -c testimage

If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip

Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

After a while the testing will be finished

Steps for LSB Automated Testing

1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.

2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):

• The #1 steps are followed for lsb images

• download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)

NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

3. The checkout step is the same as BSP.

4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro

• DISTRO ?= "poky-lsb"

NOTE: The bblayers configuration is the same as BSP.

From command line type the following commands

$ bitbake rpm psplash (wait for tasks to be executed)

$ bitbake package-index (wait for tasks to be executed)

NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)

Type the following command from terminal:

 $ bitbake core-image-lsb-sdk -c testimage 

pTest and runtime

1. pTest run steps:

1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)

• Usually tested on NUC with genericx86-64

2) Install the image on DUT (using legacy boot)

3) Boot the image and copy "ptest-runner.sh" script on DUT

4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)

* P.S. : I will send you also the script via google drive, so you can open it

2. Runtime compliance steps:

1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)

* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk

2) Install the image on DUT

3) Configure the network so it be able to work externally:

- edit /etc/resolv.conf and add the gateway ip_address

- add the ip and netmask using "ifconfig" command

- add the route using "route add default gw <ip_address>"

- export the proxy using "export http_proxy=<add your proxy link>" command

* there is a bug and if you make these steps in another order than above, it may be possible not work

4) Copy "compliance_test.py" script on DUT

5) Make sure that your network connection is working

6) Run the script like this:

- make the script executable:

"$ chmod a+x compliance_local.py" 

- run in command line the following command "

./compliance_test.py <milestone> <date>" 

- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)

7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)

8) Get the logs from DUT:

- result-<milestone>-data.fulllog

- result-<milestone>-data.log

- result-<milestone>-data.fail

- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)

* the three others are found in /opt/ltp directory, in output, temp, result folders

* send these log to Yi Zhao specifying the version and the type of image

- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz

* It has like 18 Mb

* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com

9) After this, update the tests from Testopia - Runtime test run on passed

Defect/Bug life Cycle

---

If you need more information about bug information and levels, please go to

https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels

Weekly Testing (automated)

1. Genericx86 weekly tests: a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail) b) make checkout on the poky commit from the mail (in command line): - clone a clean version of poky: "git clone git://git.yoctoproject.org/poky" - in poky directory checkout on the desired commit: "git checkout <commit-id>" c) source the environment: - in poky directory ( "cd ~/poky"): " source oe-init-build-env" d) add in local. conf : * local.conf path: ~/poky/build/conf - at the end of file add: INHERIT += "testimage"

TEST_TARGET = "simpleremote"

TEST_TARGET_IP = "DUT ip"

TEST_SERVER_IP = "workstation ip"

TEST_SUITES = " auto" - change the machine: MACHINE ??= "genericx86" e) add in bblayers.conf: * bblayers.conf path: ~/poky/build/conf - in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above f) in ~poky/build directory, in command line, type the following commands: - "bitbake rpm psplash" (wait for tasks to be executed) - "bitbake package-index" (wait for tasks to be executed) - after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path) - "bitbake core-image-sato-sdk -c testimage" if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.

3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb): - the #1 steps are followed for lsb images a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail) * the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

b) the checkout step is the same as #1 c) same as #1 d) add in local. conf : * local.conf path: ~/poky/build/conf - at the end of file add: INHERIT += "testimage"

TEST_TARGET = "simpleremote"

TEST_TARGET_IP = "DUT ip"

TEST_SERVER_IP = "workstation ip"

TEST_SUITES = " auto" - change the machine: MACHINE ??= "genericx86" (or "genericx86-64") - change the distro : DISTRO ?= "poky-lsb" e) in bblayers.conf add meta-selftest as in #1 f) in command line (~poky/build directory) type the following commands: - "bitbake rpm psplash" (wait for tasks to be executed) - "bitbake package-index" (wait for tasks to be executed) - after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path) - "bitbake core-image-lsb-sdk -c testimage"

Compliance Testing

Runtime compliance steps:

1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp) * we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk 2) Install the image on DUT 3) Configure the network so it be able to work externally: - edit /etc/resolv.conf and add the gateway ip_address – yocto -> Broadcast de maquina, sudo rounte –n - add the ip and netmask using "ifconfig" command - yocto - add the route using "route add default gw <ip_address>" -> broadcast de maquina - export the proxy using "export http_proxy=<add your proxy link>" command eg Export http_proxy=http://proxy-chain.intel.com:911. And check if you have internet connection. With $wget www.google.com * there is a bug and if you make these steps in another order than above, it may be possible not work 4) Copy "compliance_test.py" script on DUT 5) Make sure that your network connection is working 6) Run the script like this: - make the script executable: "chmod a+x compliance_local.py" - run in command line the following command "./compliance_test.py <milestone> <date>" - wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours) 7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day) 8) Get the logs from DUT: - result-<milestone>-data.fulllog - result-<milestone>-data.log - result-<milestone>-data.fail - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) * the three others are found in /opt/ltp directory, in output, temp, result folders * send these log to Yi Zhao specifying the version and the type of image * I'll fwd you an email so you could see an example - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily) * it has like 18 Mb * upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com * also I'll fwd an email to see how it looks 9) Put the tests from Testopia - Runtime test run on passed

• the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E

pTest Run

1. pTest run steps: 1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory) * we test it on NUC with genericx86-64 2) Install the image on DUT (using legacy boot) 3) Boot the image and copy "ptest-runner.sh" script on DUT 4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)