Quality Assurance yocto project: Difference between revisions

Latest revision as of 15:57, 10 October 2017

Steps for BSP Automated Testing

1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)

2. Make checkout on the poky commit from the mail (in command line):

a) Clone a clean version of poky:

$ git clone git://git.yoctoproject.org/poky"

b)In poky directory checkout on the desired commit:

$ git checkout <commit-id>

3. Source the environment, from poky directory

 $ cd ~/poky

$ source oe-init-build-env

4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
TEST_TARGET_IP = "DUT ip"  --> IP of the device under test

For releases before 2.3 also add 

TEST_SUITES = " auto" 

Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"

5. In ~poky/build directory, in command line, type the following commands:

$ bitbake rpm busybox run-postinsts
$ bitbake package-index

NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) and the testdata.json (core-image-sato-sdk-genericx86.testdata.json) from autobuilder at the following path: "~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)

$ bitbake core-image-sato-sdk -c testimage

If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip

Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

After a while the testing will be finished

NOTE: For QEMU images there are additional steps that need to be followed. They are detailed on

https://wiki.yoctoproject.org/wiki/Steps_for_Automated_Testing_on_QEMU

Steps for LSB Automated Testing

1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.

2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):

The #1 steps are followed for lsb images

download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)

NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

3. The checkout step is the same as BSP.

4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro

DISTRO ?= "poky-lsb"

From command line type the following commands

$ bitbake rpm busybox run-postinsts (wait for tasks to be executed)

$ bitbake package-index (wait for tasks to be executed)

NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path: "~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)

Type the following command from terminal:

 $ bitbake core-image-lsb-sdk -c testimage

Defect/Bug life Cycle

If you need more information about bug information and levels, please go to https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels

Weekly Testing (automated)

1. Genericx86 weekly tests:

a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)

b) make checkout on the poky commit from the mail (in command line):

- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"

- in poky directory checkout on the desired commit: "git checkout <commit-id>"

c) source the environment:

- in poky directory ( "cd ~/poky"): " source oe-init-build-env"

d) add in local. conf :

* local.conf path: ~/poky/build/conf

- at the end of file add:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "DUT ip"
TEST_SERVER_IP = "workstation ip"

- change the machine: MACHINE ??= "genericx86"

e) in ~poky/build directory, in command line, type the following commands:

- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)

- "bitbake package-index" (wait for tasks to be executed)

- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:

"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)

- "bitbake core-image-sato-sdk -c testimage"

If having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip

Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET

2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.

3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):

- the #1 steps are followed for lsb images

a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)

* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)

b) the checkout step is the same as #1

c) same as #1

d) add in local. conf :

* local.conf path: ~/poky/build/conf

- at the end of file add:

INHERIT += "testimage"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "DUT ip"
TEST_SERVER_IP = "workstation ip"

- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")

- change the distro : DISTRO ?= "poky-lsb"

e) in command line (~poky/build directory) type the following commands:

- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)

- "bitbake package-index" (wait for tasks to be executed)

- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:

"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)

- "bitbake core-image-lsb-sdk -c testimage"

Compliance Testing

Runtime compliance steps:

1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)

* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk

2) Install the image on DUT

3) Configure the network so it be able to work externally:

- Export the proxy using "export http_proxy=<add your proxy link>" command eg

$ export http_proxy=http://proxy-chain.intel.com:911

And check if you have internet connection, typing on terminal:

$ wget www.google.com

* There is a bug and if you make these steps in another order than above, it may be possible not work

4) Copy "compliance_test.py" script on DUT

- Make the script executable: "chmod a+x compliance_local.py"

- Run in command line the following command:

 ./compliance_test.py <milestone> <date>"

- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)

8) Get the logs from DUT:

- Results fulllog: /opt/ltp

- results.log: /opt/ltp/results

- results.fail: /opt/ltp/output

- Results posix: /opt/ltp/testcases/open_posix_testsuites

* 18 MB File: /var/opt/lsb/test/manager/results/x86.../x86....targz

* send these log to Yi Zhao specifying the version and the type of image

- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)

* It has like 18 Mb

* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com

9) Put the tests from Testopia - Runtime test run on passed

pTest Run

pTest run steps:

1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)

* we test it on NUC with genericx86-64

2. Install the image on DUT (using legacy boot)

3) Configure the network so it be able to work externally:

- Export the proxy using "export http_proxy=<add your proxy link>" command eg

$ export http_proxy=http://proxy-chain.intel.com:911

And check if you have internet connection, typing on terminal:

$ wget www.google.com

4. In command line type

$ ptest-runner > ptest.log

and wait for it to finish ( about 5 hours)

Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.

"> ptest.log", in order to save the test results in that file.

@@ Line 21: / Line 21: @@
 INHERIT += "testimage"
 TEST_TARGET = "simpleremote"
-TEST_TARGET_IP = "DUT ip"
+TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
-TEST_SUITES = " auto"
+TEST_TARGET_IP = "DUT ip"  --> IP of the device under test
+For releases before 2.3 also add
+TEST_SUITES = " auto"
 Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"
 </pre>
 <p>
-. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf
+. In ~poky/build directory, in command line, type the following commands:
-</p>
-<ul><li>in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above</li></ul>
-<p>
-. In ~poky/build directory, in command line, type the following commands:
 </p>
 <pre>
-$ bitbake rpm psplash
+$ bitbake rpm busybox run-postinsts
-$ bitbake package-index
 $ bitbake package-index
 </pre>
 <p>
-NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:
+NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest)  and the testdata.json (core-image-sato-sdk-genericx86.testdata.json) from autobuilder at the following path:
 "~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)
 </p>
@@ Line 46: / Line 48: @@
 <p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p>
 <p>After a while the testing will be finished</p>
+<p>NOTE: For QEMU images there are additional steps that need to be followed. They are detailed on</p>
+<p>https://wiki.yoctoproject.org/wiki/Steps_for_Automated_Testing_on_QEMU</p>
 <h2>'''Steps for LSB Automated Testing'''</h2>
@@ Line 60: / Line 65: @@
 <p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p>
 <ul><li>DISTRO ?= "poky-lsb"</li></ul>
-NOTE: The bblayers configuration is the same as BSP.
 <p>From command line type the following commands</p>
-<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre>
+<pre>$ bitbake rpm busybox run-postinsts (wait for tasks to be executed)</pre>
 <pre>$ bitbake package-index (wait for tasks to be executed)</pre>
 NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:
@@ Line 69: / Line 73: @@
 <pre> $ bitbake core-image-lsb-sdk -c testimage </pre>
-<h2>'''pTest and runtime'''</h2>
-<p>1. pTest run steps:</p>
+<h2>'''Defect/Bug life Cycle'''</h2>
-<p>1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p>
+----
-* Usually tested on NUC with genericx86-64
+[[File:Yp_BugLifeCycle.png]]
-<p>2) Install the image on DUT (using legacy boot)</p>
-<p>3) Boot the image and copy "ptest-runner.sh" script on DUT</p>
-<p>4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)</p>
-<p>* P.S. : I will send you also the script via google drive, so you can open it</p>
+<p><strong>If you need more information about bug information and levels, please go to
+https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels</strong></p>
-<p>2. Runtime compliance steps:</p>
+<h2>'''Weekly Testing (automated)'''</h2>
+<p>1. Genericx86 weekly tests:</p>
+	<p>a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)</p>
+	<p>b) make checkout on the poky commit from the mail (in command line):</p>
+		<p>- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"</p>
+		<p>- in poky directory checkout on the desired commit: "git checkout <commit-id>"</p>
+	<p>c) source the environment:</p>
+		<p>- in poky directory ( "cd ~/poky"): " source oe-init-build-env"</p>
+	<p>d) add in local. conf :</p>
+		<p>* local.conf path: ~/poky/build/conf</p>
+		<p>- at the end of file add:</p>
+<pre>
+INHERIT += "testimage"
+TEST_TARGET = "simpleremote"
+TEST_TARGET_IP = "DUT ip"
+TEST_SERVER_IP = "workstation ip"
+</pre>
+			<p>- change the machine: MACHINE ??= "genericx86"</p>
+	<p>e) in ~poky/build directory, in command line, type the following commands:</p>
+		<p>- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)</p>
+		<p>- "bitbake package-index" (wait for tasks to be executed)</p>
+		<p>	- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:</p>
+				<p>"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)</p>
+		<p>- "bitbake core-image-sato-sdk -c testimage"</p>
+	<p>If having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p>
+	<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p>
+<p>2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.</p>
+<p>3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):</p>
+	<p>- the #1 steps are followed for lsb images</p>
+	<p>a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</p>
+		<p>* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)</p>
+	<p>b) the checkout step is the same as #1</p>
+	<p>c) same as #1</p>
+	<p>d) add in local. conf :</p>
+		<p>* local.conf path: ~/poky/build/conf</p>
+		<p>- at the end of file add:</p>
+<pre>
+INHERIT += "testimage"
+TEST_TARGET = "simpleremote"
+TEST_TARGET_IP = "DUT ip"
+TEST_SERVER_IP = "workstation ip"
+</pre>
+		<p>- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")</p>
+		<p>- change the distro : DISTRO ?= "poky-lsb"</p>
+	<p>e) in command line (~poky/build directory) type the following commands:</p>
+		<p>- "bitbake rpm busybox run-postinsts" (wait for tasks to be executed)</p>
+		<p>- "bitbake package-index" (wait for tasks to be executed)</p>
+			<p>- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:</p>
+				<p>"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)</p>
+		<p>- "bitbake core-image-lsb-sdk -c testimage"</p>
+<h2> '''Compliance Testing'''</h2>
+ <p>Runtime compliance steps:</p>
 	<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p>
 		<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p>
-	<p>2) Install the image on DUT </p>
+	<p>2) Install the image on DUT</p>
-	<p>3) Configure the network so it be able to work externally: </p>
+	<p>3) Configure the network so it be able to work externally:</p>
-		<p> - edit /etc/resolv.conf and add the gateway ip_address </p>
+		<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p>
-		<p> - add the ip and netmask using "ifconfig" command </p>
+<pre>$ export http_proxy=http://proxy-chain.intel.com:911</pre>
-		<p> - add the route using "route add default gw <ip_address>" </p>
+<p>And check if you have internet connection, typing on terminal:</p>
-		<p> - export the proxy using "export http_proxy=<add your proxy link>" command </p>
+<pre>$ wget www.google.com</pre>
-		<p> * there is a bug and if you make these steps in another order than above, it may be possible not work </p>
+		<p>* There is a bug and if you make these steps in another order than above, it may be possible not work</p>
 	<p>4) Copy "compliance_test.py" script on DUT</p>
-	<p>5) Make sure that your network connection is working </p>
-	<p>6) Run the script like this: </p>
+		<p>- Make the script executable: "chmod a+x compliance_local.py"</p>
-		<p>- make the script executable:</p>
+		<p>- Run in command line the following command:</p>
-                    <pre>"$ chmod a+x compliance_local.py" </pre>
+<pre> ./compliance_test.py <milestone> <date>" </pre>
-		<p>- run in command line the following command " </p>
+		<p>- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p>
-                <pre>./compliance_test.py <milestone> <date>" </pre>
-		<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours) </p>
+	<p>8) Get the logs from DUT:</p>
-	<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day) </p>
-	<p>8) Get the logs from DUT: </p>
+		<p>- Results fulllog: /opt/ltp</p>
-		<p>- result-<milestone>-data.fulllog </p>
+		<p>- results.log: /opt/ltp/results</p>
-		<p>- result-<milestone>-data.log </p>
+		<p>- results.fail: /opt/ltp/output</p>
-		<p>- result-<milestone>-data.fail </p>
+		<p>- Results posix: /opt/ltp/testcases/open_posix_testsuites</p>
-		<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) </p>
+		<p>* 18 MB  File: /var/opt/lsb/test/manager/results/x86.../x86....targz </p>
-		<p>* the three others are found in /opt/ltp directory, in output, temp, result folders  </p>
+			<p>* send these log to Yi Zhao specifying the version and the type of image</p>
-			<p>* send these log to Yi Zhao specifying the version and the type of image </p>
-		<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz </p>
+		<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p>
-			<p>* It has like 18 Mb </p>
+			<p>* It has like 18 Mb</p>
-			<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com </p>
+			<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p>
-	<p>9) After this, update the tests from Testopia - Runtime test run on passed  </p>
+	<p>9) Put the tests from Testopia - Runtime test run on passed </p>
-<h2>'''Defect/Bug life Cycle'''</h2>
+<h2>''' pTest Run'''</h2>
-----
-[[File:Yp_BugLifeCycle.png]]
+<p>pTest run steps:</p>
+<p>1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p>
+		<p>* we test it on NUC with genericx86-64</p>
+<p>2. Install the image on DUT (using legacy boot)</p>
+<p>3) Configure the network so it be able to work externally:</p>
+		<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p>
+<pre>$ export http_proxy=http://proxy-chain.intel.com:911</pre>
+<p>And check if you have internet connection, typing on terminal:</p>
+<pre>$ wget www.google.com</pre>
+<p>4. In command line type</p>
+<pre>$ ptest-runner > ptest.log </pre>
+<p>and wait for it to finish ( about 5 hours)</p>
+<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p>
+<p> "> ptest.log", in order to save the test results in that file. </p>

Quality Assurance yocto project: Difference between revisions

Latest revision as of 15:57, 10 October 2017

Contents

Steps for BSP Automated Testing

Steps for LSB Automated Testing

Defect/Bug life Cycle

Weekly Testing (automated)

Compliance Testing

pTest Run

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools