Yocto Project - User contributions [en]

Distro Testing Plan

2017-02-02T23:46:54Z

Benjamin: /* Process To Install New Distro */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro

2. Install the requirements from the Yocto Project Quickstart

3. Start an Autobuilder staging instance

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro
$ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass

6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build

7. Wait to the buildsets to end and review the output, load bugs if failures occur

8. Stop the staging Autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Distro Testing Plan

2017-02-02T23:45:03Z

Benjamin: /* Process To Install New Distro */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro

2. Install the requirements from the Yocto Project Quickstart

3. Start an Autobuilder staging instance

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass

6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build

7. Wait to the buildsets to end and review the output, load bugs if failures occur

8. Stop the staging Autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Distro Testing Plan

2017-02-02T23:44:39Z

Benjamin: /* Process To Install New Distro */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro

2. Install the requirements from the Yocto Project Quickstart

3. Start an Autobuilder staging instance

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010

4.1 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass

6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build

7. Wait to the buildsets to end and review the output, load bugs if failures occur

8. Stop the staging Autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Distro Testing Plan

2017-02-02T23:43:28Z

Benjamin: /* Process To Install New Distro */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro

2. Install the requirements from the Yocto Project Quickstart

3. Start an Autobuilder staging instance

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010
4.1 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass

6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build

7. Wait to the buildsets to end and review the output, load bugs if failures occur

8. Stop the staging Autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Distro Testing Plan

2017-02-02T23:41:00Z

Benjamin: /* Open a Bug */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro
2. Intsall the requirements from the YP Quickstart 3. Start YP Autobuilder

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010
4.1 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass 6. Enter to http://localhost:8010/builders/nightly-qa-distro and Force a build 7. Wait to the buildsets to end and review the output, load bugs if necessary 8. Stops autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Distro Testing Plan

2017-02-02T23:40:11Z

Benjamin: /* GDC Autobuilder */

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

= About Distro Testing =

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

= Test Objectives =

* Verify that Distro executes oe-selftest
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib

= Test Strategy =

The strategy will be divided in two groups:

* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders

== Staging AutoBuilder ==

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

== Public Autobuilder ==

Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team

= Process =
This section describes the process to add a new Distro as supported on Yocto Project

[[File:Distro Steps_v2.PNG]]

== Open a Bug ==

When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candiate]] a bug should be raised on [https://bugzilla.yoctoproject.org/ Bugzilla] and have to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.

*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]

== Install Distro ==

QA responsible should ensure that the new Distro is installed properly on the local AB

=== Process To Install New Distro ===

1. Install the new distro
2. Intsall the requirements from the YP Quickstart 3. Start YP Autobuilder

$ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
$ cd yocto-autobuilder

# Generates password for AB Web
$ ./bin/htpasswd -c -b .htpasswd distrouser distropass

# Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

# start the autobuilder
$ source yocto-autobuilder-setup
$ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010
4.1 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass 6. Enter to http://localhost:8010/builders/nightly-qa-distro and Force a build 7. Wait to the buildsets to end and review the output, load bugs if necessary 8. Stops autobuilder

$ ./yocto-stop-autobuilder both

== Execute Build Sets ==

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

{| class="wikitable"
! style="text-align: center;" | RPM
! style="text-align: center;" | DEB
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}

== Results ==

The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

== Update Supported Distro List ==

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

=== Sanity Tested Distro File (Snapshot)===

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \

=Requirements=
==HW Requirements==

To be updated

==Software Requirements==

To Be Updated

==Environment Requirements==

To Be Updated

= Test execution Cycle =

== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]

== Public Autobuilder ==

* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

= Exit Criteria for Enabling an OS Distribution =

* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]

= Supported OS Distributions =

The distributions used are:

*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian

== Supported OS Distributions Criteria ==

List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros Mega Manual] and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29 Distro File] if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

=== List ===

{| class="wikitable"
! style="text-align: center;" | OS Version
! style="text-align: center;" | Status
|-
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
| Ubuntu-16.10
| align="center" style="color:green;" | PASS
|-
| Fedora-24
| align="center" style="color:green;" | PASS
|-
| Fedora-25
| align="center" style="color:blue;" | NOT YET TESTED
|-
| CentOS-7
| align="center" style="color:green;" | PASS
|-
| Debian-8
| align="center" style="color:green;" | PASS
|-
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}

== Supported OS Distributions Release Calendar ==

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

*[https://wiki.ubuntu.com/Releases Ubuntu ]
*[https://fedoraproject.org/wiki/Releases Fedora]
*[https://www.debian.org/releases/ Debian]
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

QA/JUnit XML Template

2017-01-19T17:44:37Z

Benjamin: Benjamin moved page QA/JUnit XML Template to QA/xUnit XML Template: JUnit is Java specific, xUnit is the overall standard.

#REDIRECT [[QA/xUnit XML Template]]

QA/xUnit XML Template

2017-01-19T17:44:37Z

Benjamin: Benjamin moved page QA/JUnit XML Template to QA/xUnit XML Template: JUnit is Java specific, xUnit is the overall standard.

This JUnit XML template is based on Dirk Jagdmann's [http://llg.cubic.org/docs/junit/ specification]

<pre>
<?xml version="1.0" encoding="UTF-8"?>



<testsuites disabled="" 
errors="" 
failures="" 
name=""
tests="" 
time="" 
>


<testsuite name="" 
tests="" 
disabled="" 
errors="" 
failures="" 
hostname="" 
id="" 
package="" 
skipped="" 
time="" 
timestamp="" 
>


<properties>

<property name="" value=""/>
</properties>


<testcase name="" 
assertions="" 
classname="" 
status=""
time="" 
>




<skipped/>


<error message="" 
type="" 
></error>


<failure message="" 
type="" 
></failure>


<system-out></system-out>


<system-err></system-err>
</testcase>


<system-out></system-out>

<system-err></system-err>
</testsuite>
</testsuites>
</pre>

Bug Triage

2016-11-29T16:38:17Z

Benjamin: /* Roles */

= Goal =
The output of the bug triage meeting should be that all bugs have an owner, a target milestone, and a priority.

Prior to the meeting, each Product Owner should review their Weekly New and High Bugs and be ready with any opens. An example open would be a bug that may need to change Products.

= Roles =
* Call facilitator: Stephen K. Jolley (stephen.k.jolley@intel.com)
** Ensure we keep to the agenda, don't rathole on bug resolution, and stay within our time block.
* Team Leads should pre-triage bugs for their areas of responsibility. Some guidelines on what they should/shouldn't do are:
** a) Pre-triage should happen in the presence of the appropriate team lead
** b) The bugs being pre-triaged should be the responsibility of the team in question
** c) The pre-triage team should be confident when dis-positioning the bugs, taking into account the individual's workload
** d) If the bug raises any doubt or needs information prior, defer it to the main bug triage team

= Agenda =
* Review Queries
** State number, title, and owner
** Owner makes any modifications
** Assign owner, target milestone, and priority.
* Opens
** Collect opens
** Discuss opens

Note: for Minnowboard or MinnowBoard-MAX bugs go to: [[Minnow Bug Triage]]

= Bug Criteria =
* Priority-Severity
** High: Two Week
** Medium+: Milestone
** Medium: Nice to have for milestone, should have for release
** Low: Nice to have, but no urgency

= Bug Status Dashboards =

Weekly - https://wiki.yoctoproject.org/charts/combo.html

Daily - https://wiki.yoctoproject.org/charts/daily/combo.html

= Bug Product Searches =

* [[Bug Product Triage]]

= Yocto Bug Triage Searches =
== Non-Prioritized Bugs ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|priority=Undecided
|product=!opkg
|severity=!janitors
|component=!(meta-cgl)
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Non-Targeted Bugs ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|product=!opkg
|milestone=---
|component=!(meta-cgl)
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Unassigned Bugs ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|to=unassigned@yoctoproject.org
|severity=!janitors
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone,priority,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== New This Week ==
{{#bugzilla:
|created=-1w
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=status,id
|noresultsmessage="No matching bugs found."
|total=summary,estimated
}}

== Medium+ Bugs ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|priority=!(high,medium,low,undecided)
|severity=!enhancement
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone,to
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Medium+ Enhancements ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|priority=!(high,medium,low,undecided)
|severity=enhancement
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone,to
|noresultsmessage="No matching bugs found."
|total=summary
}}

== High Enhancements/Bugs ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|priority=high
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Medium+ Bugs and Enhancements by Owner !(Future, 2.4) ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|priority=!(high,medium,low,undecided)
|product=!opkg
|milestone=!(Future, 2.4)
|group=to
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Medium Bugs and Enhancements by Owner !(Future, 2.4) ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|product=!opkg
|priority=medium
|milestone=!(Future, 2.4)
|group=to
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Major or Critical ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=major,critical
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone
|noresultsmessage="No matching bugs found."
|total=summary
}}

== NEEDINFO ==
{{#bugzilla:
|status=NEEDINFO
|columns=id,summary,to,status,severity,priority,milestone,modified,whiteboard
|hardware=!(MinnowBoard,MinnowBoard MAX)
|product=!opkg
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
|modifiedformat=date
}}

== Re-Opened ==
{{#bugzilla:
|status=REOPENED
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|hardware=!(MinnowBoard,MinnowBoard MAX)
|product=!opkg
|component=!(meta-cgl)
|sort=milestone,priority,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Old Milestone ==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=!opkg
|component=!(meta-cgl)
|milestone= !(1.8.3, 2.0.3, 2.1.3, 2.2.1, 2.3, 2.3 M1, 2.3 M2, 2.3 M3, 2.3 M4, 2.4, Q1, Q2, Q3, Q4, future)
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= !Undecided
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=milestone,priority,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Future Dot Releases==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=!opkg
|milestone= 1.8.3, 2.0.3, 2.1.3, 2.2.1
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= !Undecided
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=milestone
|noresultsmessage="No matching bugs found."
|total=summary
}}

== YP Toaster Bugs needing Triage==
* New Toaster bugs (No Priority)
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=Toaster
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= Undecided
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}
* New Toaster bugs (No milestone)
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority=!low
|product=Toaster
|milestone=---
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}
* NEEDINFO - Toaster
{{#bugzilla:
|status=NEEDINFO
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|hardware=!(MinnowBoard,MinnowBoard MAX)
|product=Toaster
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}
* REOPENDED - Toaster
{{#bugzilla:
|status=REOPENED
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|hardware=!(MinnowBoard,MinnowBoard MAX)
|product=Toaster
|sort=id
|noresultsmessage="No matching bugs found."
|total=summary
}}
* Old Milestone - Toaster
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=Toaster
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= !Undecided
|milestone= !(1.8.3, 2.0.3, 2.1.2, 2.2.1, 2.3, 2.3 M1, 2.3 M2, 2.3 M3, 2.3 M4, Q1, Q2, Q3, Q4, future)
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=priority,milestone,id
|noresultsmessage="No matching bugs found."
|total=summary
}}
* Toaster default assignee
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|hardware=!(MinnowBoard Max, MinnowBoard)
|to=toaster@fake.user
|severity=!janitors
|product=!opkg
|columns=id,summary,severity,priority,milestone,to,status,whiteboard,estimated
|sort=milestone,priority,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== YP BSP Team Triage==
Please refer to the [[BSP Team Bug Triage]] Page

== Future Milestones==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=!opkg
|milestone=Future
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= !(Undecided)
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=priority,id
|noresultsmessage="No matching bugs found."
|total=summary
}}

== Quarterly Milestones==
{{#bugzilla:
|status=!(RESOLVED,VERIFIED,CLOSED)
|product=!opkg
|milestone=Q1,Q2,Q3,Q4
|hardware=!(MinnowBoard Max, MinnowBoard)
|severity=!janitors
|priority= !Undecided
|columns=id,summary,to,status,severity,priority,milestone,whiteboard
|sort=milestone
|noresultsmessage="No matching bugs found."
|total=summary
}}

QA/Automated Test Report Template

2016-11-22T23:45:16Z

Benjamin:

=Introduction=

The test reporting in automation can enable further processes to be automated too. Specifically, results collection should be done by consumers that require no manual interaction to be fed with results data. Just as any other process flow, minimizing the number of input consumers is translated into immediate efficiency as the flowing data is standardized.

=Test and Performance Standard Report=
Performance measurements and software testing share steps in their processes but they have an important difference in the results they both provide: the performance results are expressed in scalar quantities while the testing output is expressed in a predefined set of results.

Under the premise that most of the information in a performance and a testing report is alike, being the results notation the main difference; it should be feasible to use a single report specification that accounts for the aforementioned differences.

=JUnit-XML Performance/Testing Report=
The QA testing has moved towards embracing the [http://llg.cubic.org/docs/junit/ JUnit-XML reporting standard] and there is work in progress to note the required extensions to use that format with performance measurements too.

==Execution Information Template==

There is a data set that will be common to all the test execution done to a subject, such as:

* Test subject information
** OS Version
** Hardware description
* Execution information
** Date

And others that will be defined in the [[QA/Execution_Information_Template | Execution Information Template]]

==Report Template==

See the [[QA/JUnit_XML_Template | test report template here]].

This template is WIP to be extended for the performance measurements.

===QA Sample report===

See a sample report of a [[QA/Sample_Report_oeqa.selftest.bbtests.BitbakeTests|QA test here]].

===Performance Sample Report===

WIP

QA/Automated Test Report Template

2016-11-22T23:20:59Z

Benjamin: Created page with "=Introduction= The test reporting in automation can enable further processes to be automated too. Specifically, results collection should be done by consumers that require no..."

=Introduction=

The test reporting in automation can enable further processes to be automated too. Specifically, results collection should be done by consumers that require no manual interaction to be fed with results data. Just as any other process flow, minimizing the number of input consumers is translated into immediate efficiency as the flowing data is standardized.

=Test and Performance Standard Report=
Performance measurements and software testing share steps in their processes but they have an important difference in the results they both provide: the performance results are expressed in scalar quantities while the testing output is expressed in a predefined set of results.

Under the premise that most of the information in a performance and a testing report is alike, being the results notation the main difference; it should be feasible to use a single report specification that accounts for the aforementioned differences.

==JUnit-XML Report==
The QA testing has moved towards embracing the [http://llg.cubic.org/docs/junit/ JUnit-XML reporting standard] and there is work in progress to note the required extensions to use that format with performance measurements too.

==Test Report Template==

See the [[QA/JUnit_XML_Template | test report template here]].

This template is WIP to be extended for the performance measurements.

==QA Sample report==

See a sample report of a [[QA/Sample_Report_oeqa.selftest.bbtests.BitbakeTests|QA test here]].

==Performance Sample Report==

WIP

QA/Sample Report oeqa.selftest.bbtests.BitbakeTests

2016-11-22T22:05:57Z

Benjamin: /* Test Run Information */

=Test Run Information=
The following test report exemplifies the output of [[OE-Selftest]]'s BitbakeTests with the [https://github.com/xmlrunner/unittest-xml-reporting xml-unittest-reporting] module. The xml-unittest-reporting module is an extension to the python's unittest runner, that is capable of generating [[QA/JUnit XML Template|JUnit-XML]] reports.

This sample report includes failures to complement the format appreciation.

=oeqa.selftest.bbtests.BitbakeTests JUnit-XML Sample Report=
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="2" failures="0" name="oeqa.selftest.bbtests.BitbakeTests-20161103154711" skipped="0" tests="25" time="868.219">
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bbappend_order" time="7.152"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_g" time="5.666"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_invalid_recipe" time="0.732"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_invalid_target" time="1.642"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_checkuri" time="3.107"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_dry_run" time="2.329"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_environment" time="0.970"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_event_handler" time="18.472"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_force_task_2" time="23.536"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_image_manifest" time="161.380"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_invalid_patch" time="9.976"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_invalid_recipe_src_uri" time="9.623"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_just_parse" time="1.590"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_local_sstate" time="15.510"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_non_gplv3" time="402.902"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_postfile" time="0.975"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_prefile" time="1.985"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_rename_downloaded_file" time="14.122"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_run_bitbake_from_dir_1" time="0.987"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_run_bitbake_from_dir_2" time="0.976"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_setscene_only" time="13.341"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_version" time="1.867"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_warnings_errors" time="0.743"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_continue" time="148.339">
<error message="'NoneType' object has no attribute 'group'" type="AttributeError">
<![CDATA[Traceback (most recent call last):
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/selftest/bbtests.py", line 227, in test_continue
continuepos = result.output.find('NOTE: recipe xcursor-transparent-theme-%s: task do_unpack: Started' % manver.group(1))
AttributeError: 'NoneType' object has no attribute 'group'
]]> </error>
</testcase>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_force_task_1" time="20.296">
<error message="[Errno 2] No such file or directory: '/home/besquive/sandbox/poky-working/build/tmp/work/core2-64-poky-linux/zlib/1.2.8-r0/image/usr/share/man/man3/zlib.3'" type="FileNotFoundError">
<![CDATA[Traceback (most recent call last):
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/selftest/bbtests.py", line 91, in test_force_task_1
ftools.append_file(man_file, test_data)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/ftools.py", line 18, in append_file
with open(path, "a") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/besquive/sandbox/poky-working/build/tmp/work/core2-64-poky-linux/zlib/1.2.8-r0/image/usr/share/man/man3/zlib.3'
]]> </error>
</testcase>
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testsuite>
</pre>

QA/Sample Report oeqa.selftest.bbtests.BitbakeTests

2016-11-22T22:03:27Z

Benjamin: Created page with "=Test Run Information= The following test report exemplifies the output of OE-Selftest's BitbakeTests with the [https://github.com/xmlrunner/unittest-xml-reporting xml-un..."

=Test Run Information=
The following test report exemplifies the output of [[OE-Selftest]]'s BitbakeTests with the [https://github.com/xmlrunner/unittest-xml-reporting xml-unittest-reporting] module. The xml-unittest-reporting module is an extension to the python's unittest runner that is capable of generating JUnit-XML reports.

The report includes failures to complement the format appreciation. See the [[JUnit XML Template]] for more information.

=oeqa.selftest.bbtests.BitbakeTests JUnit-XML Sample Report=
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="2" failures="0" name="oeqa.selftest.bbtests.BitbakeTests-20161103154711" skipped="0" tests="25" time="868.219">
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bbappend_order" time="7.152"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_g" time="5.666"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_invalid_recipe" time="0.732"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_bitbake_invalid_target" time="1.642"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_checkuri" time="3.107"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_dry_run" time="2.329"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_environment" time="0.970"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_event_handler" time="18.472"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_force_task_2" time="23.536"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_image_manifest" time="161.380"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_invalid_patch" time="9.976"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_invalid_recipe_src_uri" time="9.623"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_just_parse" time="1.590"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_local_sstate" time="15.510"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_non_gplv3" time="402.902"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_postfile" time="0.975"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_prefile" time="1.985"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_rename_downloaded_file" time="14.122"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_run_bitbake_from_dir_1" time="0.987"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_run_bitbake_from_dir_2" time="0.976"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_setscene_only" time="13.341"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_version" time="1.867"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_warnings_errors" time="0.743"/>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_continue" time="148.339">
<error message="'NoneType' object has no attribute 'group'" type="AttributeError">
<![CDATA[Traceback (most recent call last):
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/selftest/bbtests.py", line 227, in test_continue
continuepos = result.output.find('NOTE: recipe xcursor-transparent-theme-%s: task do_unpack: Started' % manver.group(1))
AttributeError: 'NoneType' object has no attribute 'group'
]]> </error>
</testcase>
<testcase classname="oeqa.selftest.bbtests.BitbakeTests" name="test_force_task_1" time="20.296">
<error message="[Errno 2] No such file or directory: '/home/besquive/sandbox/poky-working/build/tmp/work/core2-64-poky-linux/zlib/1.2.8-r0/image/usr/share/man/man3/zlib.3'" type="FileNotFoundError">
<![CDATA[Traceback (most recent call last):
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/selftest/bbtests.py", line 91, in test_force_task_1
ftools.append_file(man_file, test_data)
File "/home/besquive/sandbox/poky-working/meta/lib/oeqa/utils/ftools.py", line 18, in append_file
with open(path, "a") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/besquive/sandbox/poky-working/build/tmp/work/core2-64-poky-linux/zlib/1.2.8-r0/image/usr/share/man/man3/zlib.3'
]]> </error>
</testcase>
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testsuite>
</pre>

OE-Selftest

2016-11-22T22:00:17Z

Benjamin: Redirected page to Oe-selftest

#REDIRECT [[Oe-selftest]]

OE-selftest

2016-11-22T21:57:59Z

Benjamin: Redirected page to Oe-selftest

#REDIRECT [[Oe-selftest]]

QA/xUnit XML Template

2016-11-22T21:29:43Z

Benjamin: Created page with "This JUnit XML template is based on Dirk Jagdmann's [http://llg.cubic.org/docs/junit/ specification] <pre> <?xml version="1.0" encoding="UTF-8"?> <!-- a description of the JU..."

Test Report Format

2016-11-10T20:13:19Z

Benjamin: Benjamin moved page Test Report Format to Wiki Test Report Format

#REDIRECT [[Wiki Test Report Format]]

Wiki Test Report Format

2016-11-10T20:13:19Z

Benjamin: Benjamin moved page Test Report Format to Wiki Test Report Format

This page is to define Test Report Format for Yocto

==== Test Summary ====
''Here is a test result summary for a build.

==== Test Result Matrix ====
''A general Test Result Summary and a detailed Test Result Summary should be provided here.

'''Example'''
{| border="1" {{table}}
! colspan="5" align="center" style="background:#f0f0f0;"|'''Test Result Summary'''
|-
| align="center" style="background:#f0f0f0;"|'''Component'''
| align="center" style="background:#f0f0f0;"|'''Target'''
| align="center" style="background:#f0f0f0;"|'''Status'''
| align="center" style="background:#f0f0f0;"|'''Comments'''
|-
| '''BSP'''||SugarBay||bgcolor="green" |GOOD||Zypper install/remove has issue; 3D game running failed;
|-
| 　||CrownBay||bgcolor="red" |BLOCK||emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board
|-
| 　||JasperForest||bgcolor="green" |GOOD||The only issue is zypper install/remove bug
|-
| 　||Blacksand||bgcolor="orange" |Buggy||mouse/keyboard do not work in X window; zypper install/remove not work well
|-
| 　||Beagleboard||bgcolor="green" |GOOD||Only one RTC issue exists
|-
| 　||Routerstationpro||bgcolor="green" |GOOD||Only zypper install has issue
|-
| 　||Mpc8315e-rdb||bgcolor="green" |GOOD||Only zypper install has issue
|-
| '''QEMU'''||qemux86||bgcolor="orange" |Buggy||Only zypper install/remove has issue
|-
| 　||qemux86-64||bgcolor="orange" |Buggy||Only zypper install/remove has issue
|-
| 　||qemuarm||bgcolor="orange" |Buggy||zypper could not work well
|-
| 　||qemuppc||bgcolor="orange" |Buggy||zypper could not work well
|-
| 　||qemumips||bgcolor="orange" |Buggy||Only zypper install has issue
|-
| '''ADT'''||　||bgcolor="red" |BLOCK||i686 toolchain not copied to sharing folder; unfs does not work with qemuppc;
|-
| '''Core Build System'''||　||bgcolor="orange" |BUGGY||sstate and non-gplv3 build do not work correctly
|-
| '''Stress'''||　||bgcolor="green" |GOOD||Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress
|-
| '''Compliance'''||　||bgcolor="green" |GOOD||Sugarbay pass LTP and POSIX test
|-
| '''Power/Performance'''||　||bgcolor="green" |GOOD||　
|-
|
|}

'''Note:'''
{| {{table}}
|-
| bgcolor="red" | BLOCK || Critical bugs, more than 50% test cases are blocked
|-
| bgcolor="green" | GOOD ||Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
|-
| bgcolor="orange" | BUGGY ||Normal, Major and Critical bugs, more than 10% test cases failed
|-
|
|}

{| border="1" {{table}}
! colspan="6" align="center" style="background:#f0f0f0;"|'''Detailed Test Result for each component'''
|-
| align="center" style="background:#f0f0f0;"|'''Target'''
| align="center" style="background:#f0f0f0;"|'''Total TCs'''
| align="center" style="background:#f0f0f0;"|'''Not Run'''
| align="center" style="background:#f0f0f0;"|'''Passed'''
| align="center" style="background:#f0f0f0;"|'''Failed'''
| align="center" style="background:#f0f0f0;"|'''Not testable (Blocked)'''
|-
| '''Sugarbay Sato-SDK'''||64||0|| 60 || 1 (bug 1101) || 3 (bug 1112, 883) 
|-
| '''Crownbay Sato-SDK'''||60||0|| 55 || 1 (bug 1099) || 4 
|-
| '''Jasperforest LSB-SDK'''||34||0|| 34 || 0 || 0 
|-
| '''n450 Sato-SDK'''||60||0||52||2 (bug 613, 1099)||6 (bug 613) 
|-
| '''eMenlow Sato-SDK'''||60||0||48||2 (bug 1099, 503)||6 (bug 503) 
|-
| '''Routerstationpro Sato-SDK'''||29||0|| 29 ||0||0 
|-
| '''Mpc8315e-rdb Sato-SDK'''||30||0||26||4 (bug 766)||0
|-
| '''Qemux86-64 Sato'''||28||0||28||0||0
|-
| '''Qemux86-64 Sato-SDK'''||31||0||31||0||0
|-
| '''Qemux86 Sato'''||28||0||28||0||0
|-
| '''Qemux86 Sato-SDK'''||31||0||31||0||0
|-
| '''Qemumips Sato'''||26||0||26||0||0
|-
| '''Qemumips Sato-SDK'''||32||0||32||0||0
|-
| '''Qemuppc Sato'''||20||0||19||0||1 (bug 414)
|-
| '''Qemuppc Sato-SDK'''||26||0||25||0||1 (bug 414) 
|-
| '''Qemuarm Sato'''||26||0||26||0||0
|-
| '''Qemuarm Sato-SDK'''||32||0||32||0||0
|-
| '''Core Build System'''||8||0||7||1 (bug 1133)||0
|-
| '''ADT'''||6||0||5||1 (bug 414)||0
|-
| '''Power/Performance'''||4||0||4||0||0
|-
| '''Stress'''||2||0||2||0||0
|-
| '''Total'''||637||0||600||12||21
|-
|
|}

* You can check the detailed test result in attachment for each target.
* The failed/blocked case number is listed with failed cases’ bug number.

==== Commit Information ====
This section is to provide tree/branch commit information for testing.

'''Example:'''

Tree/Branch: Poky/1.1_M1

Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b

Meta Branch: 1.1_M1

Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298

==== Issue Summary ====
This section shows a list of open bugs for each components.

'''Example'''
{| border="1" {{table}}
! colspan="2" align="center" style="background:#f0f0f0;"|'''Component'''
| align="center" style="background:#f0f0f0;"|'''Bug Number'''
| align="center" style="background:#f0f0f0;"|'''Target Milestone'''
|-
| rowspan="8" | '''System & Core OS'''
| rowspan="2" align="center" | '''Installation & Boot'''
| Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) || 1.1
|-
| Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image || 1.1 M2
|-
| rowspan="2" align="center" | '''Power Mangement'''
| Bug 613: [Blacksand]system cannot enter S3 standby mode||1.1
|-
| Bug 503: [emenlow] system cannot enter S3 standby mode ||1.1
|-
| rowspan="4" align="center" | '''Others'''
| Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb||1.0 M4
|-
| Bug 1101: keyboard pop up when clicking terminal icon||1.0 M2
|-
| Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand||1.1 M2
|-
| Bug 1097: The mouse can not click accurate to the icons in qemumips X||1.1 M2
|-
| colspan="2" rowspan="2" align="center" | '''Graphics'''
| Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing||1.1 M2
|-
| Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so ||1.1
|-
| colspan="2" align="center | '''Multimedia'''||Works well in sugarbay/n450/eMenlow/crownbay||　
|-
| colspan="2" rowspan="2" align="center" | '''Core Build System'''
| New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists || 1.1 M2
|-
| New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build||1.1 M2
|-
| colspan="2" align="center | '''Compliance'''||Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result   POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result   A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build.||　
|-
| colspan="2" align="center | '''Stress'''||Helltest and Crashme could pass 24 hours testing||　
|-
| colspan="2" align="center | '''ADT''' ||Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS||1.1 M3
|-
|
|}

Test Report Tool

2016-11-10T19:48:01Z

Benjamin: Redirected page to QA/Test Report Tool

#REDIRECT [[QA/Test Report Tool]]

QA/Test Reporting Tool

2016-11-10T19:47:36Z

Benjamin: Benjamin moved page QA/Test Reporting Tool to QA/Test Report Tool: Name change to simplify its search

#REDIRECT [[QA/Test Report Tool]]

QA/Test Report Tool

2016-11-10T19:47:36Z

Benjamin: Benjamin moved page QA/Test Reporting Tool to QA/Test Report Tool: Name change to simplify its search

==General Expectations==

(add link share-ability of results to the list of requirements when it fits somewhere)

Here's a list of initial expectations of the Test Reporting Tool.

# The TRT must be able to receive test reports via a remote communication protocol.
## From expected sources of results (Auth)
## To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
##* ability to resume
##* ability to update existing results
##* ability to abort/discard
## Using existing test report protocols like XML
# The TRT should present an interface with views organized towards the following objectives:
## Test Buckets (browsing a component's test results history)
## Test Collections (The type of Milestone/Release or another defined collection)
# The TRT should be able to identify the type of result it is consuming.
## Boolean test results Passed/Failed
## Numeric test results (measurements)

==QA Data Sources==
There are several sources of quality data we'd like to be able to track in a QA dashboard:
* oe-selftest results
* bitbake-selftest results
* build performance metrics
* buildhistory trends
* ???

==Prior Art==

* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?).
* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data].
* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates.
* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output].
* [https://openqa.opensuse.org/ OpenQA] from openSUSE has a visual status overview for the tests run vs an installed OS. Code is on github: https://github.com/os-autoinst/openQA and initial release announcement provides an overview: [https://news.opensuse.org/2011/10/11/opensuse-announces-first-public-release-of-openqa/ openSUSE Announces First Public Release of openQA].

QA

2016-11-02T21:37:51Z

Benjamin: /* Test Execution */

==Current Release QA Trackers==

*Bugs that need to be implemented by QA Team
**[[2.2 qa assigned bugs]]

*Bugs that need to be verified by QA Team
**[[2.2 qa owned_bugs]]

*Features to verify
**[[2.2 qa_owned features to verify]]

*Features to implement
**[[2.2 qa owned features]]

*Old Bugs
**[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[QA Master Test Plan]]
* [[Yocto Project 2.2_Release Test Plan]]
* [[2.2 QA Status]]
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html Performance Charts ]
* [[Toaster testing plan]]
* [[Extensible SDK Test Plan (eSDK)]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.2 Test Cases]]

==Test Execution==
===Autobuilder===

The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:

* A scheduled nightly build and test execution that includes:
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
* A service for on demand testing requests. (Partially working, feature in progress at request [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9080 #9880])

Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.
===Image Testing===
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

'''Testing Images in Virtual Targets''':

The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section.

'''Testing Images in Physical Targets''':

For executing tests in physical targets it would be required to:

# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
# Run the image tests by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.

===Setting up Targets with Devauto===

Manual instructions for setting up the physical test targets appear in many parts of the Yocto Project documentation (i.e. [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware here]). It is easy to setup one target using those instructions but it becomes challenging for the cases where multiple targets have to be prepared or the case where it is required to serialize a changing setup over time, for one: testing on several images using the same target.

[[Devauto]] is the Python library and command line interface intended to manage the device automation assets that act upon the target's physical state. Refer to the [[Devauto]] documentation for more information about the hardware supported and the library and CLI functionality.

===Creating and Adding New Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]], the test manager
*[[QA/Test_Reporting_Tool|The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]
*[[qa-tools|qa-tools Git Repository]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

QA

2016-11-02T20:58:04Z

Benjamin: /* Image Testing */

==Current Release QA Trackers==

*Bugs that need to be implemented by QA Team
**[[2.2 qa assigned bugs]]

*Bugs that need to be verified by QA Team
**[[2.2 qa owned_bugs]]

*Features to verify
**[[2.2 qa_owned features to verify]]

*Features to implement
**[[2.2 qa owned features]]

*Old Bugs
**[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[QA Master Test Plan]]
* [[Yocto Project 2.2_Release Test Plan]]
* [[2.2 QA Status]]
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html Performance Charts ]
* [[Toaster testing plan]]
* [[Extensible SDK Test Plan (eSDK)]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.2 Test Cases]]

==Test Execution==
===Autobuilder===

The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:

* A scheduled nightly build and test execution that includes:
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
* A service for on demand testing requests. (Partially working, feature in progress at request [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9080 #9880])

Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.
===Image Testing===
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

'''Testing Images in Virtual Targets''':

The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section.

'''Testing Images in Physical Targets''':

For executing tests in physical targets it would be required to:

# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
# Run the image tests by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.

===Creating and Adding New Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]], the test manager
*[[QA/Test_Reporting_Tool|The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]
*[[qa-tools|qa-tools Git Repository]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

QA

2016-11-02T20:37:36Z

Benjamin: /* Test Execution */

==Current Release QA Trackers==

*Bugs that need to be implemented by QA Team
**[[2.2 qa assigned bugs]]

*Bugs that need to be verified by QA Team
**[[2.2 qa owned_bugs]]

*Features to verify
**[[2.2 qa_owned features to verify]]

*Features to implement
**[[2.2 qa owned features]]

*Old Bugs
**[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[QA Master Test Plan]]
* [[Yocto Project 2.2_Release Test Plan]]
* [[2.2 QA Status]]
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html Performance Charts ]
* [[Toaster testing plan]]
* [[Extensible SDK Test Plan (eSDK)]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.2 Test Cases]]

==Test Execution==
===Autobuilder===

The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:

* A scheduled nightly build and test execution that includes:
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
* A service for on demand testing requests. (Partially working, feature in progress at request [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9080 #9880])

Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.
===Image Testing===
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target. The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section. And for executing tests in physical targets it would be required to:

# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
# Run the image tests by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.

====Test Target Configuration====

A Device Under Test (DUT)

===Creating and Adding New Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]], the test manager
*[[QA/Test_Reporting_Tool|The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]
*[[qa-tools|qa-tools Git Repository]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

QA

2016-11-02T19:17:04Z

Benjamin: /* Autobuilder */

QA

2016-11-02T19:13:25Z

Benjamin: /* Autobuilder */

QA

2016-11-02T19:11:52Z

Benjamin: /* Autobuilder */

QA

2016-11-02T19:11:32Z

Benjamin: /* Test Automation */

QA

2016-07-19T22:37:39Z

Benjamin: /* Reporting */

QA/Test Report Tool

2016-07-15T17:08:03Z

Benjamin:

QA/Test Report Tool

2016-07-15T16:40:35Z

Benjamin:

==General Expectations==

Here's a list of initial expectations of the Test Reporting Tool.

# The TRT must be able to receive test reports via a remote communication protocol.
## From expected sources of results (Auth)
## To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
##* ability to resume
##* ability to update existing results
##* ability to abort/discard
## Using existing test report protocols like XML
# The TRT should present an interface with views organized towards the following objectives:
## Test Buckets (browsing a component's test results history)
## Test Collections (The type of Milestone/Release or another defined collection)
# The TRT should be able to identify the type of result it is consuming.
## Boolean test results Passed/Failed
## Numeric test results (measurements)

==QA Data Sources==
There are several sources of quality data we'd like to be able to track in a QA dashboard:
* oe-selftest results
* bitbake-selftest results
* build performance metrics
* buildhistory trends
* ???

==Prior Art==

* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?).
* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data].
* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates.
* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output].
* [https://openqa.opensuse.org/ OpenQA] from openSUSE has a visual status overview for the tests run vs an installed OS. Code is on github: https://github.com/os-autoinst/openQA and initial release announcement provides an overview: [https://news.opensuse.org/2011/10/11/opensuse-announces-first-public-release-of-openqa/ openSUSE Announces First Public Release of openQA].

Distro Testing Plan

2016-07-05T20:36:01Z

Benjamin: /* Exit Criteria for Enabling an OS Distribution */

Distro Testing Plan

2016-07-05T20:33:05Z

Benjamin:

Yocto 2.2 Features

2016-07-05T20:15:53Z

Benjamin: /* Yocto Project 2.2 Themes */

== Yocto Project 2.2 Feature Planning ==

Yocto Project 2.2 release planning has been underway since April, 2016. Most all new features and ideas about 2.2 release are tracked as enhancements in Bugzilla. Yocto Project 2.2 is targeted to release in Oct., 2016. Detailed schedule is at this link: [[Yocto 2.2 Schedule]]

== Yocto Project 2.2 Themes ==

* Need to switch to Python 3.x
* Move to do multiple configurations to allow multiple builds which allows for multiple machines and possibly [[distributed builds]].
* Need to better integrate patchwork, swupd, and error reporting system for a better process.
* Need to better integrate and get user adoption of Toaster, eSDK, and CROPS.

== Process for Entering New Feature Requests ==

We have been using the same Bugzilla where we manage bugs to manage all features. Features in Bugzilla should have their severity set to 'enhancement'. If you have a feature request or even an idea about something for 2.2 or future releases, please feel free to file an 'enhancement' feature in Bugzilla:

* Set the severity of bug to be 'enhancement'. Please put as much detail as possible in the 'description' field and give it a concise and meaningful 'summary'. You can always add/edit as you have more thoughts and information about the feature. Others can also use the feature entry to ask questions or provide feedback.
* If you know which functional category the feature belongs to, please put the bug # in appropriate category such as usability, performance, kernel, core build system, etc.
* Set the priority you would like this to be or indicate this in the description field. We may adjust the priority when we have a better view of the whole release later on.
* Preview your Entry to make sure it looks ok and then save it

''Note:''

'''Priority:''' 
High = Must have for the target release 
Medium+ = Very important, needs special attention compared with other medium features 
Medium = Important 
Low = Nice to have, no defined plan 

==Unsorted Features (by priority)==

=== High ===
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|priority=high
|severity=enhancement
|noresultsmessage="None"
|milestone=2.2, 2.2 M1, 2.2 M2, 2.2 M3, 2.2 M4
|sort=to
|total=estimated
|resolution=!DUPLICATE
}}

=== Medium+ ===
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|priority=Medium%2B
|severity=enhancement
|noresultsmessage="None"
|milestone=2.2, 2.2 M1, 2.2 M2, 2.2 M3, 2.2 M4
|sort=to
|total=estimated
|resolution=!DUPLICATE
}}

=== Medium ===
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|priority=medium
|severity=enhancement
|noresultsmessage="None"
|milestone=2.2, 2.2 M1, 2.2 M2, 2.2 M3, 2.2 M4
|sort=to
|total=estimated
|resolution=!DUPLICATE
}}

=== Low ===
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|priority=low
|severity=enhancement
|noresultsmessage="None"
|milestone=2.2, 2.2 M1, 2.2 M2, 2.2 M3, 2.2 M4
|sort=to
|total=estimated
|resolution=!DUPLICATE
}}

=== Undecided ===
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|priority=undecided
|severity=enhancement
|noresultsmessage="None"
|milestone=2.2, 2.2 M1, 2.2 M2, 2.2 M3, 2.2 M4
|sort=to
|total=estimated
}}

== 2.3, Future ==
{{#bugzilla:
|columns=id,to,summary,estimated,milestone,from,status,severity,priority,whiteboard
|severity=enhancement
|noresultsmessage="None"
|milestone=2.3,Future
|sort=to
|total=estimated
|status=!(resolved, verified, closed)
}}

QA/Test Report Tool

2016-07-01T17:55:57Z

Benjamin: Created page with "==General Expectations== Here's a list of initial expectations of the Test Reporting Tool. # The TRT must be able to receive test reports via a remote communication protocol..."

Qa-tools

2016-06-17T01:26:06Z

Benjamin:

{{DISPLAYTITLE:qa-tools}}

The qa-tools git repository is the recipient for the QA automation scripts and tools. You can take a look at it via the [http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/ gitweb interface].

For cloning the qa-tools repository:

<pre>$ git clone git://git.yoctoproject.org/qa-tools</pre>

Contributing to any of the yoctoproject.org repositories, including qa-tools, can be achieved with the [[Poky Contributions]] guidelines.

Poky Contributions

2016-06-17T01:23:48Z

Benjamin: /* Git workflow */

We need a system for accepting changes to Poky and other Yocto Linux components. We already have a system established in the community for this and this will work for our distribution work too.

Poky has no built in submission process or policy. I see this as an advantage as we can be flexible and can adapt the process and policies to fit our needs. Our users are also free to adapt it to their situations which are likely to have different needs too. It does mean we need to clearly document how we work though.

Our submission process is for people to push to branches on the poky-contrib tree and then send a merge request in the same way the kernel submission process works (with the difference of a shared git tree). You will have seen people on the Poky mailing list do this for community submissions.

== Patch Submission Process ==

# Keep all of your own changes in the poky-contrib tree before requesting for acceptance
# When you're ready for submission, compose a merge message by running this script:
#* scripts/create_pull_request.
# Compose a merge mail based on above body information, plus:
#* add a [PULL] prefix in the mail subject
# Send the merge mail to the poky mailing list as possible, with notes:
#* No confidential information there.
# Once seeing the merge mail, either Richard or Saul will check candidate patches in target poky-contrib branch. If there are some improvements or further discussion required, Richard/Saul will reply with explanation in mail. Quotes should be provided instead of simply providing comments.
#* If all patches in the branch are in good form, jump to step 7
# Revise your branch based on comments and jump back to step 2 for another merge request
# Richard/Saul pulls target branch into Poky upstream

== Access to git.yoctoproject.org Repositories==

There are a number of git repositories at the [http://git.yoctoproject.org/cgit/cgit.cgi/ Yocto Project git server]. If you need to contribute to any of these repos you can follow the instructions below. Consider that poky-contrib is the most solicited repo so it is also the example that appears in the guide.

=== Request Access ===

To access a repo from git.yoctoproject.org, including poky-contrib, please send an email to [mailto:mhalstead@linuxfoundation.org Michael Halstead] with an ssh key from each person needing access along with the list of repositories you're trying to contribute to.

== Poky Contrib Repository ==

Once you have access, you will be able to clone and push to:

<pre>git@git.yoctoproject.org:poky-contrib.git</pre>

Or any of the other repositories at git.yoctoproject.org.

You might want to use git remote to pull from the master poky tree so that you pull in updates from there.

The tree is viewable at:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/

Where existing user branches can be seen.

== Git workflow ==

[http://git.yoctoproject.org/cgit/cgit.cgi/poky/ Poky] is a read-only repository that only the maintainers are able to write to. For contributions, there exists [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/ poky-contrib]. The first part of the git workflow explanation intends to let you know how to use these two repositories for:

# Base your code changes from the poky repository
# Publish your contributions to the poky-contrib repository

You can skip to the [[Poky_Contributions#Clone_a_Repository_from_git.yoctoproject.org | Clone a Repository from git.yoctoproject.org]] section if you are not working in the poky-contrib repository, not all the repositories require this double relationship for contributions.

=== Git Clone Poky===

Clone the main git tree (if you haven't already):

<pre>$ git clone git://git.yoctoproject.org/poky</pre>

=== Add poky-contrib as a Git Remote ===

You have the option to add the poky-contrib remote as a read-only (perhaps for the case where you haven't sent the ssh access keys) or as read-write if you have cleared your write access as indicated in [[Poky_Contributions#Request_Access]]. Execute only one of the two following instructions:

'''Read-only:'''

<pre>$ git remote add poky-contrib git://git.yoctoproject.org/poky-contrib</pre>

'''Read-write:'''

<pre>$ git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib</pre>

=== Clone a (non-Poky) Repository from git.yoctoproject.org ===

The process is simple for contributions that are not based off of the Poky repository. For any other repository at [http://git.yoctoproject.org/cgit/cgit.cgi/ git.yoctoproject.org] that is not Poky, you can just clone it either read-only or read-write as follows:

'''Read-only:'''

<pre>$ git clone git://git.yoctoproject.org/<repository-name></pre>

'''Read-write:'''

<pre>$ git clone ssh://git@git.yoctoproject.org/<repository-name></pre>

=== Developer Git Branches ===

Create a branch, originating at the tip, that you will push:

<pre>git checkout -b name/topic-of-branch</pre>

The convention is that the branch name contains your name followed by a forward slash and then the topic of the branch e.g. "michaelw/fixes-thing".

Add changes to this branch ensuring commit messages follow the standard format defined below.

==== Pushing Developer Git Branches ====

When your changes are ready, you can push them:

'''Pushing to poky-contrib'''
<pre>git push -u poky-contrib branch-name:remote-branch-name</pre>

'''Pushing to any other repository:'''

<pre>git push branch-name:remote-branch-name</pre>

When this is done, you'll be able to see your branch at the [http://git.yoctoproject.org/cgit/cgit.cgi/ gitweb interface], just navigate to the repository you pushed it to.

Once it's no longer useful please remember to delete your branch from the remote.

== Git commit messages ==

Commit messages should follow the standard format of having a single-line subject denoting the affected area of the code and summarising the change followed by a blank line then a more detailed description of the commit (not always necessary but more usually is) followed by the sign off line (added by passing -s to the commit command).

Some example commit messages follow:

<pre>
icu-native: LD_LIBRARY_PATH is required

Back to commit ea45876d7ba3d4d2b132fd38a2c40834a2385f34, LD_LIBRARY_PATH
is disable for cross-build, however it's required for native version. So
force noldlibpath.patch for non-native case only

Signed-off-by Kevin Tian <kevin.tian@intel.com>
</pre>
----
<pre>
gypsy: Fix broken SRC_URI

Signed-off-by: Scott Garman <scott.a.garman@intel.com>
</pre>

''Note:'' the commit doesn't include a detailed message - this is suitable because the change is trivial and the subject line contains all of the required information.

Poky Contributions

2016-06-17T00:47:28Z

Benjamin:

We need a system for accepting changes to Poky and other Yocto Linux components. We already have a system established in the community for this and this will work for our distribution work too.

Poky has no built in submission process or policy. I see this as an advantage as we can be flexible and can adapt the process and policies to fit our needs. Our users are also free to adapt it to their situations which are likely to have different needs too. It does mean we need to clearly document how we work though.

Our submission process is for people to push to branches on the poky-contrib tree and then send a merge request in the same way the kernel submission process works (with the difference of a shared git tree). You will have seen people on the Poky mailing list do this for community submissions.

== Patch Submission Process ==

# Keep all of your own changes in the poky-contrib tree before requesting for acceptance
# When you're ready for submission, compose a merge message by running this script:
#* scripts/create_pull_request.
# Compose a merge mail based on above body information, plus:
#* add a [PULL] prefix in the mail subject
# Send the merge mail to the poky mailing list as possible, with notes:
#* No confidential information there.
# Once seeing the merge mail, either Richard or Saul will check candidate patches in target poky-contrib branch. If there are some improvements or further discussion required, Richard/Saul will reply with explanation in mail. Quotes should be provided instead of simply providing comments.
#* If all patches in the branch are in good form, jump to step 7
# Revise your branch based on comments and jump back to step 2 for another merge request
# Richard/Saul pulls target branch into Poky upstream

== Access to git.yoctoproject.org Repositories==

There are a number of git repositories at the [http://git.yoctoproject.org/cgit/cgit.cgi/ Yocto Project git server]. If you need to contribute to any of these repos you can follow the instructions below. Consider that poky-contrib is the most solicited repo so it is also the example that appears in the guide.

=== Request Access ===

To access a repo from git.yoctoproject.org, including poky-contrib, please send an email to [mailto:mhalstead@linuxfoundation.org Michael Halstead] with an ssh key from each person needing access along with the list of repositories you're trying to contribute to.

== Poky Contrib Repository ==

Once you have access, you will be able to clone and push to:

<pre>git@git.yoctoproject.org:poky-contrib.git</pre>

Or any of the other repositories at git.yoctoproject.org.

You might want to use git remote to pull from the master poky tree so that you pull in updates from there.

The tree is viewable at:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/

Where existing user branches can be seen.

== Git workflow ==

Clone the main git tree (if you haven't already):

git clone git://git.yoctoproject.org/poky

Add the poky-contrib as a read only remote repository:

git remote add poky-contrib git://git.yoctoproject.org/poky-contrib

- or since you probably want to be able to push your branch back to poky-contrib, check it out as a read/write repository:

git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib

Create a branch, originating at the tip, that you will push to poky-contrib:

git checkout -b name/topic-of-branch

The convention is that the branch name contains your name followed by a forward slash and then the topic of the branch e.g. "michaelw/fixes-thing".

Add changes to this branch ensuring commit messages follow the standard format defined below. When they are ready, you can push them:

git push -u poky-contrib branch-name:remote-branch-name

If you want to replace an existing branch on poky-contrib with a new one, i.e. you fixed something or the old one was merged and you want to reuse the branch for new patches, add -f to the git push command.

When this is done, you'll be able to see your branch at http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/

Once it's merged please remember to delete your branch from poky-contrib.

== Git commit messages ==

Commit messages should follow the standard format of having a single-line subject denoting the affected area of the code and summarising the change followed by a blank line then a more detailed description of the commit (not always necessary but more usually is) followed by the sign off line (added by passing -s to the commit command).

Some example commit messages follow:

<pre>
icu-native: LD_LIBRARY_PATH is required

Back to commit ea45876d7ba3d4d2b132fd38a2c40834a2385f34, LD_LIBRARY_PATH
is disable for cross-build, however it's required for native version. So
force noldlibpath.patch for non-native case only

Signed-off-by Kevin Tian <kevin.tian@intel.com>
</pre>
----
<pre>
gypsy: Fix broken SRC_URI

Signed-off-by: Scott Garman <scott.a.garman@intel.com>
</pre>

''Note:'' the commit doesn't include a detailed message - this is suitable because the change is trivial and the subject line contains all of the required information.

Qa-tools

2016-06-17T00:32:46Z

Benjamin: Created page with "{{DISPLAYTITLE:qa-tools}} The qa-tools git repository is the recipient for the QA automation scripts and tools. You can take a look at it via the [http://git.yoctoproject.org..."

QA

2016-06-17T00:00:19Z

Benjamin: /* QA Resources */

==Current Release QA Trackers==
*[[2.1 QA Owned Features]]
*[[2.1 qa owned bugs]]
*[[2.1 qa owned features to verify]]
*[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[Yocto 2.1 Overall Test Plan]]
* [[Toaster testing plan]]
* [[SDK Testing Plan]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]
* [[QA_Master_Test_Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.1 Test Cases]]

==Test Automation==
===Autobuilder===

The [[AutoBuilder]] is the automated build and test framework used by Yocto Project QA.

===Adding Automated Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]
*[[qa-tools|qa-tools Git Repository]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

Main Page

2016-06-16T23:51:05Z

Benjamin:

<h1>Welcome to the Yocto Project Wiki!</h1>

== Project planning ==

=== 2.2 (Current) Project Planning ===

* [[Planning]]

=== 2.2 Project Status and Schedule ===
* [[Yocto Project v2.2 Status]]
* [[Yocto 2.2 Schedule]]
* [[2.2 QA Status]]
;* [[meta-intel-2.0-dizzy-1.7 release planning]]

== Release Engineering ==

* [[Yocto Release Engineering | Yocto Project Release Engineering]]

== QA & Automation ==

* [[QA| Yocto Project QA Main Page]]

== Wiki reference sitemap ==

=== Main Wiki Sections ===

* [[Glossary]]
* [[Planning and Governance]]
* [[Community Guidelines]]
* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[License Infrastructure Interest Group | License Infrastructure]]
* [[Processes and Activities]]
** [[YoctoCalendar]]
* [[Projects]]
* [[Yocto Interest Groups]]
* [[FAQ]]
* [[How do I]]
* [[Contributors]]
* [[Training]]
* [[Testopia]] - The Yocto Project's community-opened test case management platform

* [[Toaster]] - the web interface | [[Web UI]] - The design plans for the new Web UI

== Other resources ==
* [http://yoctoproject.org Yocto Project Front Page]
* [http://git.yoctoproject.org/ Yocto Project Git Source Repos]
* [https://bugzilla.yoctoproject.org/ Yocto Project Bugzilla]
* [https://www.yoctoproject.org/tools-resources/community/mailing-lists Yocto Project Mailing Lists]
* [https://autobuilder.yoctoproject.org/main/ Yocto Project Autobuilder Waterfall Page]
* [https://autobuilder.yoctoproject.org/ Yocto Project Autobuilder Splash/Info Page]
* [http://downloads.yoctoproject.org/releases/yocto/ Yocto Project Releases]
* [http://autobuilder.yoctoproject.org/pub/nightly/ Yocto Project Nightly Build Images]
* [http://downloads.yoctoproject.org/mirror/sources/ Upstream Sources Mirror]
* [http://www.openembedded.org/wiki/Main_Page OpenEmbedded Wiki]
* [http://cgit.openembedded.org/ OpenEmbedded Git Repos]
* [http://buildbot.net/ BuildBot]

Main Page

2016-06-16T23:48:55Z

Benjamin:

<h1>Welcome to the Yocto Project Wiki!</h1>

== Project planning ==

=== 2.2 (Current) Project Planning ===

* [[Planning]]

=== 2.2 Project Status and Schedule ===
* [[Yocto Project v2.2 Status]]
* [[Yocto 2.2 Schedule]]
* [[2.2 QA Status]]
;* [[meta-intel-2.0-dizzy-1.7 release planning]]

== Release Engineering ==

* [[Yocto Release Engineering | Yocto Project Release Engineering]]

== Wiki reference sitemap ==

=== Main Wiki Sections ===

* [[Glossary]]
* [[Planning and Governance]]
* [[Community Guidelines]]
* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[License Infrastructure Interest Group | License Infrastructure]]
* [[Processes and Activities]]
** [[YoctoCalendar]]
* [[Projects]]
* [[Yocto Interest Groups]]
* [[FAQ]]
* [[How do I]]
* [[Contributors]]
* [[Training]]
* [[Testopia]] - The Yocto Project's community-opened test case management platform

* [[Toaster]] - the web interface | [[Web UI]] - The design plans for the new Web UI

== Other resources ==
* [http://yoctoproject.org Yocto Project Front Page]
* [http://git.yoctoproject.org/ Yocto Project Git Source Repos]
* [https://bugzilla.yoctoproject.org/ Yocto Project Bugzilla]
* [https://www.yoctoproject.org/tools-resources/community/mailing-lists Yocto Project Mailing Lists]
* [https://autobuilder.yoctoproject.org/main/ Yocto Project Autobuilder Waterfall Page]
* [https://autobuilder.yoctoproject.org/ Yocto Project Autobuilder Splash/Info Page]
* [http://downloads.yoctoproject.org/releases/yocto/ Yocto Project Releases]
* [http://autobuilder.yoctoproject.org/pub/nightly/ Yocto Project Nightly Build Images]
* [http://downloads.yoctoproject.org/mirror/sources/ Upstream Sources Mirror]
* [http://www.openembedded.org/wiki/Main_Page OpenEmbedded Wiki]
* [http://cgit.openembedded.org/ OpenEmbedded Git Repos]
* [http://buildbot.net/ BuildBot]

AutoBuilder/Adding Automated Tests

2016-06-15T19:55:02Z

Benjamin:

==Preparation==

Adding automated tests to the AutoBuilder assumes that the testing is done in either of the following formats:

*[[Oe-selftest|OpenEmbedded Selftest]] format
*[[Image tests|Bitbake Runtime tests]] format

The AutoBuilder interfaces with the test tool via a calling command. Such test calling command is used to start executing a given set of tests. The test set description can come in the way of a configuration (file, variable, argument, etc) and it ultimately explicitly lists which tests are to be run. It is an expectation of the test tool that it procures the execution of the list of tests that has been specified at the moment of calling the starting command.

===Sample Test for Instructions===

For the purpose of this guide, the following test calling command will be used:

<pre>$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors</pre>

The test tool used is in the OpenEmbedded Selftest format and the calling line selects a single test from the BBTests suite. The reason for this choice is merely simplicity for writing this guide.

You can see the code of this particular test here (referencing the Krogoth release):

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/bbtests.py?id=krogoth-15.0.0#n58

There should be about 5 lines of code there, the test can be simply explained as: execute a bitbake build from a file that doesn't exist and verify that it shows an error. It's a negative test.

====Sample Test Output====

Getting familiar with the output of the test is a good idea. Preparing the environment for running such test can be seen at the [[Oe-selftest#How_to_use]] section.

Once the test is run at the command line, the following terminal output is presented:

<pre>
$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:30,964 - selftest - INFO - Running bitbake -e to get BBPATH
2016-06-15 13:55:32,027 - selftest - INFO - Checking that everything is in order before running the tests
2016-06-15 13:55:33,073 - selftest - INFO - Running bitbake -p
2016-06-15 13:55:36,228 - selftest - INFO - Loading tests from: oeqa.selftest.bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-06-15 13:55:36 - test_warnings_errors (oeqa.selftest.bbtests.BitbakeTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.782s

OK
2016-06-15 13:55:37,012 - selftest - INFO - Finished
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from local.conf
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from bblayers.conf
</pre>

=== an AutoBuilder to Run the Test ===

An AutoBuilder will be needed to run the testing, for letting the AutoBuilder know what you want to test, a config-set is required. Once you create the config-set, an AutoBuilder is needed to test and run it when finished.

==== Basic AutoBuilder Knowledge====

====Setup an AutoBuilder Instance for Testing====

==Test Enabling Steps==

===Write the Build Set===

====Include the Test in a Build Step====

===Test the Build Set===

AutoBuilder/Adding Automated Tests

2016-06-15T19:25:08Z

Benjamin: /* Preparation */

AutoBuilder/Adding Automated Tests

2016-06-15T18:59:16Z

Benjamin: /* Preparation */

AutoBuilder/Adding Automated Tests

2016-06-15T18:58:35Z

Benjamin: /* Preparation */

==Preparation==

Adding automated tests to the AutoBuilder assumes that the testing is done in either of the following formats:

*[[Oe-selftest|OpenEmbedded Selftest]] format
*[[Image tests|Bitbake Runtime tests]] format

The AutoBuilder interfaces with the test tool via a calling command. Such test calling command is used to start executing a given set of tests. The test set description can come in the way of a configuration (file, variable, argument, etc) and it ultimately explicitly lists which tests are to be run. It is an expectation of the test tool that it procures the execution of the list of tests that has been specified at the moment of calling the starting command.

===Sample Test for Instructions===

For the purpose of this guide, the following test calling command will be used:

<pre>$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors</pre>

The test tool used is in the OpenEmbedded Selftest format and the calling line selects a single test from the BBTests suite. The reason for this choice is merely simplicity for writing this guide.

You can see the code of this particular test here (referencing the Krogoth release):

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/bbtests.py?id=krogoth-15.0.0#n58

There should be about 5 lines of code there, the test can be simply explained as: execute a bitbake build from a file that doesn't exist and verify that it shows an error. It's a negative test.

===Sample Test Output===
Getting familiar with the output of the test is a good idea. Preparing the environment for running such test can be seen at the [[Oe-selftest#How_to_use]] section.

Once the test is run at the command line, the following terminal output is presented:

<pre>
$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:30,964 - selftest - INFO - Running bitbake -e to get BBPATH
2016-06-15 13:55:32,027 - selftest - INFO - Checking that everything is in order before running the tests
2016-06-15 13:55:33,073 - selftest - INFO - Running bitbake -p
2016-06-15 13:55:36,228 - selftest - INFO - Loading tests from: oeqa.selftest.bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-06-15 13:55:36 - test_warnings_errors (oeqa.selftest.bbtests.BitbakeTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.782s

OK
2016-06-15 13:55:37,012 - selftest - INFO - Finished
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from local.conf
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from bblayers.conf
</pre>

AutoBuilder/Adding Automated Tests

2016-06-15T18:54:29Z

Benjamin: /* AutoBuilder Adding Automated Tests Guide */

QA

2016-06-15T18:17:06Z

Benjamin: /* Autobuilder */

==Current Release QA Trackers==
*[[2.1 QA Owned Features]]
*[[2.1 qa owned bugs]]
*[[2.1 qa owned features to verify]]
*[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[Yocto 2.1 Overall Test Plan]]
* [[Toaster testing plan]]
* [[SDK Testing Plan]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]
* [[QA_Master_Test_Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.1 Test Cases]]

==Test Automation==
===Autobuilder===

The [[AutoBuilder]] is the automated build and test framework used by Yocto Project QA.

===Adding Automated Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

QA

2016-06-15T18:16:47Z

Benjamin: /* Adding Automated Tests */

==Current Release QA Trackers==
*[[2.1 QA Owned Features]]
*[[2.1 qa owned bugs]]
*[[2.1 qa owned features to verify]]
*[[Old resolved bugs and features]]

==Current Release QA Test Plans==
* [[Yocto 2.1 Overall Test Plan]]
* [[Toaster testing plan]]
* [[SDK Testing Plan]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]
* [[QA_Master_Test_Plan]]

==Current Release QA Test Cases==
* [[Yocto 2.1 Test Cases]]

==Test Automation==
===Autobuilder===

The [[AutoBuilder]] is the automatic build and test framework used by Yocto Project QA.

===Adding Automated Tests===

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.

A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].

==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]]
*[http://errors.yoctoproject.org/Errors/ error report web]

==Performance testing==
[[Performance Test]]

==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]

= Archive =

You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].

= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.

==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

[[Category:QA]]

AutoBuilder

2016-06-15T18:16:04Z

Benjamin:

Here are some useful links for AutoBuilder

* Nightly build output: http://autobuilder.pokylinux.org/pub/nightly/

==AutoBuilder Automated Testing==

The AutoBuilder is able to execute automated testing too when the proper configuration is added. The AutoBuilder basically relies on what the Yocto Project exposes as automated test tools and it doesn't include support for directly executing tests. The AutoBuilder has been used for test execution(runtime, selftests) in the Yocto Project long before it was chosen as the automated test framework over LAVA. Instructions on how to add tests to the AutoBuilder [[AutoBuilder/Adding Automated Tests|can be seen here]].