Distro Testing Plan: Difference between revisions
From Yocto Project
Jump to navigationJump to search
No edit summary |
|||
| Line 1: | Line 1: | ||
This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder | |||
This article is the test plan for | |||
= About Distro Testing = | = About Distro Testing = | ||
Distro Testing is intended to catch bugs that are distribution specific | Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution. | ||
= Objectives = | = Test Objectives = | ||
* Verify that Distro executes oe-selftest | * Verify that Distro executes oe-selftest | ||
* Verify that | * Verify that Distro is able to execute components (Eclipse, Toaster, WIC) | ||
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM) | * Veirfy that Distro is able to build per package type (IPK, DEB, RPM) | ||
* Verify that Distro is able to build different image types (LSB, non LSB) | * Verify that Distro is able to build different image types (LSB, non LSB) | ||
| Line 19: | Line 18: | ||
= Test Strategy = | = Test Strategy = | ||
The strategy will be divided in two groups | The strategy will be divided in two groups: | ||
* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing) | |||
* [[SWAT]] monitoring at the Autobuilders | |||
== GDC Autobuilder == | == GDC Autobuilder == | ||
When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle. | |||
=== Steps === | === Steps === | ||
| Line 109: | Line 110: | ||
* The QA team is with the responsibility to fill the the appropriate bugs if there is any failure on the Distro. | * The QA team is with the responsibility to fill the the appropriate bugs if there is any failure on the Distro. | ||
== Update Supported Distro == | == Update Supported Distro List == | ||
Once all the build sets were executed and all were PASS at least once, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list. | |||
=== Sanity Tested Distro File (Snapshot)=== | |||
meta-poky/conf/distro/poky.conf | meta-poky/conf/distro/poky.conf | ||
| Line 145: | Line 148: | ||
= Test execution Cycle = | = Test execution Cycle = | ||
== Public Autobuilder == | == Public Autobuilder == | ||
| Line 154: | Line 153: | ||
* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team | * There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team | ||
= | = Exit Criteria for Enabling an OS Distribution = | ||
* All the defined build steps are PASS [[#Execute Build Sets]] | * All the defined build steps are PASS [[#Execute Build Sets]] | ||
* Distro was added to the supported list [[#Update Supported Distro]] | * Distro was added to the supported list [[#Update Supported Distro]] | ||
Revision as of 20:33, 5 July 2016
This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder
About Distro Testing
Distro Testing is part of the process of enabling an OS distribution at the Yocto Project Autobuilder. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.
Test Objectives
- Verify that Distro executes oe-selftest
- Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
- Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
- Verify that Distro is able to build different image types (LSB, non LSB)
- Verify that Distro is able to build per architecture (arm, x86)
- Verify that Distro is able to build per bootloader (systed, init)
- Verify that Distro is able to build poky-tiny
- Verify that Distro executes multilib
Test Strategy
The strategy will be divided in two groups:
- Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
- SWAT monitoring at the Autobuilders
GDC Autobuilder
When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by QA, this is a one-time testing activity, not executed in every milestone o release test cycle.
Steps
Public Autobuilder
Once the distro was first time validated by QA team it will be running on public autobuilder different build set already defined [[]], after that it going to be under the focus of SWAT team
Process
This section describes the process to add a new Distro as supported on Yocto Project
Install Distro
To Be Updated
Execute Build Sets
Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS
| RPM | DEB | IPK | Component | Multilib | World | oe-selftest |
|---|---|---|---|---|---|---|
| ARM | x86_64 | x86_32 | Eclipse | x86 64/32 | x86_64 | |
| init | systemd | systemd | Toaster | x86 64/x32 | systemd | |
| poky_lsb | poky | poky_tiny | buildtools | core-image-sato | poky | |
| core-image-lsb | core-image-sato-sdk | core-image-minimal | read_only rootfs | poky | core-image-sato-sdk | |
| pam | init/systemd | |||||
| Log rotate | RPM,DEB,IPK |
Results
- The QA team is with the responsibility to fill the the appropriate bugs if there is any failure on the Distro.
Update Supported Distro List
Once all the build sets were executed and all were PASS at least once, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.
Sanity Tested Distro File (Snapshot)
meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
poky-1.7 \n \
poky-1.8 \n \
poky-2.0 \n \
Ubuntu-14.04 \n \
Ubuntu-14.10 \n \
Ubuntu-15.04 \n \
Ubuntu-15.10 \n \
Ubuntu-16.04 \n \
Fedora-21 \n \
Fedora-22 \n \
Fedora-23 \n \
CentOS-6.* \n \
CentOS-7.* \n \
Debian-7.* \n \
Debian-8.* \n \
openSUSE-project-13.2 \n \
Requirements
HW Requirements
Software Requirements
Environment Requirements
Test execution Cycle
Public Autobuilder
- There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team
Exit Criteria for Enabling an OS Distribution
- All the defined build steps are PASS #Execute Build Sets
- Distro was added to the supported list #Update Supported Distro
