|
|
| Line 134: |
Line 134: |
| = Release Criteria/ Exit Criteria = | | = Release Criteria/ Exit Criteria = |
|
| |
|
| | | * All the defined build steps are PASS [#] |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| <!--
| |
| THIS IS THE OLD TESTING PLAN
| |
| | |
| = Test Areas =
| |
| Toaster consists of two big components, as follows:
| |
| | |
| == Backend ==
| |
| | |
| ==== Functionality tests ====
| |
| * [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/Toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here]; | |
| * Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_Toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
| |
| * Verify that all links in the simple UI are available;
| |
| * Verify the quality of the data collected through the simple UI;
| |
| * Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
| |
| | |
| ==== Usability tests ====
| |
| * Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the Toaster server])
| |
| | |
| == Frontend ==
| |
| | |
| ==== Functionality tests ====
| |
| * Manual testing in the first stage;
| |
| * Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;
| |
| | |
| ==== Compatibility tests ====
| |
| * Verify the behavior of the GUI on different browsers and operating systems; TBD
| |
| | |
| ==== Usability tests ====
| |
| * Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
| |
| * Friendly graphical user interface;
| |
| | |
| ==== Performance tests ====
| |
| * Stress testing (e.g. display appropriate error messages when the system is under stress);
| |
| | |
| = Test Cycle =
| |
| {|class="wikitable" style="text-align: center;"
| |
| | || || colspan="3" | Test execution cycle
| |
| |-
| |
| | || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]
| |
| |-
| |
| | rowspan="2" | Build type || Weekly || Yes || Yes ||
| |
| |-
| |
| | Release || Yes || Yes || Yes
| |
| |-
| |
| | rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes
| |
| |-
| |
| | [[#Frontend]] || Yes || Yes || Yes
| |
| |-
| |
| | rowspan="5" | Target machine || qemuarm || || Yes || Yes
| |
| |-
| |
| | qemumips|| || || Yes
| |
| |-
| |
| | qemuppc|| || || Yes
| |
| |-
| |
| | qemux86|| Yes || Yes || Yes
| |
| |-
| |
| | qemux86-64 || || || Yes
| |
| |-
| |
| | rowspan="3" | Target image|| core-image-minimal|| Yes || ||
| |
| |-
| |
| | core-image-sato-sdk|| || Yes || Yes
| |
| |}
| |
| | |
| == Weekly Test ==
| |
| *; Scope:
| |
| ** Images built weekly and released through the distribution team.
| |
| | |
| *; Objective:
| |
| ** Functionality test on most areas with minimum sets of tests;
| |
| ** Regression test with high probability to find bugs.
| |
| | |
| == Full Pass Test ==
| |
| *; Scope:
| |
| ** Images built as candidates for milestone or final release;
| |
| ** Passed [[#Weekly Test]]
| |
| | |
| *; Objective:
| |
| ** Ensure functionality of Toaster component.
| |
| | |
| == Release Test ==
| |
| *; Scope:
| |
| ** Release candidates that pass [[#Full Pass Test]]
| |
| | |
| *; Objective:
| |
| ** All scheduled features are covered, or rescheduled;
| |
| ** All relevant bugs are fixed and verified.
| |
| -->
| |
This article is the test plan for Distro testing
About Distro Testing
Distro Testing is intended to catch bugs that are distribution specific using the yocto-autobuilder.
Objectives
- Verify that Distro executes oe-selftest
- Verify that Dsitro is able to execute componenets (Eclipse, Toaster, WIC)
- Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
- Verify that Distro is able to build different image types (LSB, non LSB)
- Verify that Distro is able to build per architecture (arm, x86)
- Verify that Distro is able to build per bootloader (systed, init)
- Verify that Distro is able to build poky-tiny
- Verify that Distro executes multilib
Test Strategy
The strategy will be divided in two groups one executed once in GDC autobuilders when the Distro is released (one time testing) and another tested continuously on public autobuilders
GDC Autobuilder
The first time that Distro is released it will be installed on GDC autobuilder and a series of build steps going to be applied to that Distro,
once very build set is PASS that distro will be added to the list of supported Distros by QA, this is a only one time testing activity, not executed in every milestone o release test cycle.
Steps
Public Autobuilder
Once the distro was first time validated by QA team it will be running on public autobuilder different build set already defined [[]], after that it going to be under the focus of SWAT team
Process
This section describes the process to add a new Distro as supported on Yocto Project
Enable Distro
To Be Updated
Execute Build Sets
Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS
| RPM
|
DEB
|
IPK
|
Component
|
Multilib
|
World
|
oe-selftest
|
| ARM
|
x86_64
|
x86_32
|
Eclipse
|
x86 64/32
|
x86_64
|
|
| init
|
systemd
|
systemd
|
Toaster
|
x86 64/x32
|
systemd
|
|
| poky_lsb
|
poky
|
poky_tiny
|
buildtools
|
core-image-sato
|
poky
|
|
| core-image-lsb
|
core-image-sato-sdk
|
core-image-minimal
|
read_only rootfs
|
poky
|
core-image-sato-sdk
|
|
|
|
|
|
pam
|
init/systemd
|
|
|
|
|
|
|
Log rotate
|
RPM,DEB,IPK
|
|
|
Update Supported Distros
Once all the build sets were executed and all were PASS at least once the Distro can be added to the list of supported Dsitros
meta-poky/conf/distro/poky.conf
Requirements
HW Requirements
Software Requirements
Environment Requirements
Test execution Cycle
GDC Autobuilder
- The execution cycle should be applied only one time when Distro is installed on the autobuilder, results under focus of QA team
Public Autobuilder
- There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team
Release Criteria/ Exit Criteria
- All the defined build steps are PASS [#]
