Distro Testing Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(remove unneeded info)
 
(107 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Category:Toaster]]
This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder
This article is the test plan for Distro testing


= About Distro Testing =
= About Distro Testing =


Distro Testing is intended to catch bugs that are distribution specific using the yocto-autobuilder.
Distro Testing is part of the process of enabling an OS distribution at the Yocto Project [[Autobuilder]]. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.


= Objectives =
= Test Objectives =


* Verify that Distro executes oe-selftest  
* Verify that Distro executes oe-selftest  
* Verify that Dsitro is able to execute componenets (Eclipse, Toaster, WIC)
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build different image types (LSB, non LSB)
Line 18: Line 17:


= Test Strategy =
= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].


== Major feature groups and their testing approach ==
The strategy will be divided in two groups:
=== FrontEnd - UI Basic functionalities ===


Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.
* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
* [[SWAT]] monitoring at the Autobuilders


Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean Toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium


== Staging AutoBuilder  ==


=== BackEnd functionalities ===
When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc


Testing needs:
== Public Autobuilder ==
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.
<!--
Brian's Comments
1) I would like to remove the manual element from all the Toaster tests.
2) unless we are using something from within bitbake that I didn't see these tests do seem to be more amenable to django unit tests since they are already using the orm.models.*
3)we should avoid presuming that we are running from inside poky. referencing ../bitbake/lib/Toaster is a bad idea.
-->


*Requires sqlite3 client installed (for manual debugging/investigation)
Once the distro was first time validated by QA team it will be running on public autobuilder different [https://autobuilder.yoctoproject.org/main/builders build sets] already defined , after that it going to be under the focus of SWAT team
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest Toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode  manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster


== Test automation ==
= Process =
Toaster has 3 testing suites, targeted at verifying different parts of the system.
This section describes the process to add a new Distro as supported on Yocto Project


=== Django Unit Tests ===
[[File:Distro Steps_v2.PNG]]
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.


To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/Toaster/manage.py test


To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.
== Open a Bug ==


YP2.1 TODO: QA start running tests
When a new Distro is identified to be added as a supported [[#Supported OS Distributions Criteria | candidate]] a bug should be filed on [https://bugzilla.yoctoproject.org/ Bugzilla] and has to be assigned to the [[#QA Responsible | QA Responsible ]] of the Distro Testing.


<!--
*example of a bug [https://bugzilla.yoctoproject.org/show_bug.cgi?id=11005 bug 11005]


=== Toaster Test System (TTS) ===
== Install Distro ==


The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.
QA responsible should ensure that the new Distro is installed properly on the local AB
The tests are designed to be triggered through a continuous integration system.


To start manually it, run the tts test runner:
=== Process To Install New Distro ===
./bitbake/lib/Toaster/contrib/tts/runner.py


-->


=== The oe-selftest Toaster tests ===
1. Install the new distro
=== LEGACY === These are no longer valid as of 2.1 M3


These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/Toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.
2. Install the requirements from the Yocto Project Quickstart


In order for these tests to run properly, the following steps need to be taken before running, on a clean Toaster setup:
3. Start an Autobuilder staging instance
 
  $ git clone http://git.yoctoproject.org/git/yocto-autobuilder
 
  # Ensure the owner of cloned repo is different than root (Poky does not work with root user)
  $ chown -R ab:ab yocto-autobuilder/


*Create new project using local poky release
  $ cd yocto-autobuilder
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working


  # Load buildset configuration with nightly-qa-distro
  $ ln -sf buildset-config.yocto-qa/ buildset-config


To run these tests you need to source an environment by issuing:
  # setup the autobuilder
  source oe-init-build-env
  $ . yocto-autobuilder-setup
 
  # Generates password for AB Web
  $ htpasswd -b .htpasswd distrouser distropass


then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the Toaster.sqlite file:
  # start autobuilder
  ln -s ../Toaster.sqlite Toaster.sqlite
  $ ./yocto-start-autobuilder both


and then run the command:
  oe-selftest --run-tests _Toaster


=== The selenium automated UI tests ===
4. Open a browser and enter to http://localhost:8010
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.
  Note: If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log


The selenium tests require a couple of components:
5. Log in with the distrouser and distropass
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.


The selenium tests are located in:
6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build
./ bitbake/lib/Toaster/contrib/tts/Toasteruitest


You run them with:
7. Wait to the buildsets to end and review the output, load bugs if failures occur
./bitbake/lib/Toaster/contrib/tts/Toasteruitest/run_Toastertests.py


The above command takes the list of the tests to run from the Toaster_test.cfg file. The tests themselves are located in Toaster_automation_test.py. The Toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the Toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.
8. Stop the staging Autobuilder


In order for these tests to run properly, the following steps need to be taken before running, on a clean Toaster setup:
  $ ./yocto-stop-autobuilder both
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image


YP2.1 TODO: Update this section, since the steps are a WIP for automation.
== Execute Build Sets ==


In order to run all current tests, you need to modify the configuration file "Toaster_test.cfg" with the following line:
Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB


test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]
{| class="wikitable"
 
! style="text-align: center;" | RPM
 
! style="text-align: center;" | DEB
The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/Toaster_tests_squash
! style="text-align: center;" | IPK
! style="text-align: center;" | Component
! style="text-align: center;" | Multilib
! style="text-align: center;" | World
! oe-selftest
|-
| ARM
| x86_64
| x86_32
| Eclipse
| x86 64/32
| x86_64
|
|-
| init
| systemd
| systemd
| Toaster
| x86 64/x32
| systemd
|
|-
| poky_lsb
| poky
| poky_tiny
| buildtools
| core-image-sato
| poky
|
|-
| core-image-lsb
| core-image-sato-sdk
| core-image-minimal
| read_only rootfs
| poky  
| core-image-sato-sdk
|
|-
|
|
|
| pam
| init/systemd
|
|
|-
|
|
|
| Log rotate
| RPM,DEB,IPK
|
|
|}


== Test Approach ==
== Results ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.


=== Sanity testing ===
The [[#QA Responsible | QA Responsible]] should ensure that all the defined [[#Execute Build Sets | Build Sets]] are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.
*Not covered at this moment
*TBD following DEV discussions


=== Performance and Stress ===
== Update Supported Distro List ==
*Testing two concurrent builds
*TC…


=== Load testing ===
Once all the build sets were executed and all were PASS, then then it can be added to the list of supported Distros, also the unsupported distros should be removed from the list.
*Not covered
*TBD


=== System Integration Testing ===
To accomplish above step a patch should be sent to the [https://lists.yoctoproject.org/listinfo/poky  poky mailing list] with '''Sanity Tested Distro File''' updated, you can refer to the following guide to [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded  send a patch], remember top use the correct list.
*Not covered (maybe covered by DEV w/ Jenkins)


===Regression===
=== Sanity Tested Distro File (Snapshot)===
*All Full pass tests for
**BackEnd
meta-poky/conf/distro/poky.conf
**FrontEnd
-----
 
SANITY_TESTED_DISTROS ?= " \
== Maintaining the test cases ==
              poky-1.7 \n \
 
              poky-1.8 \n \
Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.  
              poky-2.0 \n \
 
              Ubuntu-14.04 \n \
A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.
              Ubuntu-14.10 \n \
 
              Ubuntu-15.04 \n \
As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
              Ubuntu-15.10 \n \
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
              Ubuntu-16.04 \n \
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
              Fedora-21 \n \
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
              Fedora-22 \n \
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
              Fedora-23 \n \
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043
              CentOS-6.* \n \
 
              CentOS-7.* \n \
== Submitting Bugs ==
              Debian-7.* \n \
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.  
              Debian-8.* \n \
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].
              openSUSE-project-13.2 \n \


=Requirements=
=Requirements=
==HW Requirements==
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution
==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3


==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in  meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2


*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client
=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/Toaster” command
*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication
;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake
;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
== New Features to be tested 2.1 ==
Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe
== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running Toaster as a service with Apache (e.g. production instance)
*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2
*Firefox browser
*Add Chrome as well
== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.
= Schedule =
*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on Toaster-next branch before every merge into Master by Dev team
*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].
==Test execution Cycle==
{| class="wikitable"
{| class="wikitable"
! Build
! style="text-align: center;" | Item
! Sanity Test/E2E
! style="text-align: center;" | Hardware requirements
! Weekly Test
! Full Pass
|-
|-
| Daily Master
| Processor
| Y
| Intel Core i5-6260U (1.9 GHz Turbo, Dual Core, 4 MB Cache, 15W TDP)
| Y
| NO
|-
|-
| Weekly Build
| Memory
| Y
| DDR4 32GB
| Y
| NO
|-
|-
| Milestone Build
| Disk space
| Y
| 2 TB of free disk space is required.
| Y
| Y
|}
|}


==Software Requirements==


To make sure all those packages are installed on the new distro.


;Sanity Test is described [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Overall_Test_Plan#Sanity_Testing here].
{| class="wikitable" style="border: none; background: none;"
*run on Toaster-next or poky-contrib.
! colspan="4"| Packages
**TBD
|-
***need info from DEV owners on what Sanity testing is done by them
| gawk  || make || wget || tar
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
|-
Weekly test
| bzip2|| gzip || python || unzip
*Weekly testing is performed on a weekly basis on the poky weekly build.
|-
*E2E consists of all automated tests that can be run on Toaster
| perl|| patch || diffutils || diffstat
**oe-selftest Toaster tests
|-
*selenium UI tests
| git|| cpp || gcc || gcc-c++
*TBD
|-
**add E2E integration information
| glibc-devel || texinfo || chrpath || ccache
**add more information
|-
Full Pass
| perl-Data-Dumper || perl-Text-ParseWords || perl-Thread-Queue|| socat
*On full pass all available tests will be run:
|-
**Automated tests:
| findutils || which || SDL-devel || xterm
***Django Unit Tests
|-
***oe-selftest Toaster tests
| tigervnc || tigervnc-server || python3 || tunctl
***Selenium automated UI tests
|-
**Manual tests
| gitpython || python3-pip || python3-devel || qemu-kvm
***The non-automated tests from the Toaster test runs.
|-
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.
| perl-bignum ||  ||  || 
|-
|}


=Dependencies=
= Test execution Cycle =
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)


== QA Responsible ==
TBD
meanwhile you can contact [mailto:jose.perz.carranza@intel.com José Pérez Carranza]


=Risk Assumptions=
== Public Autobuilder ==
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected


=Tools=
* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team
*Refer to Requirements section
*TBD (need more detailed info of required tools)


= Release Criteria/ Exit Criteria =
= Exit Criteria for Enabling an OS Distribution =


* All the defined build steps are PASS [[#Execute Build Sets|Execute Build Sets]]
* Distro was added to the supported list [[#Update Supported Distro List|Update Supported Distro]]


= Supported OS Distributions =


The distributions used are:


*Poky
*Fedora
*Ubuntu
*CentOS
*OpenSuse
*Debian


== Supported OS Distributions Criteria ==


List of supported OS Distributions should be the same at [http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#detailed-supported-distros  Mega Manual]  and in variable mentioned in [https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan#Sanity_Tested_Distro_File_.28Snapshot.29  Distro File] if is not equal should a bug should be filled.
The following criteria is used to see if a Distro and his specific version is supported


*'''Poky'''
** Most 2 recent releases
*'''Fedora'''
** Most 2 recent releases
*'''Ubuntu'''
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*'''CentOS'''
** Most recent release
*'''OpenSuse'''
** Most recent release
** Most recent LEAP release
*'''Debian'''
** Most recent release


BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.


=== List ===


 
{| class="wikitable"
 
! style="text-align: center;" | OS Version
 
! style="text-align: center;" | Status
 
<!--
THIS IS THE OLD TESTING PLAN
 
= Test Areas =
Toaster consists of two big components, as follows:
 
== Backend ==
 
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/Toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_Toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
 
==== Usability tests ====
*  Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the Toaster server])
 
== Frontend ==
 
==== Functionality tests ====
*  Manual testing in the first stage;
*  Automate testing using  [http://www.seleniumhq.org/ Selenium], in the second stage;
 
==== Compatibility tests ====
*  Verify the behavior of the GUI on different browsers and operating systems; TBD
 
==== Usability tests ====
*  Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
*  Friendly graphical user interface;
 
==== Performance tests ====
*  Stress testing (e.g. display appropriate error messages when the system is under stress);
 
= Test Cycle =
{|class="wikitable" style="text-align: center;"
| || || colspan="3" | Test execution cycle
|-
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]
|-
| rowspan="2" | Build type || Weekly  || Yes || Yes ||
|-
| Release || Yes || Yes || Yes
|-
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes
|-
|-
| [[#Frontend]]  || Yes || Yes || Yes
| Ubuntu-16.04 LTS
| align="center" style="color:green;" | PASS
|-
|-
| rowspan="5" | Target machine || qemuarm || || Yes || Yes
| Ubuntu-17.04
| align="center" style="color:green;" | PASS
|-
|-
| qemumips|| || || Yes
| Fedora-25
| align="center" style="color:green;" | PASS
|-
|-
| qemuppc|| || || Yes
| Fedora-26
| align="center" style="color:green;" | PASS
|-
|-
| qemux86|| Yes || Yes || Yes
| CentOS-7
| align="center" style="color:green;" | PASS
|-
|-
| qemux86-64  || ||  || Yes
| Debian-9
| align="center" style="color:green;" | PASS
|-
|-
| rowspan="3" | Target image|| core-image-minimal|| Yes || ||
| openSUSE-13.2
| align="center" style="color:green;" | PASS
|-
|-
| core-image-sato-sdk|| || Yes || Yes
| openSUSE-42.2
| align="center" style="color:green;" | PASS
|}
|}


== Weekly Test ==
== Supported OS Distributions Release Calendar ==
*; Scope:
** Images built weekly and released through the distribution team.
 
*; Objective:
** Functionality test on most areas with minimum sets of tests;
** Regression test with high probability to find bugs.
 
== Full Pass Test ==
*; Scope:
** Images built as candidates for milestone or final release;
** Passed [[#Weekly Test]]
 
*; Objective:
** Ensure functionality of Toaster component.


== Release Test ==
This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.
*; Scope:
** Release candidates that pass [[#Full Pass Test]]


*; Objective:
*[https://wiki.ubuntu.com/Releases Ubuntu ]
** All scheduled features are covered, or rescheduled;
*[https://fedoraproject.org/wiki/Releases Fedora]
** All relevant bugs are fixed and verified.
*[https://www.debian.org/releases/ Debian]
-->
*[https://en.opensuse.org/openSUSE:Roadmap OpenSuse]
*[https://wiki.centos.org/Download CentOS]

Latest revision as of 01:28, 6 March 2018

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

About Distro Testing

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project Autobuilder. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

Test Objectives

  • Verify that Distro executes oe-selftest
  • Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
  • Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
  • Verify that Distro is able to build different image types (LSB, non LSB)
  • Verify that Distro is able to build per architecture (arm, x86)
  • Verify that Distro is able to build per bootloader (systed, init)
  • Verify that Distro is able to build poky-tiny
  • Verify that Distro executes multilib

Test Strategy

The strategy will be divided in two groups:

  • Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
  • SWAT monitoring at the Autobuilders


Staging AutoBuilder

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by QA, this is a one-time testing activity, not executed in every milestone o release test cycle.

Public Autobuilder

Once the distro was first time validated by QA team it will be running on public autobuilder different build sets already defined , after that it going to be under the focus of SWAT team

Process

This section describes the process to add a new Distro as supported on Yocto Project

Distro Steps v2.PNG


Open a Bug

When a new Distro is identified to be added as a supported candidate a bug should be filed on Bugzilla and has to be assigned to the QA Responsible of the Distro Testing.

Install Distro

QA responsible should ensure that the new Distro is installed properly on the local AB

Process To Install New Distro

1. Install the new distro

2. Install the requirements from the Yocto Project Quickstart

3. Start an Autobuilder staging instance 

  $ git clone http://git.yoctoproject.org/git/yocto-autobuilder 
  
  # Ensure the owner of cloned repo is different than root (Poky does not work with root user)
  $ chown -R ab:ab yocto-autobuilder/

  $ cd yocto-autobuilder

  # Load buildset configuration with nightly-qa-distro 
  $ ln -sf buildset-config.yocto-qa/ buildset-config

  # setup the autobuilder
  $ . yocto-autobuilder-setup
  
  # Generates password for AB Web
  $ htpasswd -b .htpasswd distrouser distropass

  # start autobuilder
  $ ./yocto-start-autobuilder both


4. Open a browser and enter to http://localhost:8010 

  Note: If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass

6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build

7. Wait to the buildsets to end and review the output, load bugs if failures occur

8. Stop the staging Autobuilder 

  $ ./yocto-stop-autobuilder both

Execute Build Sets

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

RPM DEB IPK Component Multilib World oe-selftest
ARM x86_64 x86_32 Eclipse x86 64/32 x86_64
init systemd systemd Toaster x86 64/x32 systemd
poky_lsb poky poky_tiny buildtools core-image-sato poky
core-image-lsb core-image-sato-sdk core-image-minimal read_only rootfs poky core-image-sato-sdk
pam init/systemd
Log rotate RPM,DEB,IPK

Results

The QA Responsible should ensure that all the defined Build Sets are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

Update Supported Distro List

Once all the build sets were executed and all were PASS, then then it can be added to the list of supported Distros, also the unsupported distros should be removed from the list.

To accomplish above step a patch should be sent to the poky mailing list with Sanity Tested Distro File updated, you can refer to the following guide to send a patch, remember top use the correct list.

Sanity Tested Distro File (Snapshot)

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
             poky-1.7 \n \
             poky-1.8 \n \
             poky-2.0 \n \
             Ubuntu-14.04 \n \
             Ubuntu-14.10 \n \
             Ubuntu-15.04 \n \
             Ubuntu-15.10 \n \
             Ubuntu-16.04 \n \
             Fedora-21 \n \
             Fedora-22 \n \
             Fedora-23 \n \
             CentOS-6.* \n \
             CentOS-7.* \n \
             Debian-7.* \n \
             Debian-8.* \n \
             openSUSE-project-13.2 \n \

Requirements

HW Requirements

Item Hardware requirements
Processor Intel Core i5-6260U (1.9 GHz Turbo, Dual Core, 4 MB Cache, 15W TDP)
Memory DDR4 32GB
Disk space 2 TB of free disk space is required.

Software Requirements

To make sure all those packages are installed on the new distro.

Packages
gawk make wget tar
bzip2 gzip python unzip
perl patch diffutils diffstat
git cpp gcc gcc-c++
glibc-devel texinfo chrpath ccache
perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue socat
findutils which SDL-devel xterm
tigervnc tigervnc-server python3 tunctl
gitpython python3-pip python3-devel qemu-kvm
perl-bignum

Test execution Cycle

QA Responsible

TBD meanwhile you can contact José Pérez Carranza

Public Autobuilder

  • There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

Exit Criteria for Enabling an OS Distribution

Supported OS Distributions

The distributions used are:

  • Poky
  • Fedora
  • Ubuntu
  • CentOS
  • OpenSuse
  • Debian

Supported OS Distributions Criteria

List of supported OS Distributions should be the same at Mega Manual and in variable mentioned in Distro File if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

  • Poky
    • Most 2 recent releases
  • Fedora
    • Most 2 recent releases
  • Ubuntu
    • Most recent release
    • Most recent LTS release
    • When above are the same, use most recent 2 releases
  • CentOS
    • Most recent release
  • OpenSuse
    • Most recent release
    • Most recent LEAP release
  • Debian
    • Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

List

OS Version Status
Ubuntu-16.04 LTS PASS
Ubuntu-17.04 PASS
Fedora-25 PASS
Fedora-26 PASS
CentOS-7 PASS
Debian-9 PASS
openSUSE-13.2 PASS
openSUSE-42.2 PASS

Supported OS Distributions Release Calendar

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Distro_Testing_Plan&oldid=36634"