Yocto Project - User contributions [en]

Toaster testing plan

2016-03-04T12:34:38Z

Mihail Stanciu: Undo revision 17769 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.


*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===
=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:34:20Z

Mihail Stanciu: Undo revision 17770 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.


*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===
=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:34:07Z

Mihail Stanciu: Undo revision 17771 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===
=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:33:53Z

Mihail Stanciu: Undo revision 17772 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:33:39Z

Mihail Stanciu: Undo revision 17773 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:33:24Z

Mihail Stanciu: Undo revision 17774 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:33:11Z

Mihail Stanciu: Undo revision 17775 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:32:58Z

Mihail Stanciu: Undo revision 17776 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T12:32:31Z

Mihail Stanciu: Undo revision 17777 by Mihail Stanciu (talk)

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression testing is being performed by the QA team for Frontend according to the Schedule matrix. Part of regression Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:07:45Z

Mihail Stanciu: /* Frontend */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression testing is being performed by the QA team for Frontend according to the Schedule matrix. Part of regression Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:07:29Z

Mihail Stanciu: /* Test Approach */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression testing is being performed by the QA team for Frontend according to the Schedule matrix. Part of regression Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:07:05Z

Mihail Stanciu: /* Scope */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:06:47Z

Mihail Stanciu: /* Objectives */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:06:34Z

Mihail Stanciu: /* Backend */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:05:28Z

Mihail Stanciu: /* Regression */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:05:11Z

Mihail Stanciu: /* The oe-selftest toaster tests */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:04:52Z

Mihail Stanciu: /* BackEnd functionalities */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===
=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:04:05Z

Mihail Stanciu: /* Maintaining the test cases */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.


*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===
=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T08:03:28Z

Mihail Stanciu: /* The oe-selftest toaster tests */

Toaster testing plan

2016-03-04T08:02:38Z

Mihail Stanciu: /* BackEnd functionalities */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.


*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Toaster testing plan

2016-03-04T07:59:01Z

Mihail Stanciu: /* Objectives */

[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].

= About Toaster =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities

= Objectives =
Verify all Toaster components to be fully functional.

Components to be verified:
* UI interface

The testing objective involves only positive testing for existing features on Toaster.
Perform exploratory testing focusing on newer features
*this can sometimes generate new test cases.
Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information

= Team members =
==QA Team involved in Toaster testing==

[mailto:stanciux.mihail@intel.com Mihail, Stanciu]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]

= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table

Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data

YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

= Test Strategy =
There are several test approaches for Toaster Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
**Maintain current test cases and update them accordingly
*Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].

== Major feature groups and their testing approach ==
=== FrontEnd - UI Basic functionalities ===

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium

=== BackEnd functionalities ===
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.


*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
**currently working on their automation
**AR Cristina + Mihai:
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.

=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests



=== The oe-selftest toaster tests ===

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:
source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
ln -s ../toaster.sqlite toaster.sqlite

and then run the command:
oe-selftest --run-tests _toaster

=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash

== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

=== Sanity testing ===
*Not covered at this moment
*TBD following DEV discussions

=== Performance and Stress ===
*Testing two concurrent builds
*TC…

=== Load testing ===
*Not covered
*TBD

=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)

===Regression===
*All Full pass tests for
**BackEnd
**FrontEnd

== Maintaining the test cases ==

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043

== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].

=Requirements=
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

==Software Requirements==
*Django 1.6
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3

==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
*Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*SQLITE 3 Client

=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command

*Data integrity
**Data presence - the information is being recorded into the database
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake
*Toaster and Bitbake complete cycle communication

;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030

===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake

;The test runs templates for Toaster frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029

== New Features to be tested 2.1 ==

Image Customization
*Create, edit, delete custom image
*Add/remove packages
*Build image
*Image details page
*Downloading the image custom recipe

== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)

*YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2

*Firefox browser
*Add Chrome as well

== We don’t cover yet ==
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.

= Schedule =

*Toaster is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team

*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].

==Test execution Cycle==
{| class="wikitable"
! Build
! Sanity Test/E2E
! Weekly Test
! Full Pass
|-
| Daily Master
| Y
| Y
| NO
|-
| Weekly Build
| Y
| Y
| NO
|-
| Milestone Build
| Y
| Y
| Y
|}

;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

=Dependencies=
*YP Dependencies
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)

=Risk Assumptions=
*TBD (AlexG)
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected

=Tools=
*Refer to Requirements section
*TBD (need more detailed info of required tools)

= Release Criteria/ Exit Criteria =

Weekly 5b12268f6e17574999f91628a60e21711cf62ee4

2016-02-11T15:03:50Z

Mihail Stanciu:

[https://wiki.yoctoproject.org/wiki/QA_sanity_history < Back]
 Ran on a genericx86_minnowmax and compared with commit dd0ba9ea4a11ab15348d4fe3574e4b28784db82f
{| class="wikitable"
|+Results
|- style="background-color:#F0F8FC; padding:10px;"
|Package Name ||Tests ran ||Passed ||Failed ||Passrate ||Last Passrate ||Run difference
|-
|genericx86_minnowmax ||57 ||style="background-color:#C4FF33;" |56 ||style="background-color:#F78D72;" |1 ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86_minnowmax weekly runtime 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 1testcase-215 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 0Tests that failed now: 1testcase-1059</div></div>
|-
|genericx86-64_minnowmax ||57 ||style="background-color:#C4FF33;" |56 ||style="background-color:#F78D72;" |1 ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64_minnowmax weekly runtime 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 1testcase-215 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 0Tests that failed now: 1testcase-1059</div></div>
|-
|qemux86 ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|genericx86-lsb_minnowmax ||58 ||style="background-color:#C4FF33;" |55 ||style="background-color:#F78D72;" |2 ||style="background-color:#C4FF33;"|95.0% ||style="background-color:#C4FF33;"|93.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-lsb_minnowmax weekly runtime 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 1testcase-215 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 0Tests that failed now: 2testcase-1059 testcase-316</div></div>
|-
|toaster ||11 ||style="background-color:#F78D72;" |8 ||style="background-color:#F78D72;" |3 ||style="background-color:#F78D72;"|73.0% ||style="background-color:#C4FF33;"|88.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for toaster weekly toaster 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 31testcase-test_946 testcase-test_956 testcase-test_955 testcase-test_948 testcase-test_1093 testcase-test_1091 testcase-test_913 testcase-test_912 testcase-test_911 testcase-test_910 testcase-test_916 testcase-test_915 testcase-test_914 testcase-test_1088 testcase-test_1072 testcase-test_940 testcase-test_1084 testcase-test_943 testcase-test_944 testcase-test_923 testcase-test_1083 testcase-test_947 testcase-test_904 testcase-test_949 testcase-test_906 testcase-test_1089 testcase-test_901 testcase-test_902 testcase-test_903 testcase-test_941 testcase-test_942 Tests that were skipped last time and worked now: 0Tests that failed now: 3testcase-test_1076 testcase-test_1092 testcase-test_1090</div></div>
|-
|qemux86_64 ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|qemumips ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|qemuppc ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|eclipse-luna ||23 ||style="background-color:#C4FF33;" |23 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#F78D72;"|50.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for eclipse-luna weekly eclipse-luna 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 2testcase-25 testcase-26 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 19testcase-39 testcase-38 testcase-32 testcase-31 testcase-388 testcase-380 testcase-385 testcase-386 testcase-387 testcase-42 testcase-27 testcase-153 testcase-157 testcase-159 testcase-374 testcase-377 testcase-373 testcase-379 testcase-378Tests that failed now: 0</div></div>
|-
|genericx86-64-lsb_nuc ||55 ||style="background-color:#C4FF33;" |55 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#F78D72;"|66.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64-lsb_nuc weekly runtime 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 9testcase-965 testcase-1096 testcase-719 testcase-208 testcase-1059 testcase-726 testcase-215 testcase-211 testcase-718 Tests that were skipped now and worked last time: 3testcase-289 testcase-202 testcase-test_1_logrotate_setup Tests that were skipped last time and worked now: 2testcase-195 testcase-234Tests that failed now: 0</div></div>
|-
|genericx86-64_nuc ||57 ||style="background-color:#C4FF33;" |56 ||style="background-color:#F78D72;" |1 ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#C4FF33;"|85.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64_nuc weekly runtime 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 7testcase-965 testcase-208 testcase-723 testcase-1059 testcase-726 testcase-215 testcase-211 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 2testcase-195 testcase-234Tests that failed now: 1testcase-1096</div></div>
|-
|eclipse-mars ||23 ||style="background-color:#C4FF33;" |23 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#F78D72;"|0.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for eclipse-mars weekly eclipse-mars 2016-02-09: ********************************Tests that failed now and worked last time: 0 Tests that passed now and and failed last time: 1testcase-24 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 22testcase-39 testcase-38 testcase-32 testcase-31 testcase-388 testcase-380 testcase-385 testcase-386 testcase-387 testcase-739 testcase-42 testcase-25 testcase-26 testcase-27 testcase-153 testcase-157 testcase-159 testcase-374 testcase-377 testcase-373 testcase-379 testcase-378Tests that failed now: 0</div></div>

Setting up a production instance of Toaster

2016-02-09T15:08:52Z

Mihail Stanciu: /* Requirements */

[[Category:Toaster]]

'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

Edit BUILD_MODE:
<code>
BUILD_MODE = True
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Weekly dd0ba9ea4a11ab15348d4fe3574e4b28784db82f

2016-01-19T14:56:15Z

Mihail Stanciu:

[https://wiki.yoctoproject.org/wiki/QA_sanity_history < Back]
 Ran on a genericx86_minnowmax and compared with commit 5e3e2e0cbb0a49986f4653e64c4c8d2b5461645e
{| class="wikitable"
|+Results
|- style="background-color:#F0F8FC; padding:10px;"
|Package Name ||Tests ran ||Passed ||Failed ||Passrate ||Last Passrate ||Run difference
|-
|genericx86_minnowmax ||57 ||style="background-color:#C4FF33;" |55 ||style="background-color:#F78D72;" |2 ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86_minnowmax weekly runtime 2016-01-18: ********************************Tests that failed now and worked last time: 1testcase-215 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 1testcase-215Tests that failed now: 2testcase-1059 testcase-215</div></div>
|-
|qemux86 ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|genericx86-64_minnowmax ||57 ||style="background-color:#C4FF33;" |55 ||style="background-color:#F78D72;" |2 ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64_minnowmax weekly runtime 2016-01-18: ********************************Tests that failed now and worked last time: 1testcase-215 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 1testcase-215Tests that failed now: 2testcase-1059 testcase-215</div></div>
|-
|genericx86-lsb_minnowmax ||58 ||style="background-color:#C4FF33;" |54 ||style="background-color:#F78D72;" |3 ||style="background-color:#C4FF33;"|93.0% ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-lsb_minnowmax weekly runtime 2016-01-18: ********************************Tests that failed now and worked last time: 2testcase-215 testcase-316 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 0 Tests that were skipped last time and worked now: 1testcase-215Tests that failed now: 3testcase-1059 testcase-215 testcase-316</div></div>
|-
|qemux86_64 ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|genericx86-64_nuc ||55 ||style="background-color:#C4FF33;" |47 ||style="background-color:#F78D72;" |8 ||style="background-color:#C4FF33;"|85.0% ||style="background-color:#C4FF33;"|98.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64_nuc weekly runtime 2016-01-18: ********************************Tests that failed now and worked last time: 7testcase-965 testcase-1096 testcase-208 testcase-723 testcase-726 testcase-215 testcase-211 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 2testcase-195 testcase-234 Tests that were skipped last time and worked now: 1testcase-215Tests that failed now: 8testcase-965 testcase-1096 testcase-208 testcase-723 testcase-726 testcase-1059 testcase-215 testcase-211</div></div>
|-
|genericx86-64-lsb_nuc ||56 ||style="background-color:#F78D72;" |44 ||style="background-color:#F78D72;" |10 ||style="background-color:#F78D72;"|79.0% ||style="background-color:#C4FF33;"|96.0% ||style="background-color:#F78D72;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">Yes<div class="mw-collapsible-content">Results for genericx86-64-lsb_nuc weekly runtime 2016-01-18: ********************************Tests that failed now and worked last time: 9testcase-965 testcase-1096 testcase-208 testcase-723 testcase-726 testcase-test_1_logrotate_setup testcase-215 testcase-211 testcase-718 Tests that passed now and and failed last time: 0 Tests that were skipped now and worked last time: 2testcase-195 testcase-234 Tests that were skipped last time and worked now: 1testcase-215Tests that failed now: 10testcase-965 testcase-1096 testcase-208 testcase-723 testcase-726 testcase-1059 testcase-test_1_logrotate_setup testcase-215 testcase-211 testcase-718</div></div>
|-
|qemuarm ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|qemumips ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>
|-
|qemuppc ||57 ||style="background-color:#C4FF33;" |57 ||style="background-color:#C4FF33;" |0 ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|100.0% ||style="background-color:#C4FF33;"|<div class="toccolours mw-collapsible mw-collapsed" style="overflow: hidden;">No<div class="mw-collapsible-content"></div></div>

Testing Toaster

2015-08-21T11:57:27Z

Mihail Stanciu: /* The oe-selftest toaster tests */

[[Category:Toaster]]

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

== Django Unit Tests ==

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

== Toaster Test System (TTS) ==

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

== The oe-selftest toaster tests ==

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Create new project using local poky release
* Create 1 core-image-minimal image
* Create 1 core-image-sato image to check building from cache
* Break a recipe file (by modifying a link, for example)
* Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:

ln -s ../toaster.sqlite toaster.sqlite

and then run the command:

oe-selftest --run-tests _toaster

== The selenium automated UI tests ==

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

Testing Toaster

2015-08-21T09:58:47Z

Mihail Stanciu: /* The oe-selftest toaster tests */

[[Category:Toaster]]

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

== Django Unit Tests ==

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

== Toaster Test System (TTS) ==

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

== The oe-selftest toaster tests ==

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Create new project using local poky release
* Create 1 core-image-minimal image
* Create 1 core-image-sato image to check building from cache
* Break a recipe file (by modifying a link, for example)
* Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer and then run the command:

oe-selftest --run-tests _toaster

== The selenium automated UI tests ==

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

Testing Toaster

2015-08-20T15:24:20Z

Mihail Stanciu: /* The selenium automated UI tests */

[[Category:Toaster]]

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

== Django Unit Tests ==

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

== Toaster Test System (TTS) ==

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

== The oe-selftest toaster tests ==

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Create new project using local poky release
* Create 1 core-image-minimal image
* Create 1 core-image-sato image to check building from cache
* Break a recipe file (by modifying a link, for example)
* Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

source oe-init-build-env

and then run the command:

oe-selftest --run-tests _toaster

== The selenium automated UI tests ==

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

Testing Toaster

2015-08-20T15:11:53Z

Mihail Stanciu: /* The oe-selftest toaster tests */

[[Category:Toaster]]

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

== Django Unit Tests ==

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

== Toaster Test System (TTS) ==

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

== The oe-selftest toaster tests ==

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Create new project using local poky release
* Create 1 core-image-minimal image
* Create 1 core-image-sato image to check building from cache
* Break a recipe file (by modifying a link, for example)
* Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

source oe-init-build-env

and then run the command:

oe-selftest --run-tests _toaster

== The selenium automated UI tests ==

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

Testing Toaster

2015-08-20T15:09:29Z

Mihail Stanciu: /* The oe-selftest toaster tests */

[[Category:Toaster]]

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

== Django Unit Tests ==

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

== Toaster Test System (TTS) ==

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

== The oe-selftest toaster tests ==

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Create new project using local poky release
* Create 1 core-image-minimal image
* Create 1 core-image-sato image to check building from cache
* Break a recipe file (by modifying a link, for example)
* Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment and then run the command "oe-selftest --run-tests _toaster"

== The selenium automated UI tests ==

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py