Toaster testing plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 1: Line 1:
[[Category:Toaster]]
[[Category:eSDK]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob eSDK].


= About Toaster =
= About eSDK =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.
eSDK is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so eSDK provides information and statistics about the build process. eSDK can be used to configure and start builds on multiple remote build servers.


Currently positive testing is being executed for existing features.
Currently positive testing is being executed for existing features.
Line 12: Line 12:


= Objectives =
= Objectives =
Verify all Toaster components to be fully functional.
Verify all eSDK components to be fully functional.
   
   
Components to be verified:
Components to be verified:
* UI interface
* UI interface


The testing objective involves only positive testing for existing features on Toaster.
The testing objective involves only positive testing for existing features on eSDK.
Perform exploratory testing focusing on newer features  
Perform exploratory testing focusing on newer features  
*this can sometimes generate new test cases.
*this can sometimes generate new test cases.
Line 27: Line 27:


= Team members =
= Team members =
==QA Team involved in Toaster testing==
==QA Team involved in eSDK testing==


  [mailto:stanciux.mihail@intel.com Mihail, Stanciu]
  [mailto:stanciux.mihail@intel.com Mihail, Stanciu]
Line 47: Line 47:


Backend features:
Backend features:
*Toaster installation and first time configuration
*eSDK installation and first time configuration
*Data presence
*Data presence
*Data consistency  
*Data consistency  
*Build data
*Build data


  YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
  YP2.1 TODO - we need eSDK-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
  YP2.1 TODO - add mysql testing as well
  YP2.1 TODO - add mysql testing as well


= Test Strategy =
= Test Strategy =
There are several test approaches for Toaster Testing, such as:
There are several test approaches for eSDK Testing, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
*Write new test cases based on developer request or new features requires it.
*Write new test cases based on developer request or new features requires it.
Line 66: Line 66:
=== FrontEnd - UI Basic functionalities ===
=== FrontEnd - UI Basic functionalities ===


Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.
eSDK UI features are tested such as links, buttons, tables, pages basic functionalities.


Testing needs :
Testing needs :
Line 72: Line 72:
*Latest version of Selenium as python module
*Latest version of Selenium as python module
*Latest version of Firefox
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean eSDK setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_eSDK
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs  
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs  
**currently working on their automation using Selenium
**currently working on their automation using Selenium
Line 80: Line 80:
=== BackEnd functionalities ===
=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc
Covers data integrity, data transfer between Bitbake and eSDK, Start/Stop, building images, configuration, etc


Testing needs:
Testing needs:
Line 88: Line 88:
<!--  
<!--  
Brian's Comments
Brian's Comments
1) I would like to remove the manual element from all the toaster tests.
1) I would like to remove the manual element from all the eSDK tests.
2) unless we are using something from within bitbake that I didn't see these tests do seem to be more amenable to django unit tests since they are already using the orm.models.*  
2) unless we are using something from within bitbake that I didn't see these tests do seem to be more amenable to django unit tests since they are already using the orm.models.*  
3)we should avoid presuming that we are running from inside poky. referencing ../bitbake/lib/toaster is a bad idea.
3)we should avoid presuming that we are running from inside poky. referencing ../bitbake/lib/eSDK is a bad idea.
-->
-->


*Requires sqlite3 client installed (for manual debugging/investigation)
*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
**In order for these tests to run properly, check the the steps for “The oe-selftest eSDK tests” from https://wiki.yoctoproject.org/wiki/Testing_eSDK
*For BackEnd managed mode  manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030  
*For BackEnd managed mode  manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030  
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster
*please validate and add info from Testing eSDK wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_eSDK


== Test automation ==
== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.
eSDK has 3 testing suites, targeted at verifying different parts of the system.


=== Django Unit Tests ===
=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.
eSDK is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.


To run the django unit tests, invoke them as with any Django test suite
To run the django unit tests, invoke them as with any Django test suite
  ./bitbake/lib/toaster/manage.py test
  ./bitbake/lib/eSDK/manage.py test


To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.
To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.
Line 114: Line 114:
<!--
<!--


=== Toaster Test System (TTS) ===
=== eSDK Test System (TTS) ===


The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.
The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start eSDK in both modes, and runs HTML5 validation on HTML-returning pages.
The tests are designed to be triggered through a continuous integration system.
The tests are designed to be triggered through a continuous integration system.


To start manually it, run the tts test runner:
To start manually it, run the tts test runner:
  ./bitbake/lib/toaster/contrib/tts/runner.py
  ./bitbake/lib/eSDK/contrib/tts/runner.py


-->
-->


=== The oe-selftest toaster tests ===
=== The oe-selftest eSDK tests ===
=== LEGACY === These are no longer valid as of 2.1 M3
=== LEGACY === These are no longer valid as of 2.1 M3


These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.
These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/eSDKui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.


In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
In order for these tests to run properly, the following steps need to be taken before running, on a clean eSDK setup:


*Create new project using local poky release
*Create new project using local poky release
Line 141: Line 141:
   source oe-init-build-env
   source oe-init-build-env


then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the eSDK.sqlite file:
   ln -s ../toaster.sqlite toaster.sqlite
   ln -s ../eSDK.sqlite eSDK.sqlite


and then run the command:
and then run the command:
   oe-selftest --run-tests _toaster
   oe-selftest --run-tests _eSDK


=== The selenium automated UI tests ===
=== The selenium automated UI tests ===
Line 164: Line 164:


The selenium tests are located in:
The selenium tests are located in:
  ./ bitbake/lib/toaster/contrib/tts/toasteruitest
  ./ bitbake/lib/eSDK/contrib/tts/eSDKuitest


You run them with:
You run them with:
  ./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py
  ./bitbake/lib/eSDK/contrib/tts/eSDKuitest/run_eSDKtests.py


The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.
The above command takes the list of the tests to run from the eSDK_test.cfg file. The tests themselves are located in eSDK_automation_test.py. The eSDK_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the eSDK instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.


In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
In order for these tests to run properly, the following steps need to be taken before running, on a clean eSDK setup:
*Create new project named "selenium-project" using local poky release
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-minimal image
Line 181: Line 181:
  YP2.1 TODO: Update this section, since the steps are a WIP for automation.
  YP2.1 TODO: Update this section, since the steps are a WIP for automation.


In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:
In order to run all current tests, you need to modify the configuration file "eSDK_test.cfg" with the following line:


  test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]
  test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]




The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash
The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/eSDK_tests_squash


== Test Approach ==
== Test Approach ==
Line 214: Line 214:
== Maintaining the test cases ==
== Maintaining the test cases ==


Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.  
eSDK test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.  


A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.
A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.


As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
As currently Testopia is structured, all eSDK test cases are stored under Classifications/Build System & Metadata/ eSDK component within Testopia. All the test cases are organized in Test Run Templates as following:
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*eSDK Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*eSDK Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*eSDK UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*eSDK UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043
*eSDK Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043


== Submitting Bugs ==
== Submitting Bugs ==
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.  
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.  
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].


=Requirements=
=Requirements=
Line 260: Line 260:


*Latest version of Selenium installed as python module
*Latest version of Selenium installed as python module
*YP QS (Maybe, depending where Toaster server is installed)
*YP QS (Maybe, depending where eSDK server is installed)
*SQLITE 3 Client
*SQLITE 3 Client


Line 267: Line 267:
==Features to be tested==
==Features to be tested==
===Backend===
===Backend===
*Toaster installation and first time configuration
*eSDK installation and first time configuration
**Download poky  
**Download poky  
**Checkout on desired version
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command
**Start eSDK with “bitbake/bin/eSDK” command


*Data integrity  
*Data integrity  
Line 276: Line 276:
**Data consistency - the information is being recorded properly when modified from UI
**Data consistency - the information is being recorded properly when modified from UI
**Build data - all the information from bitbake process is being transmitted properly to the database
**Build data - all the information from bitbake process is being transmitted properly to the database
*Data transfer between Toaster and Bitbake  
*Data transfer between eSDK and Bitbake  
*Toaster and Bitbake complete cycle communication
*eSDK and Bitbake complete cycle communication


;The test runs templates for Toaster backend are are found in Testopia, as following
;The test runs templates for eSDK backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*eSDK Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*eSDK Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030


===Frontend===
===Frontend===
Line 326: Line 326:
****bitbake
****bitbake


;The test runs templates for Toaster frontend are are found in Testopia, as following:
;The test runs templates for eSDK frontend are are found in Testopia, as following:
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*eSDK UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*eSDK UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029


== New Features to be tested 2.1 ==
== New Features to be tested 2.1 ==
Line 342: Line 342:
*SQLITE database
*SQLITE database
**add MySQL database testing
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)
**add running eSDK as a service with Apache (e.g. production instance)


*YP 2.0 Release:
*YP 2.0 Release:
Line 361: Line 361:
*Layer integrity (covering all layers is impossible)
*Layer integrity (covering all layers is impossible)
**AR Mihai to get Brian’s supported layer list
**AR Mihai to get Brian’s supported layer list
*Toaster-next branch
*eSDK-next branch
**what’s the easiest way to run the automated tests for a person.
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
**holy grail - make a single framework solution.
Line 369: Line 369:
= Schedule =
= Schedule =


*Toaster is tested only at Full Pass by QA Team
*eSDK is tested only at Full Pass by QA Team
**the plan is to run it according to the Test execution Cycle, after automation is in place.
**the plan is to run it according to the Test execution Cycle, after automation is in place.
*Toaster is tested on toaster-next branch before every merge into Master by Dev team
*eSDK is tested on eSDK-next branch before every merge into Master by Dev team


*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].
*eSDK will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].




Line 402: Line 402:


;Sanity Test is described [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Overall_Test_Plan#Sanity_Testing here].
;Sanity Test is described [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Overall_Test_Plan#Sanity_Testing here].
*run on toaster-next or poky-contrib.  
*run on eSDK-next or poky-contrib.  
**TBD  
**TBD  
***need info from DEV owners on what Sanity testing is done by them
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
***detail the set of tests. e.g. django unit tests, eSDK oe-selftest
Weekly test
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
*E2E consists of all automated tests that can be run on eSDK
**oe-selftest toaster tests
**oe-selftest eSDK tests
*selenium UI tests
*selenium UI tests
*TBD  
*TBD  
Line 418: Line 418:
**Automated tests:
**Automated tests:
***Django Unit Tests
***Django Unit Tests
***oe-selftest toaster tests
***oe-selftest eSDK tests
***Selenium automated UI tests
***Selenium automated UI tests
**Manual tests
**Manual tests
***The non-automated tests from the Toaster test runs.
***The non-automated tests from the eSDK test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.


Line 427: Line 427:
*YP Dependencies
*YP Dependencies
**Bitbake works
**Bitbake works
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium  
*Some changes in the eSDK UI might affect running the UI automated tests using Selenium  
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
**e.g. Changing the element ID’s in the eSDK pages reflected in failing all UI related automated TC’s
*recipe image we’re testing to get build data (metadata broken)
*recipe image we’re testing to get build data (metadata broken)


Line 436: Line 436:
*Project Risks
*Project Risks
**QA and DEV to define this
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**eSDK doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Changes in Bitbake, bitbake logging might affect eSDK functionality
**Is reliant by host/browser environment
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
***TBD - expand the eSDK test framework to the supported host distributions of YP
*Testing Risks
*Testing Risks
**Toaster quality might be affected
**eSDK quality might be affected


=Tools=
=Tools=
Line 467: Line 467:


= Test Areas =
= Test Areas =
Toaster consists of two big components, as follows:
eSDK consists of two big components, as follows:


== Backend ==
== Backend ==


==== Functionality tests ====
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected through the simple UI;
Line 479: Line 479:


==== Usability tests ====
==== Usability tests ====
*  Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the toaster server])
*  Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])


== Frontend ==
== Frontend ==
Line 540: Line 540:


*; Objective:
*; Objective:
** Ensure functionality of Toaster component.
** Ensure functionality of eSDK component.


== Release Test ==
== Release Test ==

Revision as of 13:27, 19 May 2016

This article is the test plan for eSDK.

About eSDK

eSDK is a web interface to the Yocto Project’s OpenEmbedded build system that allows a user to run and build his own builds. The information is collected and stored in a database, so eSDK provides information and statistics about the build process. eSDK can be used to configure and start builds on multiple remote build servers.

Currently positive testing is being executed for existing features. Tasks:

  • Validating the current build (running regression test)
  • Bug & Feature verification
  • Exploratory on existing functionalities

Objectives

Verify all eSDK components to be fully functional.

Components to be verified:

  • UI interface

The testing objective involves only positive testing for existing features on eSDK. Perform exploratory testing focusing on newer features

  • this can sometimes generate new test cases.

Verifying the bugs and features.

=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:

  • Backend interaction with bitbake for build purposes
  • Backend interaction with database for storing and accessing build information

Team members

QA Team involved in eSDK testing

Mihail, Stanciu
Agurida, Cristina 
Georgescu, Alexandru

Scope

Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:

  • Create new project
  • Modify project settings and variables
  • Add/Remove layers to project
  • Building images
  • Build details
  • All builds report table

Backend features:

  • eSDK installation and first time configuration
  • Data presence
  • Data consistency
  • Build data
YP2.1 TODO - we need eSDK-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well

Test Strategy

There are several test approaches for eSDK Testing, such as:

  • Perform test cases agreed upon the development during periodic full passes according to the Schedule
  • Write new test cases based on developer request or new features requires it.
    • Maintain current test cases and update them accordingly
  • Perform exploratory testing on existing functionalities.
    • YP 2.1 Testopia template here.

Major feature groups and their testing approach

FrontEnd - UI Basic functionalities

eSDK UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :


BackEnd functionalities

=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing. Covers data integrity, data transfer between Bitbake and eSDK, Start/Stop, building images, configuration, etc

Testing needs:

  • Built in python 2.7
  • Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.

Test automation

eSDK has 3 testing suites, targeted at verifying different parts of the system.

Django Unit Tests

eSDK is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/eSDK/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

YP2.1 TODO: QA start running tests


The oe-selftest eSDK tests

=== LEGACY === These are no longer valid as of 2.1 M3

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/eSDKui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean eSDK setup:

  • Create new project using local poky release
  • Create 1 core-image-minimal image
  • Create 1 core-image-sato image to check building from cache
  • Break a recipe file (by modifying a link, for example)
  • Create a new image using broken recipe to check error detection is working


To run these tests you need to source an environment by issuing:

 source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the eSDK.sqlite file:

 ln -s ../eSDK.sqlite eSDK.sqlite

and then run the command:

 oe-selftest --run-tests _eSDK

The selenium automated UI tests

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:

  • scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
  • Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate

The selenium tests are located in:

./ bitbake/lib/eSDK/contrib/tts/eSDKuitest

You run them with:

./bitbake/lib/eSDK/contrib/tts/eSDKuitest/run_eSDKtests.py

The above command takes the list of the tests to run from the eSDK_test.cfg file. The tests themselves are located in eSDK_automation_test.py. The eSDK_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the eSDK instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean eSDK setup:

  • Create new project named "selenium-project" using local poky release
  • Create 1 core-image-minimal image
  • Break a recipe file (by modifying a link, for example)
  • Create a new image using broken recipe to check error detection is working
  • Fix the recipe broken in previous step
  • Create 1 core-image-sato image
YP2.1 TODO: Update this section, since the steps are a WIP for automation.

In order to run all current tests, you need to modify the configuration file "eSDK_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]


The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/eSDK_tests_squash

Test Approach

Regression Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.

Sanity testing

  • Not covered at this moment
  • TBD following DEV discussions

Performance and Stress

  • Testing two concurrent builds
  • TC…

Load testing

  • Not covered
  • TBD

System Integration Testing

  • Not covered (maybe covered by DEV w/ Jenkins)

Regression

  • All Full pass tests for
    • BackEnd
    • FrontEnd

Maintaining the test cases

eSDK test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all eSDK test cases are stored under Classifications/Build System & Metadata/ eSDK component within Testopia. All the test cases are organized in Test Run Templates as following:

Submitting Bugs

Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

HW Requirements

A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

Software Requirements

Environment Requirements

  • Supported Browsers – no official browser selected yet, usually Firefox is used
  • Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
    • YP 2.0 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2
  • Latest version of Selenium installed as python module
  • YP QS (Maybe, depending where eSDK server is installed)
  • SQLITE 3 Client


Features

Features to be tested

Backend

  • eSDK installation and first time configuration
    • Download poky
    • Checkout on desired version
    • Start eSDK with “bitbake/bin/eSDK” command
  • Data integrity
    • Data presence - the information is being recorded into the database
    • Data consistency - the information is being recorded properly when modified from UI
    • Build data - all the information from bitbake process is being transmitted properly to the database
  • Data transfer between eSDK and Bitbake
  • eSDK and Bitbake complete cycle communication
The test runs templates for eSDK backend are are found in Testopia, as following

Frontend

  • Verifying if all view elements work (buttons, tables, sort buttons, etc.)
  • Configurations
    • Communication between UI-BackEnd
    • Browse and build any layers published in OpenEmbedded Metadata Index
    • Import your own layers for building
    • Add/Remove layers from configuration
    • Set configurations variables
    • Select a target or multiple targets to build
  • Building images
    • Build base images(core-image-minimal, core-image-sato etc)
    • Build images with added layers
    • Build images after modifying project settings
  • Data Presentation
    • All builds report table
    • View what was built and what packages were installed into final image
    • Browse the directory structure of the image
    • View the value of all variables form build configuration
    • Examine errors, warnings and trace messages
    • View information about the Bitbake task executed and reused during a build
    • View dependency relations between recipes, packages and tasks
    • View performance informations such as build time, task time, CPU usage and disk I/O
    • Build details
      • Build configuration page
        • Build artifacts
        • Bitbake variables table
      • Tasks report table
      • Metrics report tables
        • CPU usage
        • Disk I/O
        • Time
      • Included packages report table
        • Package details page
      • Recipes used during build
        • Recipe details page
    • Definition data from JSON file
      • file path: “/poky/meta-yocto/conf”
      • contains default settings and informations for:
        • configuration
        • layer source
        • releases
        • bitbake
The test runs templates for eSDK frontend are are found in Testopia, as following

New Features to be tested 2.1

Image Customization

  • Create, edit, delete custom image
  • Add/remove packages
  • Build image
  • Image details page
  • Downloading the image custom recipe

Setup covered by QA tests

  • SQLITE database
    • add MySQL database testing
    • add running eSDK as a service with Apache (e.g. production instance)
  • YP 2.0 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2
  • Firefox browser
  • Add Chrome as well

We don’t cover yet

  • Layer integrity (covering all layers is impossible)
    • AR Mihai to get Brian’s supported layer list
  • eSDK-next branch
    • what’s the easiest way to run the automated tests for a person.
    • holy grail - make a single framework solution.
  • Safari browser
  • TBD - to check with DEV what it’s not covered yet.

Schedule

  • eSDK is tested only at Full Pass by QA Team
    • the plan is to run it according to the Test execution Cycle, after automation is in place.
  • eSDK is tested on eSDK-next branch before every merge into Master by Dev team


Test execution Cycle

Build Sanity Test/E2E Weekly Test Full Pass
Daily Master Y Y NO
Weekly Build Y Y NO
Milestone Build Y Y Y


Sanity Test is described here.
  • run on eSDK-next or poky-contrib.
    • TBD
      • need info from DEV owners on what Sanity testing is done by them
      • detail the set of tests. e.g. django unit tests, eSDK oe-selftest

Weekly test

  • Weekly testing is performed on a weekly basis on the poky weekly build.
  • E2E consists of all automated tests that can be run on eSDK
    • oe-selftest eSDK tests
  • selenium UI tests
  • TBD
    • add E2E integration information
    • add more information

Full Pass

  • On full pass all available tests will be run:
    • Automated tests:
      • Django Unit Tests
      • oe-selftest eSDK tests
      • Selenium automated UI tests
    • Manual tests
      • The non-automated tests from the eSDK test runs.
  • Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

Dependencies

  • YP Dependencies
    • Bitbake works
  • Some changes in the eSDK UI might affect running the UI automated tests using Selenium
    • e.g. Changing the element ID’s in the eSDK pages reflected in failing all UI related automated TC’s
  • recipe image we’re testing to get build data (metadata broken)


Risk Assumptions

  • TBD (AlexG)
  • Project Risks
    • QA and DEV to define this
    • eSDK doesn’t affect atm any other YP component (confirmed)
    • Changes in Bitbake, bitbake logging might affect eSDK functionality
    • Is reliant by host/browser environment
      • TBD - add opensuse as well at some point
      • TBD - expand the eSDK test framework to the supported host distributions of YP
  • Testing Risks
    • eSDK quality might be affected

Tools

  • Refer to Requirements section
  • TBD (need more detailed info of required tools)

Release Criteria/ Exit Criteria