Extensible SDK Test Plan (eSDK): Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with "Category:Toaster This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster]. = About Toaster = Toaster is a web interface to the [http://www.yo...")
 
 
(108 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:Toaster]]
[[Category:eSDK]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].
This article is the test plan for eSDK.


= About Toaster =
= About eSDK =
Toaster is a web interface to the [http://www.yoctoproject.org/docs/2.0/dev-manual/dev-manual.html#build-system-term Yocto Project’s OpenEmbedded] build system that allows a user to run and build his own builds. The information is collected and stored in a database, so Toaster provides information and statistics about the build process. Toaster can be used to configure and start builds on multiple remote build servers.
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]
 
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]
Currently positive testing is being executed for existing features.
Tasks:
*Validating the current build (running regression test)
*Bug & Feature verification
*Exploratory on existing functionalities


= Objectives =
= Objectives =
Verify all Toaster components to be fully functional.
Verify all Extensible SDK components to be fully functional.
Components to be verified:
Components to be verified:
* UI interface


The testing objective involves only positive testing for existing features on Toaster.
= Team members =
Perform exploratory testing focusing on newer features
==QA Team involved in eSDK testing==
*this can sometimes generate new test cases.
Verifying the bugs and features.
 
=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
* Backend interaction with bitbake for build purposes
* Backend interaction with database for storing and accessing build information


= Team members =
==QA Team involved in Toaster testing==


  [mailto:stanciux.mihail@intel.com Mihail, Stanciu]
  [mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]
[mailto:cristina-danielax.agurida@intel.com Agurida, Cristina ]
[mailto:alexandru.c.georgescu@intel.com Georgescu, Alexandru]


= Scope =
= Scope =
Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.
<b><p>eSDK</p></b>
* eSDK Installation.
* eSDK Setup.
<b><p>Devtool features to be tested:</p></b>
* Add a new recipe.
* Modify the source for an existing recipe.
* Upgrade an existing recipe.
* Show work-space status.
* Search available recipes.
* Edit a recipe file in your work-space.
* Get help on configure script options.
* Build a recipe.
* Apply changes from external source tree to recipe.
* Remove a recipe from your work-space.
* Deploy recipe output files to live target machine.
* Un-deploy recipe output files in live target machine.
* Build packages for a recipe.
* Add Native Tools.
* Build image including work-space recipe packages.
* Run QEMU on the specified image.
* Build a derivative SDK of this one.
* Extract the source for an existing recipe.
* Synchronize the source tree for an existing recipe.
* Update SDK components.
* Install additional SDK components.
* Locating Pre-Built SDK Installers.


New features are to be tested as they are finished with test cases written for them once features are stabilized.
<p><b>Node.js</b></p>
 
* Add Node.js Modules
UI features being tested:
*Create new project
*Modify project settings and variables
*Add/Remove layers to project
*Building images
*Build details
*All builds report table
 
Backend features:
*Toaster installation and first time configuration
*Data presence
*Data consistency
*Build data
 
YP2.1 TODO - we need toaster-next on poky-contrib to be tested. Basically to add some of the poky-contrib branches to be tested routinely
YP2.1 TODO - add mysql testing as well


= Test Strategy =
= Test Strategy =
There are several test approaches for Toaster Testing, such as:
There are several test approaches for Extensible SDK, such as:
*Perform test cases agreed upon the development during periodic full passes according to the Schedule
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.
*Write new test cases based on developer request or new features requires it.
Write new test cases based on developer request or new features as required.
**Maintain current test cases and update them accordingly
Maintain current test cases and update them accordingly the new changes.
*Perform exploratory testing on existing functionalities.
Perform exploratory testing on existing functionalities.
**YP 2.1 Testopia template [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043 here].


== Major feature groups  and their testing approach ==
=== FrontEnd - UI Basic functionalities ===
Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.
Testing needs :
*Built only in python 2.7
*Latest version of Selenium as python module
*Latest version of Firefox
*In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
**ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
**currently working on their automation using Selenium
=== BackEnd functionalities ===
=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc
Testing needs:
*Built in python 2.7
*Direct sql requests to local sqlite instance
YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.
<!--
Brian's Comments
1) I would like to remove the manual element from all the toaster tests.
2) unless we are using something from within bitbake that I didn't see these tests do seem to be more amenable to django unit tests since they are already using the orm.models.*
3)we should avoid presuming that we are running from inside poky. referencing ../bitbake/lib/toaster is a bad idea.
-->
*Requires sqlite3 client installed (for manual debugging/investigation)
*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
*For BackEnd managed mode  manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster


== Test automation ==
== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.


=== Django Unit Tests ===
Currently the execution of the cases is using "testsdkext" automation below are the three scenarios that should be tested
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.


To run the django unit tests, invoke them as with any Django test suite
* Test Building the eSDK installer:
./bitbake/lib/toaster/manage.py test


To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.
  1. Clone poky and source the environment
      $ source oe-init-build-env


YP2.1 TODO: QA start running tests
  2. Add proper configuration to local.conf
      $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf
      $ echo 'INHERIT += "testsdk"' >> conf/local.conf


<!--
  3. Build eSDK intaller
 
      $ bitbake core-image-minimal  -c populate_sdk_ext -v
=== Toaster Test System (TTS) ===
 
The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.
The tests are designed to be triggered through a continuous integration system.
 
To start manually it, run the tts test runner:
./bitbake/lib/toaster/contrib/tts/runner.py


-->
  4. Execute the tests
      $ bitbake core-image-minimal -c testsdkext


=== The oe-selftest toaster tests ===
  5. All test should be PASSED
=== LEGACY === These are no longer valid as of 2.1 M3


These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.


In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Test using Pre-Built SDK Installers:


*Create new project using local poky release
  1. Under poky/ do another build folder different from the previous scenario
*Create 1 core-image-minimal image
      $ source oe-init-build-env build-pre-built
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working


  2. Add proper configuration to local.conf
      $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf
      $ echo 'INHERIT += "testsdk"' >> conf/local.conf


To run these tests you need to source an environment by issuing:
  3. Create sdk directory
  source oe-init-build-env
      $ mkdir -p tmp/deploy/sdk


then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
  4. Download artifacts form AB under new folder created
  ln -s ../toaster.sqlite toaster.sqlite
      $ cd  tmp/deploy/sdk 
      $ wget <AB_LINK>/toolchain/x86_64/<ARTIFACTS_NEEDED>
       
          Example:
          $ wget https://autobuilder.yocto.io/pub/releases/yocto-2.4_M3.rc1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest 
         
          NOTE: All the artifacts that should be downloaded are:
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.target.manifest
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.sh
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.testdata.jason


and then run the command:
    5. Give execution permissions to sh files. Rename files, only if it asks.
  oe-selftest --run-tests _toaster
      $ chmod +x *.sh
      $ for file in *<RELEASE>*; do mv "$file" "${file/<RELEASE>/<RELEASE>+snapshot}"; done
        Example execute below command:  
        $ for file in *2.3*; do mv "$file" "${file/2.3/2.3+snapshot}"; done


=== The selenium automated UI tests ===
    6. Go back to poky build folder "poky/build-pre-built"
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.
      $ cd -


The selenium tests require a couple of components:
    7. Prepare the environment to execute tests   
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
      $ bitbake libxml2
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.


The selenium tests are located in:
    8. Execute the test 
./ bitbake/lib/toaster/contrib/tts/toasteruitest
      $ bitbake core-image-minimal -c testsdkext


You run them with:
    9. All test should be PASSED
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py


The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.
    NOTE: There are 4 combinations of tests that should be executed
        - qemux86 + minimal --> (poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext*)
        - qemux86 + sato    --> (poky-glibc-x86_64-core-image-sato-i586-toolchain-ext*)
        - qemux86-64 + minimal  --> (poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext*)
        - qemux86-64 + sato    --> (poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext*)


In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
* Test eSDK as part of the oe-selftest suite
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image


YP2.1 TODO: Update this section, since the steps are a WIP for automation.
  1. Under poky/ do another build folder different from the previous scenario
      $ source oe-init-build-env build-esdk-selftest


In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:
  2. Avoid Sanity Distro validation
      $  echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf


test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]
  3. Execute the tests
      $ oe-selftest -r eSDK


 
  4. All test cases should be PASS
The code for running Selenium UI tests is available here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash


== Test Approach ==
== Test Approach ==
Regression
Regression testing is being performed by the QA team for Backend and Frontend according to the Schedule matrix. Part of regression Backend and Frontend are covered.


=== Sanity testing ===
=== Sanity testing ===
*Not covered at this moment
* Not covered at this moment
*TBD following DEV discussions
* TBD following DEV discussions


=== Performance and Stress ===
=== Performance and Stress ===
*Testing two concurrent builds
* Usability
*TC…
 
=== Load testing ===
*Not covered
*TBD
 
=== System Integration Testing ===
*Not covered (maybe covered by DEV w/ Jenkins)


===Regression===
===Regression===
*All Full pass tests for
* Regression testing will be covered on Automation execution.
**BackEnd
**FrontEnd


== Maintaining the test cases ==
== Maintaining the test cases ==
== Submitting Bugs ==
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].


Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.
=Requirements=
 
A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.


As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043


== Submitting Bugs ==
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p>
Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.  
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p>
Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p>
 
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p>
=Requirements=
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake.   - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p>
==HW Requirements==
==HW Requirements==
A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution
* Any machine able to build.


==Software Requirements==
==Software Requirements==
*Django 1.6
* eSDK.
*South 0.8.4
*argparse 1.2.1
*wsgiref 0.1.2
*beautifulsoup4 4.4.0 or greater
*A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
*YP [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start] requirements
**Git 1.7.8 or greater
**tar 1.24 or greater
**Python 2.7.3


==Environment Requirements==
==Environment Requirements==
*Supported Browsers – no official browser selected yet, usually Firefox is used
**YP 2.1 Release:
*Supported operating systems as per SANITY_TESTED_DISTROS listed in  meta-yocto/conf/distro/poky.conf
**YP 2.0 Release:
**Ubuntu-14.04
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-14.10
Line 259: Line 194:
**openSUSE-project-13.2
**openSUSE-project-13.2


*Latest version of Selenium installed as python module
=Features=
*YP QS (Maybe, depending where Toaster server is installed)
<b> Features to be Tested </b>
*SQLITE 3 Client
 
<b><p>1. Beggining on a recipe.</p></b>
<p>1.1 add. </p>
<p>1.2 modify. Done in oe-selftest</p>
<p>1.3 upgrade. Done in oe-selftest</p>
 


<b><p>2. Getting Information.</p></b>
<p>2.1 Status.</p>
<p>2.2 Check.</p>


=Features=
==Features to be tested==
===Backend===
*Toaster installation and first time configuration
**Download poky
**Checkout on desired version
**Start Toaster with “bitbake/bin/toaster” command


*Data integrity
<b><p>3. Working on a recipe in the workspace.</p></b>
**Data presence - the information is being recorded into the database
<p>3.1 edit-recipe.</p>
**Data consistency - the information is being recorded properly when modified from UI
<p>3.2 configure-help.</p>
**Build data - all the information from bitbake process is being transmitted properly to the database
<p>3.3 build. - Done in oe-selftest</p>
*Data transfer between Toaster and Bitbake
<p>3.4 update recipe. - Done in oe-selftest</p>
*Toaster and Bitbake complete cycle communication
<p>3.5 reset. - Done in oe-selftest</p>


;The test runs templates for Toaster backend are are found in Testopia, as following
*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030


===Frontend===
*Verifying if all view elements work (buttons, tables, sort buttons, etc.)
*Configurations
**Communication between UI-BackEnd
**Browse and build any layers published in [http://layers.openembedded.org/layerindex/branch/master/layers/ OpenEmbedded Metadata Index]
**Import your own layers for building
**Add/Remove layers from configuration
**Set configurations variables
**Select a target or multiple targets to build
*Building images
**Build base images(core-image-minimal, core-image-sato etc)
**Build images with added layers
**Build images after modifying project settings
*Data Presentation
**All builds report table
**View what was built and what packages were installed into final image
**Browse the directory structure of the image
**View the value of all variables form build configuration
**Examine errors, warnings and trace messages
**View information about the Bitbake task executed and reused during a build
**View dependency relations between recipes, packages and tasks
**View performance informations such as build time, task time, CPU usage and disk I/O
**Build details
***Build configuration page
****Build artifacts
****Bitbake variables table
***Tasks report table
***Metrics report tables
****CPU usage
****Disk I/O
****Time
***Included packages report table
****Package details page
***Recipes used during build
****Recipe details page
**Definition data from JSON file
***file path: “/poky/meta-yocto/conf”
***contains default settings and informations for:
****configuration
****layer source
****releases
****bitbake


;The test runs templates for Toaster frontend are are found in Testopia, as following:
<b><p>4. Testing changes on target.</p></b>
*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
<p>4.1 deploy-target. - Done in oe-selftest</p>
*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
<p>4.2 undeploy-target.</p>
<p>4.3 package.</p>
<p>4.4 build-image.- Done in oe-selftest</p>
<p>4.5 runqemu. - Done in oe-selftest </p>


== New Features to be tested 2.1 ==


Image Customization
<b><p>5. Advanced.</p></b>
*Create, edit, delete custom image
<p>5.1 build-sdk.</p>
*Add/remove packages
<p>5.2 extract. Done in oe-selftest</p>
*Build image
<p>5.3 sync.</p>
*Image details page
*Downloading the image custom recipe


== Setup covered by QA tests ==
*SQLITE database
**add MySQL database testing
**add running TOaster as a service with Apache (e.g. production instance)


*YP 2.0 Release:
<b><p>6. SDK maintenance.</p></b>
**Ubuntu-14.04
<p>6.1 sdk-update.</p>
**Ubuntu-14.10
<p>6.2 sdk-install.</p>
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2


*Firefox browser
<b><p> 7. sstate relevant features. </p></b>
*Add Chrome as well
<p>7.1 Ability to generate target packages in SDK. </p>
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p>


== We don’t cover yet ==
<b><p> 8. Kernel relevant recipe and build. </p></b>
*Layer integrity (covering all layers is impossible)
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p>
**AR Mihai to get Brian’s supported layer list
<p>8.2 Devtool: enable kernel development.</p>
*Toaster-next branch
**what’s the easiest way to run the automated tests for a person.
**holy grail - make a single framework solution.
*Safari browser
*TBD - to check with DEV what it’s not covered yet.


= Schedule =


*Toaster is tested only at Full Pass by QA Team
<b><p> 9. Node.js dependencies installation. </p></b>
**the plan is to run it according to the Test execution Cycle, after automation is in place.
<p>9.1 Add Node.js modules. </p>
*Toaster is tested on toaster-next branch before every merge into Master by Dev team


*Toaster will be tested according to the [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Schedule YP 2.1 Schedule].
= Schedule =




Line 379: Line 253:
{| class="wikitable"
{| class="wikitable"
! Build
! Build
! Sanity Test/E2E
! Automated Test
! Weekly Test
! Manual Test
! Full Pass
|-
|-
| Daily Master
| Daily Master
| Y
| Y
| Y
| NO
| NO
|-
|-
| Weekly Build
| Weekly Build
| In progress
| Y
| Y
| Y
| NO
|-
|-
| Milestone Build
| Milestone Build
| Y
| Y
| Y
| Y
| Y
|}
|}
;Sanity Test is described [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Overall_Test_Plan#Sanity_Testing here].
*run on toaster-next or poky-contrib.
**TBD
***need info from DEV owners on what Sanity testing is done by them
***detail the set of tests. e.g. django unit tests, Toaster oe-selftest
Weekly test
*Weekly testing is performed on a weekly basis on the poky weekly build.
*E2E consists of all automated tests that can be run on Toaster
**oe-selftest toaster tests
*selenium UI tests
*TBD
**add E2E integration information
**add more information
Full Pass
*On full pass all available tests will be run:
**Automated tests:
***Django Unit Tests
***oe-selftest toaster tests
***Selenium automated UI tests
**Manual tests
***The non-automated tests from the Toaster test runs.
*Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.


=Dependencies=
=Dependencies=
*YP Dependencies
*YP Dependencies
**Bitbake works
** Devtool.
*Some changes in the Toaster UI might affect running the UI automated tests using Selenium
** poky-glibc script.
**e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
** toolchain
*recipe image we’re testing to get build data (metadata broken)
 


=Risk Assumptions=
=Risk Assumptions=
*TBD (AlexG)
* The complete features will not be tested now, as per the automation is in process now.
*Project Risks
**QA and DEV to define this
**Toaster doesn’t affect atm any other YP component (confirmed)
**Changes in Bitbake, bitbake logging might affect Toaster functionality
**Is reliant by host/browser environment
***TBD - add opensuse as well at some point
***TBD - expand the Toaster test framework to the supported host distributions of YP
*Testing Risks
**Toaster quality might be affected


=Tools=
=Tools=
*Refer to Requirements section
<p> Software Development Kit </p>
*TBD (need more detailed info of required tools)


= Release Criteria/ Exit Criteria =
= Release Criteria/ Exit Criteria =
Line 467: Line 300:


= Test Areas =
= Test Areas =
Toaster consists of two big components, as follows:
eSDK consists of two big components, as follows:


== Backend ==
== Backend ==


==== Functionality tests ====
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected through the simple UI;
Line 479: Line 312:


==== Usability tests ====
==== Usability tests ====
*  Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the toaster server])
*  Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])


== Frontend ==
== Frontend ==
Line 540: Line 373:


*; Objective:
*; Objective:
** Ensure functionality of Toaster component.
** Ensure functionality of eSDK component.


== Release Test ==
== Release Test ==

Latest revision as of 19:50, 26 September 2017

This article is the test plan for eSDK.

About eSDK

Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of OpenEmbedded build system. In order to Setting up the extensible SDK please go to Setting Up to Use the Extensible SDK.

Objectives

Verify all Extensible SDK components to be fully functional. Components to be verified:

Team members

QA Team involved in eSDK testing

Francisco Pedraza

Scope

eSDK
  • eSDK Installation.
  • eSDK Setup.

Devtool features to be tested:
  • Add a new recipe.
  • Modify the source for an existing recipe.
  • Upgrade an existing recipe.
  • Show work-space status.
  • Search available recipes.
  • Edit a recipe file in your work-space.
  • Get help on configure script options.
  • Build a recipe.
  • Apply changes from external source tree to recipe.
  • Remove a recipe from your work-space.
  • Deploy recipe output files to live target machine.
  • Un-deploy recipe output files in live target machine.
  • Build packages for a recipe.
  • Add Native Tools.
  • Build image including work-space recipe packages.
  • Run QEMU on the specified image.
  • Build a derivative SDK of this one.
  • Extract the source for an existing recipe.
  • Synchronize the source tree for an existing recipe.
  • Update SDK components.
  • Install additional SDK components.
  • Locating Pre-Built SDK Installers.

Node.js

  • Add Node.js Modules

Test Strategy

There are several test approaches for Extensible SDK, such as: Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule. Write new test cases based on developer request or new features as required. Maintain current test cases and update them accordingly the new changes. Perform exploratory testing on existing functionalities.


Test automation

Currently the execution of the cases is using "testsdkext" automation below are the three scenarios that should be tested

  • Test Building the eSDK installer:
  1. Clone poky and source the environment 
     $ source oe-init-build-env 

  2. Add proper configuration to local.conf
     $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf 
     $ echo 'INHERIT += "testsdk"' >> conf/local.conf

  3. Build eSDK intaller
     $ bitbake core-image-minimal  -c populate_sdk_ext -v 

  4. Execute the tests 
     $ bitbake core-image-minimal -c testsdkext

  5. All test should be PASSED


  • Test using Pre-Built SDK Installers:
  1. Under poky/ do another build folder different from the previous scenario 
     $ source oe-init-build-env build-pre-built

  2. Add proper configuration to local.conf
     $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf 
     $ echo 'INHERIT += "testsdk"' >> conf/local.conf

  3. Create sdk directory
     $ mkdir -p tmp/deploy/sdk

  4. Download artifacts form AB under new folder created 
     $ cd  tmp/deploy/sdk  
     $ wget <AB_LINK>/toolchain/x86_64/<ARTIFACTS_NEEDED>
        
         Example:
         $ wget https://autobuilder.yocto.io/pub/releases/yocto-2.4_M3.rc1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest   
         
         NOTE: All the artifacts that should be downloaded are:
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.target.manifest
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.sh
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.testdata.jason

   5. Give execution permissions to sh files. Rename files, only if it asks.
     $ chmod +x *.sh
     $ for file in *<RELEASE>*; do mv "$file" "${file/<RELEASE>/<RELEASE>+snapshot}"; done
       Example execute below command: 
        $ for file in *2.3*; do mv "$file" "${file/2.3/2.3+snapshot}"; done

   6. Go back to poky build folder "poky/build-pre-built"
     $ cd - 

   7. Prepare the environment to execute tests     
     $ bitbake libxml2

   8. Execute the test   
     $ bitbake core-image-minimal -c testsdkext

   9. All test should be PASSED

   NOTE: There are 4 combinations of tests that should be executed 
        - qemux86 + minimal --> (poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext*)
        - qemux86 + sato    --> (poky-glibc-x86_64-core-image-sato-i586-toolchain-ext*)
        - qemux86-64 + minimal  --> (poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext*)
        - qemux86-64 + sato     --> (poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext*)
  • Test eSDK as part of the oe-selftest suite
  1. Under poky/ do another build folder different from the previous scenario 
     $ source oe-init-build-env build-esdk-selftest

  2. Avoid Sanity Distro validation 
     $  echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf 

  3. Execute the tests 
     $ oe-selftest -r eSDK

  4. All test cases should be PASS

Test Approach

Sanity testing

  • Not covered at this moment
  • TBD following DEV discussions

Performance and Stress

  • Usability

Regression

  • Regression testing will be covered on Automation execution.

Maintaining the test cases

Submitting Bugs

Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.

8137 As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Test Case 1471

8136 As a developer I want to be able to generate target packages in SDK - Test Case 1473

8135 As a developer I want to be able to generate images out of binary feeds. - Test Case 1472

8136 As a developer I want the ability to Public eSDK with modifications/addons. - Test Case 1474

8974 As a developer I want to be able to create eSDK image benchmark. - Test Case 1475

8974 As a developer I want to be able to develop eSDK software benchmark. - Test Case 1476

8960 As a developer I want to be able to install node.js modules' in bitbake output. - Test Case 1477

8690 As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Test Case 1478

7635 As a developer I want to be able to extend cmake recipe creation on recipetool. - Test Case 1479

6658 8999As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Test Case 1480

8138As a developer I want to be able to store additional detailed information about builds history. - Test Case 1481

7634 As a developer I want to be able to extend autotools recipe creation. - Test Case 1482

9040 As a developer I want to be able to list all the content of bundles using Bitbake. - Test Case 1483

8982 As a developer I want to be able to add support out-of-tree kernel modules. - Test Case 1484

8975 As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Test Case 1485

6658 As a developer I want to be able to enable kernel development from devtool. - Test Case 1486

8883 As a developer I want to be able to update eSDK. - Test Case 1487

HW Requirements

  • Any machine able to build.

Software Requirements

  • eSDK.

Environment Requirements

    • YP 2.1 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2

Features

Features to be Tested

1. Beggining on a recipe.

1.1 add.

1.2 modify. Done in oe-selftest

1.3 upgrade. Done in oe-selftest


2. Getting Information.

2.1 Status.

2.2 Check.


3. Working on a recipe in the workspace.

3.1 edit-recipe.

3.2 configure-help.

3.3 build. - Done in oe-selftest

3.4 update recipe. - Done in oe-selftest

3.5 reset. - Done in oe-selftest


4. Testing changes on target.

4.1 deploy-target. - Done in oe-selftest

4.2 undeploy-target.

4.3 package.

4.4 build-image.- Done in oe-selftest

4.5 runqemu. - Done in oe-selftest


5. Advanced.

5.1 build-sdk.

5.2 extract. Done in oe-selftest

5.3 sync.


6. SDK maintenance.

6.1 sdk-update.

6.2 sdk-install.

7. sstate relevant features.

7.1 Ability to generate target packages in SDK.

7.2 Ability to install needed development libraries and headers from published package feeds.

8. Kernel relevant recipe and build.

8.1 Devtool: add: support out-of-tree kernel modules.

8.2 Devtool: enable kernel development.


9. Node.js dependencies installation.

9.1 Add Node.js modules.

Schedule

Test execution Cycle

Build Automated Test Manual Test
Daily Master Y NO
Weekly Build In progress Y
Milestone Build Y Y

Dependencies

  • YP Dependencies
    • Devtool.
    • poky-glibc script.
    • toolchain

Risk Assumptions

  • The complete features will not be tested now, as per the automation is in process now.

Tools

Software Development Kit

Release Criteria/ Exit Criteria

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&oldid=31947"