https://wiki.yoctoproject.org/wiki/api.php?action=feedcontributions&feedformat=atom&user=Francisco+PedrazaYocto Project - User contributions [en]2026-05-12T11:23:19ZUser contributionsMediaWiki 1.39.5https://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26889Kernel Development QA2017-05-17T21:44:03Z<p>Francisco Pedraza: /* Objectives */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional. This approach is based from Manual Testing.<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
<p>As a Developer I want to be able to work with local source with parallel meta.. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.building external modules (hello-mod).</p></b><br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* For each change in kernel version and kernel cache, the team needs to change many configuration in order to achieve this.<br />
* Just like any software, the kernel is also susceptible to bugs. A common “Long Term Support”.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26888Kernel Development QA2017-05-17T21:42:05Z<p>Francisco Pedraza: /* Objectives */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
<p>As a Developer I want to be able to work with local source with parallel meta.. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.building external modules (hello-mod).</p></b><br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* For each change in kernel version and kernel cache, the team needs to change many configuration in order to achieve this.<br />
* Just like any software, the kernel is also susceptible to bugs. A common “Long Term Support”.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26886Kernel Development QA2017-05-17T21:41:27Z<p>Francisco Pedraza: /* Risk Assumptions */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
<p>As a Developer I want to be able to work with local source with parallel meta.. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.building external modules (hello-mod).</p></b><br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* For each change in kernel version and kernel cache, the team needs to change many configuration in order to achieve this.<br />
* Just like any software, the kernel is also susceptible to bugs. A common “Long Term Support”.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26885Kernel Development QA2017-05-17T21:36:45Z<p>Francisco Pedraza: /* Features */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
<p>As a Developer I want to be able to work with local source with parallel meta.. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.building external modules (hello-mod).</p></b><br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26881Kernel Development QA2017-05-17T20:28:59Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
<p>As a Developer I want to be able to work with local source with parallel meta.. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26880Kernel Development QA2017-05-17T20:28:02Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p> <br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p> <br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26879Kernel Development QA2017-05-17T20:18:40Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p> <br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26878Kernel Development QA2017-05-17T20:17:10Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p> <br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p> <br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26877Kernel Development QA2017-05-17T20:13:33Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p> <br />
<br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26876Kernel Development QA2017-05-17T20:08:52Z<p>Francisco Pedraza: </p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to work with local source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to work with recipe-space meta. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to work with External Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<p>As a Developer I want to be able to build external modules (hello-mod). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26605Kernel Development QA2017-05-04T20:20:47Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26604Kernel Development QA2017-05-04T20:20:05Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<h2>Cell that spans two columns:</h2><br />
<table style="width:100%"><br />
<tr><br />
<th>Name</th><br />
<th colspan="2">Telephone</th><br />
</tr><br />
<tr><br />
<td>Bill Gates</td><br />
<td>55577854</td><br />
<td>55577855</td><br />
</tr><br />
</table><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26603Kernel Development QA2017-05-04T20:19:54Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<style><br />
table, th, td {<br />
border: 1px solid black;<br />
border-collapse: collapse;<br />
}<br />
th, td {<br />
padding: 5px;<br />
text-align: left; <br />
}<br />
</style><br />
<h2>Cell that spans two columns:</h2><br />
<table style="width:100%"><br />
<tr><br />
<th>Name</th><br />
<th colspan="2">Telephone</th><br />
</tr><br />
<tr><br />
<td>Bill Gates</td><br />
<td>55577854</td><br />
<td>55577855</td><br />
</tr><br />
</table><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26602Kernel Development QA2017-05-04T20:19:32Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<style><br />
table, th, td {<br />
border: 1px solid black;<br />
border-collapse: collapse;<br />
}<br />
th, td {<br />
padding: 5px;<br />
text-align: left; <br />
}<br />
</style><br />
</head><br />
<body><br />
<br />
<h2>Cell that spans two columns:</h2><br />
<table style="width:100%"><br />
<tr><br />
<th>Name</th><br />
<th colspan="2">Telephone</th><br />
</tr><br />
<tr><br />
<td>Bill Gates</td><br />
<td>55577854</td><br />
<td>55577855</td><br />
</tr><br />
</table><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26601Kernel Development QA2017-05-04T20:18:49Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<!DOCTYPE html><br />
<html><br />
<head><br />
<style><br />
table, th, td {<br />
border: 1px solid black;<br />
border-collapse: collapse;<br />
}<br />
th, td {<br />
padding: 5px;<br />
text-align: left; <br />
}<br />
</style><br />
</head><br />
<body><br />
<br />
<h2>Cell that spans two columns:</h2><br />
<table style="width:100%"><br />
<tr><br />
<th>Name</th><br />
<th colspan="2">Telephone</th><br />
</tr><br />
<tr><br />
<td>Bill Gates</td><br />
<td>55577854</td><br />
<td>55577855</td><br />
</tr><br />
</table><br />
<br />
</body><br />
</html><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=26600Kernel Development QA2017-05-04T20:17:56Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<table style="width:100%"><br />
<tr><br />
<th>Firstname</th><br />
<th>Lastname</th> <br />
<th>Age</th><br />
</tr><br />
<tr><br />
<td>Jill</td><br />
<td>Smith</td> <br />
<td>50</td><br />
</tr><br />
<tr><br />
<td>Eve</td><br />
<td>Jackson</td> <br />
<td>94</td><br />
</tr><br />
</table><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23524Kernel Development QA2017-01-31T16:44:55Z<p>Francisco Pedraza: /* Test execution Cycle */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| TBD<br />
| Y<br />
|-<br />
| Milestone Build<br />
| TBD<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23481Kernel Development QA2017-01-30T21:36:37Z<p>Francisco Pedraza: /* About Kernel Development */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Describes common tasks you can perform using the kernel tools, and shows you how to use the kernel Metadata needed to work with the kernel inside the Yocto Project. <br />
For more information you can review [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23478Kernel Development QA2017-01-30T21:34:12Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
The complete test cases are not completed due current design.<br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23477Kernel Development QA2017-01-30T21:33:41Z<p>Francisco Pedraza: /* Test execution Cycle */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
Additional detailed information about builds history. - <br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| NO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23476Kernel Development QA2017-01-30T21:33:24Z<p>Francisco Pedraza: /* Test execution Cycle */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
Additional detailed information about builds history. - <br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| nO<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23475Kernel Development QA2017-01-30T21:32:53Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>As a Developer I want to be able to apply single patch to Linux Kernel Source. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p> <br />
<br />
<p>As a Developer I want to to be able to work with your own sources. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p><br />
<br />
<p>As a Developer I want to be able to generate and change configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p><br />
<br />
<br />
Additional detailed information about builds history. - <br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* Poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.3 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23474Kernel Development QA2017-01-30T21:26:14Z<p>Francisco Pedraza: /* Objectives */</p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p> additional detailed information about builds history. - <br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=23473Kernel Development QA2017-01-30T21:25:32Z<p>Francisco Pedraza: </p>
<hr />
<div>[[Category:Kernel Development QA]]<br />
This article is the test plan for Kernel Development.<br />
<br />
= About Kernel Development =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Kernel Development components to be fully functional.<br />
Components to be verified:<br />
<br />
* linux-yocto-custom<br />
* local source<br />
* local source with parallel meta<br />
* local source with recipe-space meta<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments<br />
* building external modules (hello-mod)<br />
<br />
= Team members =<br />
==QA Team involved in Kernel Development testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
* linux-yocto-custom<br />
* local source.<br />
* local source with parallel meta.<br />
* local source with recipe-space meta.<br />
* External Source<br />
* defconfig<br />
* defconfig + fragments<br />
* linux-yocto meta data + local fragments.<br />
* building external modules (hello-mod).<br />
<br />
<br />
<br />
<br />
= Test Strategy =<br />
There are several test approaches for Kernel Development, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
Not required until now.<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p> additional detailed information about builds history. - <br />
<br />
==HW Requirements==<br />
* Any machine able to build Yocto Project.<br />
<br />
==Software Requirements==<br />
* poky.<br />
* GNU/Linux environment<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. linux-yocto-custom.</p></b><br />
<p>1.1 local source. </p><br />
<p>1.2 local source with parallel meta</p><br />
<p>1.3 local source with recipe-space meta</p><br />
<br />
<br />
<b><p>2. External Source.</p></b><br />
<br />
<b><p>3.Defconfig.</p></b><br />
<b><p>4.Defconfig + Fragments.</p></b><br />
<b><p>5.linux-yocto meta data + local fragments.</p></b><br />
<b><p>6.building external modules (hello-mod).</p></b><br />
<br />
<br />
<br />
= Schedule =<br />
Every cycle depending on Milestone and release candidate<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** poky<br />
** meta-kerneltest recipe<br />
<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, the design of the test cases is under construction.<br />
<br />
=Tools=<br />
<p> TBD </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=22969Extensible SDK Test Plan (eSDK)2017-01-16T18:09:51Z<p>Francisco Pedraza: /* Tools */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
<br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p><br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, as per the automation is in process now.<br />
<br />
=Tools=<br />
<p> Software Development Kit </p><br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=22968Quality Assurance yocto project2017-01-16T18:05:52Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test<br />
TEST_SUITES = " auto" <br />
<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<p>NOTE: For QEMU images there are additional steps that need to be followed. They are detailed on</p><br />
<p>https://wiki.yoctoproject.org/wiki/Steps_for_Automated_Testing_on_QEMU</p><br />
<br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<p><strong>If you need more information about bug information and levels, please go to<br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels</strong></p><br />
<br />
<br />
<h2>'''Weekly Testing (automated)'''</h2><br />
<br />
<p>1. Genericx86 weekly tests:</p><br />
<p>a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)</p><br />
<p>b) make checkout on the poky commit from the mail (in command line):</p><br />
<p>- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"</p><br />
<p>- in poky directory checkout on the desired commit: "git checkout <commit-id>"</p><br />
<p>c) source the environment:</p><br />
<p>- in poky directory ( "cd ~/poky"): " source oe-init-build-env"</p><br />
<p>d) add in local. conf :</p><br />
<p>* local.conf path: ~/poky/build/conf</p><br />
<p>- at the end of file add:</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_TARGET_IP = "DUT ip"<br />
TEST_SERVER_IP = "workstation ip"<br />
</pre><br />
<p>- change the machine: MACHINE ??= "genericx86"</p><br />
<p>e) in ~poky/build directory, in command line, type the following commands:</p><br />
<p>- "bitbake rpm psplash" (wait for tasks to be executed)</p><br />
<p>- "bitbake package-index" (wait for tasks to be executed)</p><br />
<p> - after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:</p><br />
<p>"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)</p><br />
<p>- "bitbake core-image-sato-sdk -c testimage"</p><br />
<p>If having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<br />
<p>2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.</p><br />
<br />
<p>3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):</p><br />
<p>- the #1 steps are followed for lsb images</p><br />
<p>a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</p><br />
<p>* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)</p><br />
<br />
<p>b) the checkout step is the same as #1</p><br />
<p>c) same as #1</p><br />
<p>d) add in local. conf :</p><br />
<p>* local.conf path: ~/poky/build/conf</p><br />
<p>- at the end of file add:</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_TARGET_IP = "DUT ip"<br />
TEST_SERVER_IP = "workstation ip"<br />
</pre><br />
<p>- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")</p><br />
<p>- change the distro : DISTRO ?= "poky-lsb"</p><br />
<p>e) in command line (~poky/build directory) type the following commands:</p><br />
<p>- "bitbake rpm psplash" (wait for tasks to be executed)</p><br />
<p>- "bitbake package-index" (wait for tasks to be executed)</p><br />
<p>- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:</p><br />
<p>"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)</p><br />
<p>- "bitbake core-image-lsb-sdk -c testimage"</p><br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<pre>$ export http_proxy=http://proxy-chain.intel.com:911</pre><br />
<p>And check if you have internet connection, typing on terminal:</p><br />
<pre>$ wget www.google.com</pre><br />
<p>* There is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<br />
<p>- Make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- Run in command line the following command:</p><br />
<pre> ./compliance_test.py <milestone> <date>" </pre><br />
<p>- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<br />
<p>8) Get the logs from DUT:</p><br />
<br />
<p>- Results fulllog: /opt/ltp</p><br />
<p>- results.log: /opt/ltp/results</p><br />
<p>- results.fail: /opt/ltp/output</p><br />
<p>- Results posix: /opt/ltp/testcases/open_posix_testsuites</p><br />
<p>* 18 MB File: /var/opt/lsb/test/manager/results/x86.../x86....targz </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* It has like 18 Mb</p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
<p>pTest run steps:</p><br />
<br />
<p>1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p><br />
<p>* we test it on NUC with genericx86-64</p><br />
<br />
<p>2. Install the image on DUT (using legacy boot)</p><br />
<br />
<p>3. Boot the image</p><br />
<br />
<p>4. In command line type</p><br />
<pre>$ ptest-runner > ptest.log </pre><br />
<p>and wait for it to finish ( about 5 hours)</p><br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=22967Extensible SDK Test Plan (eSDK)2017-01-16T18:03:06Z<p>Francisco Pedraza: /* Scope */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
<br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p><br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, as per the automation is in process now.<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&diff=22014Kernel Development QA2016-12-13T22:27:23Z<p>Francisco Pedraza: Created page with "This is going to be the Kernel Development Quality Assurance wiki."</p>
<hr />
<div>This is going to be the Kernel Development Quality Assurance wiki.</div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20720Quality Assurance yocto project2016-10-19T15:27:44Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test<br />
TEST_SUITES = " auto" <br />
<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<pre>$ export http_proxy=http://proxy-chain.intel.com:911</pre><br />
<p>And check if you have internet connection, typing on terminal:<br />
<pre>$ wget www.google.com</pre><br />
<p>* There is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<br />
<p>- Make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- Run in command line the following command:<br />
<pre> ./compliance_test.py <milestone> <date>" </pre><br />
<p>- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<br />
<p>8) Get the logs from DUT:</p><br />
<p>- Result-<milestone>-data.fulllog</p><br />
<p>- Result-<milestone>-data.log</p><br />
<p>- Result-<milestone>-data.fail</p><br />
<p>- Posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* The three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* It has like 18 Mb</p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type <br />
<pre>$ ptest-runner > ptest.log </pre><br />
and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20594Quality Assurance yocto project2016-10-17T17:20:06Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<pre>$ export http_proxy=http://proxy-chain.intel.com:911</pre><br />
<p>And check if you have internet connection, typing on terminal:<br />
<pre>$ wget www.google.com</pre><br />
<p>* There is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<br />
<p>- Make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- Run in command line the following command:<br />
<pre> ./compliance_test.py <milestone> <date>" </pre><br />
<p>- Wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<br />
<p>8) Get the logs from DUT:</p><br />
<p>- Result-<milestone>-data.fulllog</p><br />
<p>- Result-<milestone>-data.log</p><br />
<p>- Result-<milestone>-data.fail</p><br />
<p>- Posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* The three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* It has like 18 Mb</p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<p>* Also I'll fwd an email to see how it looks</p><br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<p>*The script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E</p><br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type <br />
<pre>$ ptest-runner > ptest.log </pre><br />
and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20592Quality Assurance yocto project2016-10-17T17:14:33Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- Export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<p><pre>$ export http_proxy=http://proxy-chain.intel.com:911. </p></pre><br />
<p>And check if you have internet connection, typing on terminal:<br />
<p><pre>$ wget www.google.com</p></pre><br />
<p>* there is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working</p><br />
<p>6) Run the script like this:</p><br />
<p>- make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- run in command line the following command "./compliance_test.py <milestone> <date>"</p><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)</p><br />
<p>8) Get the logs from DUT:</p><br />
<p>- result-<milestone>-data.fulllog</p><br />
<p>- result-<milestone>-data.log</p><br />
<p>- result-<milestone>-data.fail</p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* it has like 18 Mb</p><br />
<p>* upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<p>* also I'll fwd an email to see how it looks</p><br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<p>*the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E</p><br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type <br />
<pre>$ ptest-runner > ptest.log </pre><br />
and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20587Quality Assurance yocto project2016-10-17T17:09:18Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
TEST_SUITES = " auto"<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- edit /etc/resolv.conf and add the gateway ip_address – yocto -> Broadcast de maquina, sudo rounte –n</p><br />
<p>- add the ip and netmask using "ifconfig" command - yocto</p><br />
<p>- add the route using "route add default gw <ip_address>" -> broadcast de maquina</p><br />
<p>- export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<p>Export http_proxy=http://proxy-chain.intel.com:911. And check if you have internet connection. With $wget www.google.com</p><br />
<p>* there is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working</p><br />
<p>6) Run the script like this:</p><br />
<p>- make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- run in command line the following command "./compliance_test.py <milestone> <date>"</p><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)</p><br />
<p>8) Get the logs from DUT:</p><br />
<p>- result-<milestone>-data.fulllog</p><br />
<p>- result-<milestone>-data.log</p><br />
<p>- result-<milestone>-data.fail</p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* it has like 18 Mb</p><br />
<p>* upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<p>* also I'll fwd an email to see how it looks</p><br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<p>*the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E</p><br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type $ ptest-runner > ptest.log and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20586Quality Assurance yocto project2016-10-17T17:07:53Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
TEST_SUITES = " auto"<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<h2>'''pTest and runtime'''</h2><br />
<br />
<p>1. pTest run steps:</p><br />
<p>1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p><br />
* Usually tested on NUC with genericx86-64 <br />
<p>2) Install the image on DUT (using legacy boot)</p><br />
<p>3) Boot the image and copy "ptest-runner.sh" script on DUT</p><br />
<p>4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)</p><br />
<p>* P.S. : I will send you also the script via google drive, so you can open it</p><br />
<br />
<br />
<p>2. Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT </p><br />
<p>3) Configure the network so it be able to work externally: </p><br />
<p> - edit /etc/resolv.conf and add the gateway ip_address </p><br />
<p> - add the ip and netmask using "ifconfig" command </p> <br />
<p> - add the route using "route add default gw <ip_address>" </p><br />
<p> - export the proxy using "export http_proxy=<add your proxy link>" command </p><br />
<p> * there is a bug and if you make these steps in another order than above, it may be possible not work </p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working </p><br />
<p>6) Run the script like this: </p><br />
<p>- make the script executable:</p><br />
<pre>"$ chmod a+x compliance_local.py" </pre><br />
<p>- run in command line the following command " </p><br />
<pre>./compliance_test.py <milestone> <date>" </pre><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours) </p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day) </p><br />
<p>8) Get the logs from DUT: </p><br />
<p>- result-<milestone>-data.fulllog </p><br />
<p>- result-<milestone>-data.log </p><br />
<p>- result-<milestone>-data.fail </p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) </p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image </p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz </p><br />
<p>* It has like 18 Mb </p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com </p><br />
<br />
<p>9) After this, update the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> '''Compliance Testing'''</h2><br />
<br />
<p>Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- edit /etc/resolv.conf and add the gateway ip_address – yocto -> Broadcast de maquina, sudo rounte –n</p><br />
<p>- add the ip and netmask using "ifconfig" command - yocto</p><br />
<p>- add the route using "route add default gw <ip_address>" -> broadcast de maquina</p><br />
<p>- export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<p>Export http_proxy=http://proxy-chain.intel.com:911. And check if you have internet connection. With $wget www.google.com</p><br />
<p>* there is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working</p><br />
<p>6) Run the script like this:</p><br />
<p>- make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- run in command line the following command "./compliance_test.py <milestone> <date>"</p><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)</p><br />
<p>8) Get the logs from DUT:</p><br />
<p>- result-<milestone>-data.fulllog</p><br />
<p>- result-<milestone>-data.log</p><br />
<p>- result-<milestone>-data.fail</p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* it has like 18 Mb</p><br />
<p>* upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<p>* also I'll fwd an email to see how it looks</p><br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<p>*the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E</p><br />
<br />
<br />
<h2>''' pTest Run'''</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type $ ptest-runner > ptest.log and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20582Quality Assurance yocto project2016-10-17T17:03:14Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
TEST_SUITES = " auto"<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<h2>'''pTest and runtime'''</h2><br />
<br />
<p>1. pTest run steps:</p><br />
<p>1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p><br />
* Usually tested on NUC with genericx86-64 <br />
<p>2) Install the image on DUT (using legacy boot)</p><br />
<p>3) Boot the image and copy "ptest-runner.sh" script on DUT</p><br />
<p>4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)</p><br />
<p>* P.S. : I will send you also the script via google drive, so you can open it</p><br />
<br />
<br />
<p>2. Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT </p><br />
<p>3) Configure the network so it be able to work externally: </p><br />
<p> - edit /etc/resolv.conf and add the gateway ip_address </p><br />
<p> - add the ip and netmask using "ifconfig" command </p> <br />
<p> - add the route using "route add default gw <ip_address>" </p><br />
<p> - export the proxy using "export http_proxy=<add your proxy link>" command </p><br />
<p> * there is a bug and if you make these steps in another order than above, it may be possible not work </p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working </p><br />
<p>6) Run the script like this: </p><br />
<p>- make the script executable:</p><br />
<pre>"$ chmod a+x compliance_local.py" </pre><br />
<p>- run in command line the following command " </p><br />
<pre>./compliance_test.py <milestone> <date>" </pre><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours) </p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day) </p><br />
<p>8) Get the logs from DUT: </p><br />
<p>- result-<milestone>-data.fulllog </p><br />
<p>- result-<milestone>-data.log </p><br />
<p>- result-<milestone>-data.fail </p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) </p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image </p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz </p><br />
<p>* It has like 18 Mb </p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com </p><br />
<br />
<p>9) After this, update the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> Compliance Testing</h2><br />
<br />
Runtime compliance steps:<br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT</p><br />
<p>3) Configure the network so it be able to work externally:</p><br />
<p>- edit /etc/resolv.conf and add the gateway ip_address – yocto -> Broadcast de maquina, sudo rounte –n</p><br />
<p>- add the ip and netmask using "ifconfig" command - yocto</p><br />
<p>- add the route using "route add default gw <ip_address>" -> broadcast de maquina</p><br />
<p>- export the proxy using "export http_proxy=<add your proxy link>" command eg</p><br />
<p>Export http_proxy=http://proxy-chain.intel.com:911. And check if you have internet connection. With $wget www.google.com</p><br />
<p>* there is a bug and if you make these steps in another order than above, it may be possible not work</p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working</p><br />
<p>6) Run the script like this:</p><br />
<p>- make the script executable: "chmod a+x compliance_local.py"</p><br />
<p>- run in command line the following command "./compliance_test.py <milestone> <date>"</p><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)</p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)</p><br />
<p>8) Get the logs from DUT:</p><br />
<p>- result-<milestone>-data.fulllog</p><br />
<p>- result-<milestone>-data.log</p><br />
<p>- result-<milestone>-data.fail</p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)</p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image</p><br />
<br />
<p>- in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)</p><br />
<p>* it has like 18 Mb</p><br />
<p>* upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com</p><br />
<p>* also I'll fwd an email to see how it looks</p><br />
<p>9) Put the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<p>*the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E</p><br />
<br />
<br />
<h2> pTest Run</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type $ ptest-runner > ptest.log and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Quality_Assurance_yocto_project&diff=20578Quality Assurance yocto project2016-10-17T16:58:05Z<p>Francisco Pedraza: </p>
<hr />
<div><h2>'''Steps for BSP Automated Testing'''</h2><br />
<p><br />
1. Download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
</p><br />
<p><br />
2. Make checkout on the poky commit from the mail (in command line):<br />
</p><br />
a) Clone a clean version of poky: <br />
<pre>$ git clone git://git.yoctoproject.org/poky"</pre><br />
b)In poky directory checkout on the desired commit:<br />
<pre>$ git checkout <commit-id></pre><br />
<p><br />
3. Source the environment, from poky directory <pre> $ cd ~/poky </pre><br />
</p><br />
<br />
<pre>$ source oe-init-build-env</pre><br />
<p><br />
4. Edit local.conf. PATH: ~/poky/build/conf, at the en of the file adding: <br />
</p><br />
<pre><br />
INHERIT += "testimage"<br />
TEST_TARGET = "simpleremote"<br />
TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests<br />
TEST_TARGET_IP = "DUT ip" --> IP of the device under test <br />
TEST_SUITES = " auto"<br />
Remember to add the machine for example, if genericx86 is needed, MACHINE ?= "genericx86"<br />
<br />
<br />
</pre><br />
<br />
<p><br />
5. Edit bblayers.conf, bblayers.conf PATH:~/poky/build/conf<br />
</p><br />
<ul><li>in " BBLAYERS ?= " \ " section add "meta-selftest" respecting the pattern from above</li></ul><br />
<p><br />
<br />
6. In ~poky/build directory, in command line, type the following commands:<br />
</p><br />
<pre><br />
$ bitbake rpm psplash<br />
$ bitbake package-index<br />
</pre><br />
<p><br />
NOTE: After finishing the tasks, copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
</p><br />
<pre>$ bitbake core-image-sato-sdk -c testimage</pre><br />
<p>If you face issues with the keys from your host, use this command: $ ssh-keygen -f /home/user/.ssh/known_hosts -R targetip</p><br />
<p>Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET</p><br />
<p>After a while the testing will be finished</p><br />
<br />
<h2>'''Steps for LSB Automated Testing'''</h2><br />
<p><br />
1. In general the steps are very similar to BSP, just make sure to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
</p><br />
<p><br />
2. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
</p> <br />
<ul><li>The #1 steps are followed for lsb images</li></ul><br />
<ul><li>download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)</li></ul><br />
NOTE: the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<p>3. The checkout step is the same as BSP.</p><br />
<p>4. Edit local.conf, adding the same configuration as BSP. Just one small changing the distro</p> <br />
<ul><li>DISTRO ?= "poky-lsb"</li></ul><br />
NOTE: The bblayers configuration is the same as BSP.<br />
<p>From command line type the following commands</p><br />
<pre>$ bitbake rpm psplash (wait for tasks to be executed)</pre><br />
<pre>$ bitbake package-index (wait for tasks to be executed)</pre><br />
NOTE: After finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
<p>Type the following command from terminal:</p><br />
<pre> $ bitbake core-image-lsb-sdk -c testimage </pre><br />
<br />
<h2>'''pTest and runtime'''</h2><br />
<br />
<p>1. pTest run steps:</p><br />
<p>1) Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)</p><br />
* Usually tested on NUC with genericx86-64 <br />
<p>2) Install the image on DUT (using legacy boot)</p><br />
<p>3) Boot the image and copy "ptest-runner.sh" script on DUT</p><br />
<p>4) In command line run "ptest-runner.sh > ptest.log" and wait for it to finish ( about 5 hours)</p><br />
<p>* P.S. : I will send you also the script via google drive, so you can open it</p><br />
<br />
<br />
<p>2. Runtime compliance steps:</p><br />
<p>1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)</p><br />
<p>* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk</p><br />
<p>2) Install the image on DUT </p><br />
<p>3) Configure the network so it be able to work externally: </p><br />
<p> - edit /etc/resolv.conf and add the gateway ip_address </p><br />
<p> - add the ip and netmask using "ifconfig" command </p> <br />
<p> - add the route using "route add default gw <ip_address>" </p><br />
<p> - export the proxy using "export http_proxy=<add your proxy link>" command </p><br />
<p> * there is a bug and if you make these steps in another order than above, it may be possible not work </p><br />
<p>4) Copy "compliance_test.py" script on DUT</p><br />
<p>5) Make sure that your network connection is working </p><br />
<p>6) Run the script like this: </p><br />
<p>- make the script executable:</p><br />
<pre>"$ chmod a+x compliance_local.py" </pre><br />
<p>- run in command line the following command " </p><br />
<pre>./compliance_test.py <milestone> <date>" </pre><br />
<p>- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours) </p><br />
<p>7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day) </p><br />
<p>8) Get the logs from DUT: </p><br />
<p>- result-<milestone>-data.fulllog </p><br />
<p>- result-<milestone>-data.log </p><br />
<p>- result-<milestone>-data.fail </p><br />
<p>- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite) </p><br />
<p>* the three others are found in /opt/ltp directory, in output, temp, result folders </p><br />
<p>* send these log to Yi Zhao specifying the version and the type of image </p><br />
<br />
<p>- In /var/opt/lsb/test/manager/results/x86.../x86....tar.gz </p><br />
<p>* It has like 18 Mb </p><br />
<p>* Upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com </p><br />
<br />
<p>9) After this, update the tests from Testopia - Runtime test run on passed </p><br />
<br />
<br />
<h2>'''Defect/Bug life Cycle'''</h2><br />
----<br />
[[File:Yp_BugLifeCycle.png]]<br />
<br />
<h3>If you need more information about bug information and levels, please go to </h3><br />
https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels<br />
<br />
<h2> Weekly Testing (automated)</h2><br />
<br />
1. Genericx86 weekly tests:<br />
a) download the core-image-sato-sdk image from autobuilder( link available in the point release mail)<br />
b) make checkout on the poky commit from the mail (in command line):<br />
- clone a clean version of poky: "git clone git://git.yoctoproject.org/poky"<br />
- in poky directory checkout on the desired commit: "git checkout <commit-id>"<br />
c) source the environment:<br />
- in poky directory ( "cd ~/poky"): " source oe-init-build-env"<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86"<br />
e) add in bblayers.conf:<br />
* bblayers.conf path: ~/poky/build/conf<br />
- in " BBLAYERS ?= " \ " section add meta-selftest respecting the pattern from above<br />
f) in ~poky/build directory, in command line, type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-sato-sdk-genericx86.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/genericx86/" ( if "/tmp/deploy/images/genericx86" does not exist create new directories with this path)<br />
- "bitbake core-image-sato-sdk -c testimage"<br />
if having issues with keys use this command: ssh-keygen -f /home/user/.ssh/known_hosts -R targetip<br />
Then add the key to the host again ssh , SSH TARGETNAME@IPTARGET<br />
<br />
2. Use the same steps as above for Genericx86-64 BSP but take care to download the correct image and modify accordingly the machine in local.conf and path for manifest.<br />
<br />
3. Test generic weekly LSB images (genericx86-lsb and genericx86-64-lsb):<br />
- the #1 steps are followed for lsb images<br />
a) download the core-image-lsb-sdk image from autobuilder( link available in the point release mail)<br />
* the images are available in sato and lsb form( access genericx86-lsb and genericx86-64-lsb directories for lsb images)<br />
<br />
b) the checkout step is the same as #1<br />
c) same as #1<br />
d) add in local. conf :<br />
* local.conf path: ~/poky/build/conf<br />
- at the end of file add: <br />
INHERIT += "testimage"<br />
<br />
TEST_TARGET = "simpleremote"<br />
<br />
TEST_TARGET_IP = "DUT ip"<br />
<br />
TEST_SERVER_IP = "workstation ip"<br />
<br />
TEST_SUITES = " auto"<br />
- change the machine: MACHINE ??= "genericx86" (or "genericx86-64")<br />
- change the distro : DISTRO ?= "poky-lsb"<br />
e) in bblayers.conf add meta-selftest as in #1<br />
f) in command line (~poky/build directory) type the following commands:<br />
- "bitbake rpm psplash" (wait for tasks to be executed)<br />
- "bitbake package-index" (wait for tasks to be executed)<br />
- after finishing the tasks copy the image manifest (core-image-lsb-sdk-machine.manifest) from autobuilder at the following path:<br />
"~poky/build/tmp/deploy/images/<machine_name>-lsb/" ( if "/tmp/deploy/images/<machine_name>-lsb" does not exist create new directories with this path)<br />
- "bitbake core-image-lsb-sdk -c testimage"<br />
<br />
<br />
<h2> Compliance Testing</h2><br />
<br />
Runtime compliance steps:<br />
1) Download lsb image from autobuilder( same image as in LSB weekly testing for genericx86-64-lsb bsp)<br />
* we test compliance on NUC with genericx86-64-lsb, core-image-lsb-sdk<br />
2) Install the image on DUT<br />
3) Configure the network so it be able to work externally:<br />
- edit /etc/resolv.conf and add the gateway ip_address – yocto -> Broadcast de maquina, sudo rounte –n<br />
- add the ip and netmask using "ifconfig" command - yocto<br />
- add the route using "route add default gw <ip_address>" -> broadcast de maquina<br />
- export the proxy using "export http_proxy=<add your proxy link>" command eg<br />
Export http_proxy=http://proxy-chain.intel.com:911. And check if you have internet connection. With $wget www.google.com<br />
* there is a bug and if you make these steps in another order than above, it may be possible not work<br />
4) Copy "compliance_test.py" script on DUT<br />
5) Make sure that your network connection is working<br />
6) Run the script like this:<br />
- make the script executable: "chmod a+x compliance_local.py"<br />
- run in command line the following command "./compliance_test.py <milestone> <date>"<br />
- wait until "Configuration done. LSB script must be started from machine." in command line( about 8-12 hours)<br />
7) Run "LSB_test.sh" via ssh or manually and wait for it to finish( about a day)<br />
8) Get the logs from DUT:<br />
- result-<milestone>-data.fulllog<br />
- result-<milestone>-data.log<br />
- result-<milestone>-data.fail<br />
- posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)<br />
* the three others are found in /opt/ltp directory, in output, temp, result folders <br />
* send these log to Yi Zhao specifying the version and the type of image<br />
* I'll fwd you an email so you could see an example<br />
- in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)<br />
* it has like 18 Mb<br />
* upload this file on drive and send the link to hongxu.jia@windriver.com and mark.hatle@windriver.com<br />
* also I'll fwd an email to see how it looks<br />
9) Put the tests from Testopia - Runtime test run on passed <br />
<br />
<br />
*the script can be found at this link: https://drive.google.com/open?id=0B4B8zTi4deOyNjF6NkM0cW01S0E<br />
<br />
<br />
<h2> pTest Run</h2><br />
<br />
pTest run steps:<br />
<br />
1. Download pTest image from autobuilder( you can find core-image-sato-sdk image in pTest directory)<br />
* we test it on NUC with genericx86-64 <br />
<br />
2. Install the image on DUT (using legacy boot) <br />
<br />
3. Boot the image<br />
<br />
4. In command line type $ ptest-runner > ptest.log and wait for it to finish ( about 5 hours)<br />
<p>Note: The "script" is already placed on the distro, you just have to type ptest...TAB and the terminal should autocomplete the command.</p><br />
<p> "> ptest.log", in order to save the test results in that file. </p></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Https://wiki.yoctoproject.org/wiki/QA_sanity_history_test&diff=19998Https://wiki.yoctoproject.org/wiki/QA sanity history test2016-08-25T15:57:05Z<p>Francisco Pedraza: </p>
<hr />
<div>Tests for ptest - QA sanity history</div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Https://wiki.yoctoproject.org/wiki/QA_sanity_history_test&diff=19996Https://wiki.yoctoproject.org/wiki/QA sanity history test2016-08-25T15:55:43Z<p>Francisco Pedraza: The purpose of this page is to test some functionalities for ptest uploader</p>
<hr />
<div>Tests for ptest</div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19183Extensible SDK Test Plan (eSDK)2016-06-23T16:20:38Z<p>Francisco Pedraza: /* Risk Assumptions */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p><br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
* The complete features will not be tested now, as per the automation is in process now.<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19182Extensible SDK Test Plan (eSDK)2016-06-23T16:19:17Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p><br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19181Extensible SDK Test Plan (eSDK)2016-06-23T16:16:48Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19180Extensible SDK Test Plan (eSDK)2016-06-23T16:06:18Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19179Extensible SDK Test Plan (eSDK)2016-06-23T15:59:40Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472] </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19178Extensible SDK Test Plan (eSDK)2016-06-23T15:38:53Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19177Extensible SDK Test Plan (eSDK)2016-06-23T15:37:47Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471] Test Case 1471 </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19176Extensible SDK Test Plan (eSDK)2016-06-23T15:36:19Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - 1471 [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - </p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19175Extensible SDK Test Plan (eSDK)2016-06-23T15:35:46Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - 1471 [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471]</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19156Extensible SDK Test Plan (eSDK)2016-06-20T16:55:55Z<p>Francisco Pedraza: /* Requirements */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19117Extensible SDK Test Plan (eSDK)2016-06-16T15:08:30Z<p>Francisco Pedraza: /* Features */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 Add Node.js modules. </p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19116Extensible SDK Test Plan (eSDK)2016-06-16T15:06:57Z<p>Francisco Pedraza: /* Test execution Cycle */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 TBD. </p><br />
<p>9.2 TBD.</p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| Y<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedrazahttps://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&diff=19115Extensible SDK Test Plan (eSDK)2016-06-16T15:06:37Z<p>Francisco Pedraza: /* Test execution Cycle */</p>
<hr />
<div>[[Category:eSDK]]<br />
This article is the test plan for eSDK.<br />
<br />
= About eSDK =<br />
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]<br />
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]<br />
<br />
= Objectives =<br />
Verify all Extensible SDK components to be fully functional.<br />
Components to be verified:<br />
<br />
= Team members =<br />
==QA Team involved in eSDK testing==<br />
<br />
<br />
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]<br />
<br />
= Scope =<br />
<b><p>eSDK</p></b><br />
* eSDK Installation.<br />
* eSDK Setup.<br />
<b><p>Devtool features to be tested:</p></b><br />
* Add a new recipe.<br />
* Modify the source for an existing recipe.<br />
* Upgrade an existing recipe.<br />
* Show work-space status.<br />
* Search available recipes.<br />
* Edit a recipe file in your work-space.<br />
* Get help on configure script options.<br />
* Build a recipe.<br />
* Apply changes from external source tree to recipe.<br />
* Remove a recipe from your work-space.<br />
* Deploy recipe output files to live target machine.<br />
* Un-deploy recipe output files in live target machine.<br />
* Build packages for a recipe.<br />
* Add Native Tools.<br />
* Build image including work-space recipe packages.<br />
* Run QEMU on the specified image.<br />
* Build a derivative SDK of this one.<br />
* Extract the source for an existing recipe.<br />
* Synchronize the source tree for an existing recipe.<br />
* Update SDK components.<br />
* Install additional SDK components.<br />
* Locating Pre-Built SDK Installers.<br />
* <br />
<p><b>Node.js</b></p><br />
* Add Node.js Modules<br />
<br />
= Test Strategy =<br />
There are several test approaches for Extensible SDK, such as:<br />
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.<br />
Write new test cases based on developer request or new features as required.<br />
Maintain current test cases and update them accordingly the new changes.<br />
Perform exploratory testing on existing functionalities.<br />
<br />
<br />
== Test automation ==<br />
<br />
== Test Approach ==<br />
<br />
=== Sanity testing ===<br />
* Not covered at this moment<br />
* TBD following DEV discussions<br />
<br />
=== Performance and Stress ===<br />
* Usability<br />
<br />
===Regression===<br />
* Regression testing will be covered on Automation execution.<br />
<br />
== Maintaining the test cases ==<br />
== Submitting Bugs ==<br />
Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. <br />
eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].<br />
<br />
=Requirements=<br />
<br />
This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.<br />
<br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8960] As a developer I want to be able to install node.js modules' in bitbake output. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Pending TC</p><br />
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - Pending TC</p><br />
<br />
==HW Requirements==<br />
* Any machine able to build.<br />
<br />
==Software Requirements==<br />
* eSDK.<br />
<br />
==Environment Requirements==<br />
**YP 2.1 Release:<br />
**Ubuntu-14.04<br />
**Ubuntu-14.10<br />
**Ubuntu-15.04<br />
**Fedora-21<br />
**CentOS-6.*<br />
**CentOS-7.*<br />
**Debian-7.*<br />
**Debian-8.*<br />
**openSUSE-project-13.2<br />
<br />
=Features=<br />
<b> Features to be Tested </b><br />
<br />
<b><p>1. Beggining on a recipe.</p></b><br />
<p>1.1 add. </p><br />
<p>1.2 modify. Done in oe-selftest</p><br />
<p>1.3 upgrade. Done in oe-selftest</p><br />
<br />
<br />
<b><p>2. Getting Information.</p></b><br />
<p>2.1 Status.</p><br />
<p>2.2 Check.</p><br />
<br />
<br />
<b><p>3. Working on a recipe in the workspace.</p></b><br />
<p>3.1 edit-recipe.</p><br />
<p>3.2 configure-help.</p><br />
<p>3.3 build. - Done in oe-selftest</p><br />
<p>3.4 update recipe. - Done in oe-selftest</p><br />
<p>3.5 reset. - Done in oe-selftest</p><br />
<br />
<br />
<br />
<b><p>4. Testing changes on target.</p></b><br />
<p>4.1 deploy-target. - Done in oe-selftest</p><br />
<p>4.2 undeploy-target.</p><br />
<p>4.3 package.</p><br />
<p>4.4 build-image.- Done in oe-selftest</p><br />
<p>4.5 runqemu. - Done in oe-selftest </p><br />
<br />
<br />
<b><p>5. Advanced.</p></b><br />
<p>5.1 build-sdk.</p><br />
<p>5.2 extract. Done in oe-selftest</p><br />
<p>5.3 sync.</p><br />
<br />
<br />
<b><p>6. SDK maintenance.</p></b><br />
<p>6.1 sdk-update.</p><br />
<p>6.2 sdk-install.</p><br />
<br />
<b><p> 7. sstate relevant features. </p></b><br />
<p>7.1 Ability to generate target packages in SDK. </p><br />
<p>7.2 Ability to install needed development libraries and headers from published package feeds.</p><br />
<br />
<b><p> 8. Kernel relevant recipe and build. </p></b><br />
<p>8.1 Devtool: add: support out-of-tree kernel modules. </p><br />
<p>8.2 Devtool: enable kernel development.</p><br />
<br />
<br />
<b><p> 9. Node.js dependencies installation. </p></b><br />
<p>9.1 TBD. </p><br />
<p>9.2 TBD.</p><br />
<br />
= Schedule =<br />
<br />
<br />
==Test execution Cycle==<br />
{| class="wikitable"<br />
! Build<br />
! Automated Test<br />
! Manual Test<br />
|-<br />
| Daily Master<br />
| Y<br />
| NO<br />
|-<br />
| Weekly Build<br />
| In progress<br />
| In progress<br />
|-<br />
| Milestone Build<br />
| Y<br />
| Y<br />
|}<br />
<br />
=Dependencies=<br />
*YP Dependencies<br />
** Devtool.<br />
** poky-glibc script.<br />
** toolchain<br />
<br />
=Risk Assumptions=<br />
<br />
<br />
=Tools=<br />
<br />
<br />
= Release Criteria/ Exit Criteria =<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<!--<br />
THIS IS THE OLD TESTING PLAN <br />
<br />
= Test Areas =<br />
eSDK consists of two big components, as follows:<br />
<br />
== Backend ==<br />
<br />
==== Functionality tests ====<br />
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];<br />
* Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);<br />
* Verify that all links in the simple UI are available;<br />
* Verify the quality of the data collected through the simple UI;<br />
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;<br />
<br />
==== Usability tests ====<br />
* Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])<br />
<br />
== Frontend ==<br />
<br />
==== Functionality tests ====<br />
* Manual testing in the first stage;<br />
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage;<br />
<br />
==== Compatibility tests ====<br />
* Verify the behavior of the GUI on different browsers and operating systems; TBD<br />
<br />
==== Usability tests ==== <br />
* Verify if the GUI design is as described here: http://yoctoproject.org/webhob;<br />
* Friendly graphical user interface;<br />
<br />
==== Performance tests ====<br />
* Stress testing (e.g. display appropriate error messages when the system is under stress);<br />
<br />
= Test Cycle =<br />
{|class="wikitable" style="text-align: center;"<br />
| || || colspan="3" | Test execution cycle<br />
|-<br />
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]<br />
|-<br />
| rowspan="2" | Build type || Weekly || Yes || Yes ||<br />
|-<br />
| Release || Yes || Yes || Yes<br />
|-<br />
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes <br />
|-<br />
| [[#Frontend]] || Yes || Yes || Yes<br />
|-<br />
| rowspan="5" | Target machine || qemuarm || || Yes || Yes<br />
|-<br />
| qemumips|| || || Yes<br />
|-<br />
| qemuppc|| || || Yes<br />
|-<br />
| qemux86|| Yes || Yes || Yes<br />
|-<br />
| qemux86-64 || || || Yes<br />
|-<br />
| rowspan="3" | Target image|| core-image-minimal|| Yes || || <br />
|-<br />
| core-image-sato-sdk|| || Yes || Yes<br />
|}<br />
<br />
== Weekly Test ==<br />
*; Scope:<br />
** Images built weekly and released through the distribution team.<br />
<br />
*; Objective:<br />
** Functionality test on most areas with minimum sets of tests;<br />
** Regression test with high probability to find bugs.<br />
<br />
== Full Pass Test ==<br />
*; Scope:<br />
** Images built as candidates for milestone or final release;<br />
** Passed [[#Weekly Test]]<br />
<br />
*; Objective:<br />
** Ensure functionality of eSDK component.<br />
<br />
== Release Test ==<br />
*; Scope:<br />
** Release candidates that pass [[#Full Pass Test]]<br />
<br />
*; Objective:<br />
** All scheduled features are covered, or rescheduled;<br />
** All relevant bugs are fixed and verified.<br />
--></div>Francisco Pedraza