Kernel Development QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Category:Kernel Development QA]]
This article is the test plan for kernel development features in Yocto Project.
This article is the test plan for Kernel Development.


= About Kernel Development =
==About Kernel Development==
Describes common tasks you can perform on Yocto Project using the kernel tools, and shows you how to use the metadata required to work with the kernel.
For more information you can review  [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development.]


= Objectives =
Describes common tasks you can perform on Yocto Project using kernel tools, and shows you how to use the metadata required to work with the kernel.
Verify all kernel development components are fully functional. The approach is to use manual testing.
For more information you can review the [http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html Kernel Development manual].


= Team members =
==Objectives==
==QA Team involved in Kernel Development testing==
Verify all kernel development components are fully functional.


==Team members==
QA team members involved in Kernel Development testing:


  [mailto:jair.de.jesus.gonzalez.plascencia@intel.com Jair Gonzalez ]
  [mailto:jair.de.jesus.gonzalez.plascencia@intel.com Jair Gonzalez]


= Scope =
==Scope==
* linux-yocto-custom
* local source
* local source with parallel meta
* local source with recipe-space meta
* External source
* defconfig
* defconfig + fragments
* linux-yocto meta data + local fragments
* building external modules (hello-mod)


= Test Strategy =
===Types of Tests===
There are several test approaches for Kernel Development, such as:
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.
Write new test cases based on developer request or new features as required.
Maintain current test cases and update them accordingly the new changes.
Perform exploratory testing on existing functionalities.


* Manual tests on different platforms
* No automated tests at the moment


== Test automation ==
===Features Tested===
Not required until now.
: 1. linux-yocto-custom
== Test Approach ==
:: 1.1 local source
:: 1.2 local source with parallel meta
:: 1.3 local source with recipe-space meta
: 2. External Source
: 3. defconfig
: 4. defconfig + fragments
: 5. linux-yocto meta data + local fragments (WIP)
: 6. building external modules (hello-mod)


=== Sanity testing ===
The complete set of test cases are documented on the [[Kernel Development Test Cases]] wiki. They are also listed on the master Kernel test plan in Testopia, that can be reached following this [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 link].
* Not covered at this moment
* TBD following DEV discussions


==Test Strategy==
There are several test approaches for kernel development, such as:
* Perform test cases agreed upon development during periodic full pass test cycles, according to the schedule.
* Write new test cases based on developer requests or new features added, as required.
* Maintain current test cases and update them according to changes.
* Perform exploratory testing on existing functionalities.


== Maintaining the test cases ==
===Test automation===
== Submitting Bugs ==
Tests will be gradually automated, after determining the feasibility to do so following discussions with the development team.
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].


=Requirements=
===Sanity testing===
* Not covered at this moment
* TBD following discussions with the development team


This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.
===Test Process===


<p>As a Developer I want to be able to  generate and change  configuration files (defconfig). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1612 Test Case 1612]</p>
: 1. Follow the steps as indicated on the [[Kernel Development Test Cases]] wiki.
: 2. Verify that all test cases pass. If not, raise bugs properly.
: 3. Update results to corresponding Testopia test run.


<p>As a Developer I want to be able to generate and change  configuration files (defconfig + fragments). - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1613 Test Case 1613]</p>
===Submitting Bugs===
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Kernel Development bugs are no different than other Yocto Project bugs and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].


<p>As a Developer I want to be able to apply single patch to Linux Kernel Source.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1614 Test Case 1614]</p>
==Requirements==
Use cases are documented in the corresponding section for each test case on [[Kernel Development Test Cases]] wiki.


<p>As a Developer I want to to be able to work with your own sources.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1615 Test Case 1615]</p>
===HW Requirements===
 
<p>As a Developer I want to be able to work with own sources for linux-yocto-custom-local-source.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1616 Test Case 1616]</p>
 
<p>As a Developer I want to be able to work with recipe-space meta.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1617 Test Case 1617]</p>
 
<p>As a Developer I want to be able to work with External Source.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p>
 
<p>As a Developer I want to be able to configure linux-yocto meta data + local fragments.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1618 Test Case 1618]</p>
 
<p>As a Developer I want to be able to build external modules (hello-mod).  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1832 Test Case 1832]</p>
 
<p>As a Developer I want to be able to work with local source with parallel meta..  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1833 Test Case 1833]</p>
 
The complete test cases are not completed due current design.
 
==HW Requirements==
* Any machine able to build Yocto Project.
* Any machine able to build Yocto Project.


==Software Requirements==
===Software Requirements===
* Poky.
* Poky
* GNU/Linux environment
* GNU/Linux environment


==Environment Requirements==
===Environment Requirements===
**YP 2.3 Release:
*YP 2.3 Release:
**Ubuntu-14.04
*Ubuntu-14.04
**Ubuntu-14.10
*Ubuntu-14.10
**Ubuntu-15.04
*Ubuntu-15.04
**Fedora-21
*Fedora-21
**CentOS-6.*
*CentOS-6.*
**CentOS-7.*
*CentOS-7.*
**Debian-7.*
*Debian-7.*
**Debian-8.*
*Debian-8.*
**openSUSE-project-13.2
*openSUSE-project-13.2
 
=Features=
<b> Features to be Tested </b>
 
<b><p>1. linux-yocto-custom.</p></b>
<p>1.1 local source. </p>
<p>1.2 local source with parallel meta</p>
<p>1.3 local source with recipe-space meta</p>
 
 
<b><p>2. External Source.</p></b>


<b><p>3.Defconfig.</p></b>
==Schedule==
<b><p>4.Defconfig + Fragments.</p></b>
<b><p>5.building external modules (hello-mod).</p></b>
 
= Schedule =
Every cycle depending on Milestone and release candidate
Every cycle depending on Milestone and release candidate


==Test execution Cycle==
===Test execution Cycle===
{| class="wikitable"
{| class="wikitable"
! Build
! Build
Line 123: Line 95:
| Weekly Build
| Weekly Build
| TBD
| TBD
| Y
| TBD
|-
|-
| Milestone Build
| Milestone Build
Line 130: Line 102:
|}
|}


=Dependencies=
==Dependencies==
*YP Dependencies
*YP Dependencies
** poky
** poky
** meta-kerneltest recipe
** meta-kerneltest layer
 
 
=Risk Assumptions=
* For each change in kernel version and kernel cache, the team needs to change many configuration in order to achieve this.
* Just like any software, the kernel is also susceptible to bugs. A common “Long Term Support”.
 
=Tools=
<p> TBD </p>
 
= Release Criteria/ Exit Criteria =
 
 
 
 
<!--
THIS IS THE OLD TESTING PLAN
 
= Test Areas =
eSDK consists of two big components, as follows:
 
== Backend ==
 
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
 
==== Usability tests ====
*  Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])
 
== Frontend ==
 
==== Functionality tests ====
*  Manual testing in the first stage;
*  Automate testing using  [http://www.seleniumhq.org/ Selenium], in the second stage;
 
==== Compatibility tests ====
*  Verify the behavior of the GUI on different browsers and operating systems; TBD
 
==== Usability tests ====
*  Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
*  Friendly graphical user interface;
 
==== Performance tests ====
*  Stress testing (e.g. display appropriate error messages when the system is under stress);
 
= Test Cycle =
{|class="wikitable" style="text-align: center;"
| || || colspan="3" | Test execution cycle
|-
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]
|-
| rowspan="2" | Build type || Weekly  || Yes || Yes ||
|-
| Release || Yes || Yes || Yes
|-
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes
|-
| [[#Frontend]]  || Yes || Yes || Yes
|-
| rowspan="5" | Target machine || qemuarm || || Yes || Yes
|-
| qemumips|| || || Yes
|-
| qemuppc|| || || Yes
|-
| qemux86|| Yes || Yes || Yes
|-
| qemux86-64  ||  ||  || Yes
|-
| rowspan="3" | Target image|| core-image-minimal|| Yes || ||
|-
| core-image-sato-sdk|| || Yes || Yes
|}
 
== Weekly Test ==
*; Scope:
** Images built weekly and released through the distribution team.
 
*; Objective:
** Functionality test on most areas with minimum sets of tests;
** Regression test with high probability to find bugs.


== Full Pass Test ==
==Risk Assumptions==
*; Scope:
* For each change in kernel version and kernel cache, the team needs to change many configuration files in order to achieve a correct execution.
** Images built as candidates for milestone or final release;
* Just like any software, the kernel is also susceptible to bugs.
** Passed [[#Weekly Test]]


*; Objective:
==Tools==
** Ensure functionality of eSDK component.
* bitbake
* yocto-layer
* bitbake-layers
* any available text editor


== Release Test ==
==Release Criteria/ Exit Criteria==
*; Scope:
* All test cases pass.
** Release candidates that pass [[#Full Pass Test]]
* No blocking issues are found.


*; Objective:
==References==
** All scheduled features are covered, or rescheduled;
* http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html
** All relevant bugs are fixed and verified.
* http://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html
-->
Compendium of Yocto Project manuals, including the two above:
* https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html

Latest revision as of 21:35, 6 November 2017

This article is the test plan for kernel development features in Yocto Project.

About Kernel Development

Describes common tasks you can perform on Yocto Project using kernel tools, and shows you how to use the metadata required to work with the kernel. For more information you can review the Kernel Development manual.

Objectives

Verify all kernel development components are fully functional.

Team members

QA team members involved in Kernel Development testing: 

Jair Gonzalez

Scope

Types of Tests

  • Manual tests on different platforms
  • No automated tests at the moment

Features Tested

1. linux-yocto-custom
1.1 local source
1.2 local source with parallel meta
1.3 local source with recipe-space meta
2. External Source
3. defconfig
4. defconfig + fragments
5. linux-yocto meta data + local fragments (WIP)
6. building external modules (hello-mod)

The complete set of test cases are documented on the Kernel Development Test Cases wiki. They are also listed on the master Kernel test plan in Testopia, that can be reached following this link.

Test Strategy

There are several test approaches for kernel development, such as:

  • Perform test cases agreed upon development during periodic full pass test cycles, according to the schedule.
  • Write new test cases based on developer requests or new features added, as required.
  • Maintain current test cases and update them according to changes.
  • Perform exploratory testing on existing functionalities.

Test automation

Tests will be gradually automated, after determining the feasibility to do so following discussions with the development team.

Sanity testing

  • Not covered at this moment
  • TBD following discussions with the development team

Test Process

1. Follow the steps as indicated on the Kernel Development Test Cases wiki.
2. Verify that all test cases pass. If not, raise bugs properly.
3. Update results to corresponding Testopia test run.

Submitting Bugs

Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. Kernel Development bugs are no different than other Yocto Project bugs and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

Use cases are documented in the corresponding section for each test case on Kernel Development Test Cases wiki.

HW Requirements

  • Any machine able to build Yocto Project.

Software Requirements

  • Poky
  • GNU/Linux environment

Environment Requirements

  • YP 2.3 Release:
  • Ubuntu-14.04
  • Ubuntu-14.10
  • Ubuntu-15.04
  • Fedora-21
  • CentOS-6.*
  • CentOS-7.*
  • Debian-7.*
  • Debian-8.*
  • openSUSE-project-13.2

Schedule

Every cycle depending on Milestone and release candidate

Test execution Cycle

Build Automated Test Manual Test
Daily Master NO NO
Weekly Build TBD TBD
Milestone Build TBD Y

Dependencies

  • YP Dependencies
    • poky
    • meta-kerneltest layer

Risk Assumptions

  • For each change in kernel version and kernel cache, the team needs to change many configuration files in order to achieve a correct execution.
  • Just like any software, the kernel is also susceptible to bugs.

Tools

  • bitbake
  • yocto-layer
  • bitbake-layers
  • any available text editor

Release Criteria/ Exit Criteria

  • All test cases pass.
  • No blocking issues are found.

References

Compendium of Yocto Project manuals, including the two above:

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&oldid=32913"