Kernel Development QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with "This is going to be the Kernel Development Quality Assurance wiki.")
 
No edit summary
Line 1: Line 1:
This is going to be the Kernel Development Quality Assurance wiki.
[[Category:Kernel Development QA]]
This article is the test plan for  Kernel Development.
 
= About Kernel Development =
Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of [http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#build-system-term OpenEmbedded build system.]
In order to Setting up the extensible SDK please go to [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-setting-up-to-use-the-extensible-sdk Setting Up to Use the Extensible SDK.]
 
= Objectives =
Verify all Kernel Development components to be fully functional.
Components to be verified:
 
* linux-yocto-custom
  * local source
  * local source with parallel meta
  * local source with recipe-space meta
* External Source
* defconfig
* defconfig + fragments
* linux-yocto meta data + local fragments
* building external modules (hello-mod)
 
= Team members =
==QA Team involved in Kernel Development testing==
 
 
[mailto:francisco.j.pedraza.gonzalez@intel.com Francisco Pedraza ]
 
= Scope =
* linux-yocto-custom
* local source.
* local source with parallel meta.
* local source with recipe-space meta.
* External Source
* defconfig
* defconfig + fragments
* linux-yocto meta data + local fragments.
* building external modules (hello-mod).
 
 
 
 
= Test Strategy =
There are several test approaches for Kernel Development, such as:
Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule.
Write new test cases based on developer request or new features as required.
Maintain current test cases and update them accordingly the new changes.
Perform exploratory testing on existing functionalities.
 
 
== Test automation ==
Not required until now.
== Test Approach ==
 
=== Sanity testing ===
* Not covered at this moment
* TBD following DEV discussions
 
=== Performance and Stress ===
* Usability
 
===Regression===
* Regression testing will be covered on Automation execution.
 
== Maintaining the test cases ==
== Submitting Bugs ==
Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines.
Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about [https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking our process for reporting bugs].
 
=Requirements=
 
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.
 
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p> additional detailed information about builds history.  -
 
==HW Requirements==
* Any machine able to build Yocto Project.
 
==Software Requirements==
* poky.
* GNU/Linux environment
 
==Environment Requirements==
**YP 2.1 Release:
**Ubuntu-14.04
**Ubuntu-14.10
**Ubuntu-15.04
**Fedora-21
**CentOS-6.*
**CentOS-7.*
**Debian-7.*
**Debian-8.*
**openSUSE-project-13.2
 
=Features=
<b> Features to be Tested </b>
 
<b><p>1. linux-yocto-custom.</p></b>
<p>1.1 local source. </p>
<p>1.2 local source with parallel meta</p>
<p>1.3 local source with recipe-space meta</p>
 
 
<b><p>2. External Source.</p></b>
 
<b><p>3.Defconfig.</p></b>
<b><p>4.Defconfig + Fragments.</p></b>
<b><p>5.linux-yocto meta data + local fragments.</p></b>
<b><p>6.building external modules (hello-mod).</p></b>
 
 
 
= Schedule =
Every cycle depending on Milestone and release candidate
 
==Test execution Cycle==
{| class="wikitable"
! Build
! Automated Test
! Manual Test
|-
| Daily Master
| Y
| NO
|-
| Weekly Build
| In progress
| Y
|-
| Milestone Build
| Y
| Y
|}
 
=Dependencies=
*YP Dependencies
** poky
** meta-kerneltest recipe
 
 
=Risk Assumptions=
* The complete features will not be tested now, the design of the test cases is under construction.
 
=Tools=
<p> TBD </p>
 
= Release Criteria/ Exit Criteria =
 
 
 
 
<!--
THIS IS THE OLD TESTING PLAN
 
= Test Areas =
eSDK consists of two big components, as follows:
 
== Backend ==
 
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/eSDK-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_eSDK.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
 
==== Usability tests ====
*  Verify the easy usage of eSDK ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the eSDK server])
 
== Frontend ==
 
==== Functionality tests ====
*  Manual testing in the first stage;
*  Automate testing using  [http://www.seleniumhq.org/ Selenium], in the second stage;
 
==== Compatibility tests ====
*  Verify the behavior of the GUI on different browsers and operating systems; TBD
 
==== Usability tests ====
*  Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
*  Friendly graphical user interface;
 
==== Performance tests ====
*  Stress testing (e.g. display appropriate error messages when the system is under stress);
 
= Test Cycle =
{|class="wikitable" style="text-align: center;"
| || || colspan="3" | Test execution cycle
|-
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]
|-
| rowspan="2" | Build type || Weekly  || Yes || Yes ||
|-
| Release || Yes || Yes || Yes
|-
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes
|-
| [[#Frontend]]  || Yes || Yes || Yes
|-
| rowspan="5" | Target machine || qemuarm || || Yes || Yes
|-
| qemumips|| || || Yes
|-
| qemuppc|| || || Yes
|-
| qemux86|| Yes || Yes || Yes
|-
| qemux86-64  ||  ||  || Yes
|-
| rowspan="3" | Target image|| core-image-minimal|| Yes || ||
|-
| core-image-sato-sdk|| || Yes || Yes
|}
 
== Weekly Test ==
*; Scope:
** Images built weekly and released through the distribution team.
 
*; Objective:
** Functionality test on most areas with minimum sets of tests;
** Regression test with high probability to find bugs.
 
== Full Pass Test ==
*; Scope:
** Images built as candidates for milestone or final release;
** Passed [[#Weekly Test]]
 
*; Objective:
** Ensure functionality of eSDK component.
 
== Release Test ==
*; Scope:
** Release candidates that pass [[#Full Pass Test]]
 
*; Objective:
** All scheduled features are covered, or rescheduled;
** All relevant bugs are fixed and verified.
-->

Revision as of 21:25, 30 January 2017

This article is the test plan for Kernel Development.

About Kernel Development

Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of OpenEmbedded build system. In order to Setting up the extensible SDK please go to Setting Up to Use the Extensible SDK.

Objectives

Verify all Kernel Development components to be fully functional. Components to be verified:

  • linux-yocto-custom
 * local source
 * local source with parallel meta
 * local source with recipe-space meta
  • External Source
  • defconfig
  • defconfig + fragments
  • linux-yocto meta data + local fragments
  • building external modules (hello-mod)

Team members

QA Team involved in Kernel Development testing

Francisco Pedraza

Scope

  • linux-yocto-custom
  • local source.
  • local source with parallel meta.
  • local source with recipe-space meta.
  • External Source
  • defconfig
  • defconfig + fragments
  • linux-yocto meta data + local fragments.
  • building external modules (hello-mod).



Test Strategy

There are several test approaches for Kernel Development, such as: Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule. Write new test cases based on developer request or new features as required. Maintain current test cases and update them accordingly the new changes. Perform exploratory testing on existing functionalities.


Test automation

Not required until now.

Test Approach

Sanity testing

  • Not covered at this moment
  • TBD following DEV discussions

Performance and Stress

  • Usability

Regression

  • Regression testing will be covered on Automation execution.

Maintaining the test cases

Submitting Bugs

Being part of the Yocto Project, Kernel Development follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. Kernel Development bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.

8137 As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Test Case 1471

additional detailed information about builds history. -

HW Requirements

  • Any machine able to build Yocto Project.

Software Requirements

  • poky.
  • GNU/Linux environment

Environment Requirements

    • YP 2.1 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2

Features

Features to be Tested

1. linux-yocto-custom.

1.1 local source.

1.2 local source with parallel meta

1.3 local source with recipe-space meta


2. External Source.

3.Defconfig.

4.Defconfig + Fragments.

5.linux-yocto meta data + local fragments.

6.building external modules (hello-mod).


Schedule

Every cycle depending on Milestone and release candidate

Test execution Cycle

Build Automated Test Manual Test
Daily Master Y NO
Weekly Build In progress Y
Milestone Build Y Y

Dependencies

  • YP Dependencies
    • poky
    • meta-kerneltest recipe


Risk Assumptions

  • The complete features will not be tested now, the design of the test cases is under construction.

Tools

TBD

Release Criteria/ Exit Criteria

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Kernel_Development_QA&oldid=23473"