Extensible SDK Test Plan (eSDK): Difference between revisions

From Yocto Project
Jump to navigationJump to search
 
(27 intermediate revisions by 2 users not shown)
Line 43: Line 43:
* Install additional SDK components.
* Install additional SDK components.
* Locating Pre-Built SDK Installers.
* Locating Pre-Built SDK Installers.
*
 
<p><b>Node.js</b></p>
<p><b>Node.js</b></p>
* Add Node.js Modules
* Add Node.js Modules
Line 56: Line 56:


== Test automation ==
== Test automation ==
Currently the execution of the cases is using "testsdkext" automation below are the three scenarios that should be tested
* Test Building the eSDK installer:
  1. Clone poky and source the environment
      $ source oe-init-build-env
  2. Add proper configuration to local.conf
      $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf
      $ echo 'INHERIT += "testsdk"' >> conf/local.conf
  3. Build eSDK intaller
      $ bitbake core-image-minimal  -c populate_sdk_ext -v
  4. Execute the tests
      $ bitbake core-image-minimal -c testsdkext
  5. All test should be PASSED
* Test using Pre-Built SDK Installers:
  1. Under poky/ do another build folder different from the previous scenario
      $ source oe-init-build-env build-pre-built
  2. Add proper configuration to local.conf
      $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf
      $ echo 'INHERIT += "testsdk"' >> conf/local.conf
  3. Create sdk directory
      $ mkdir -p tmp/deploy/sdk
  4. Download artifacts form AB under new folder created
      $ cd  tmp/deploy/sdk 
      $ wget <AB_LINK>/toolchain/x86_64/<ARTIFACTS_NEEDED>
       
          Example:
          $ wget https://autobuilder.yocto.io/pub/releases/yocto-2.4_M3.rc1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest 
         
          NOTE: All the artifacts that should be downloaded are:
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.target.manifest
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.sh
            - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.testdata.jason
    5. Give execution permissions to sh files. Rename files, only if it asks.
      $ chmod +x *.sh
      $ for file in *<RELEASE>*; do mv "$file" "${file/<RELEASE>/<RELEASE>+snapshot}"; done
        Example execute below command:
        $ for file in *2.3*; do mv "$file" "${file/2.3/2.3+snapshot}"; done
    6. Go back to poky build folder "poky/build-pre-built"
      $ cd -
    7. Prepare the environment to execute tests   
      $ bitbake libxml2
    8. Execute the test 
      $ bitbake core-image-minimal -c testsdkext
    9. All test should be PASSED
    NOTE: There are 4 combinations of tests that should be executed
        - qemux86 + minimal --> (poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext*)
        - qemux86 + sato    --> (poky-glibc-x86_64-core-image-sato-i586-toolchain-ext*)
        - qemux86-64 + minimal  --> (poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext*)
        - qemux86-64 + sato    --> (poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext*)
* Test eSDK as part of the oe-selftest suite
  1. Under poky/ do another build folder different from the previous scenario
      $ source oe-init-build-env build-esdk-selftest
  2. Avoid Sanity Distro validation
      $  echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf
  3. Execute the tests
      $ oe-selftest -r eSDK
  4. All test cases should be PASS


== Test Approach ==
== Test Approach ==
Line 76: Line 157:
=Requirements=
=Requirements=


This set of requirements is listed in a two column format, bugzilla entries vs. requirement description in the form of a user story.
This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.
 
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to to install needed development libraries and headers from published sstate feeds.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want distributable sstate cache for individual components.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8960] As a developer I want to be able to install node.js modules' in bitbake output.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant.</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool.</p>


<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8137 8137 ] As a developer I want to be able to install needed development libraries and headers from published sstate feeds.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1471 Test Case 1471]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want to be able to generate target packages in SDK - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1473 Test Case 1473] </p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8135 8135] As a developer I want to be able to generate images out of binary feeds.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1472 Test Case 1472]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8136 8136] As a developer I want the ability to Public eSDK with modifications/addons.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1474 Test Case 1474]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to create eSDK image benchmark.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1475 Test Case 1475]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8974 8974] As a developer I want to be able to develop eSDK software benchmark.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1476 Test Case 1476]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8960 8960] As a developer I want to be able to install node.js modules' in bitbake output.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1477 Test Case 1477]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8690 8690] As a developer I want to be able to create proper recipes for Node.js modules for devtool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1478 Test Case 1478]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7635 7635] As a developer I want to be able to extend cmake recipe creation on recipetool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1479 Test Case 1479]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=8999 8999]As a developer I want to be able to create a kernel recipe with custom .config for devtool.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1480 Test Case 1480]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 8138]As a developer I want to be able to store additional detailed information about builds history.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1481 Test Case 1481]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=7634 7634] As a developer I want to be able to extend autotools recipe creation.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1482 Test Case 1482]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=9040 9040] As a developer I want to be able to list all the content of bundles using Bitbake.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1483 Test Case 1483]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8982 8982] As a developer I want to be able to add support out-of-tree kernel modules. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1484 Test Case 1484]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8975 8975] As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant.  - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1485 Test Case 1485]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=6658 6658] As a developer I want to be able to enable kernel development from devtool. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1486 Test Case 1486]</p>
<p>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8883 8883] As a developer I want to be able to update eSDK. - [https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=1487 Test Case 1487]</p>
==HW Requirements==
==HW Requirements==
* Any machine able to build.
* Any machine able to build.
Line 117: Line 198:


<b><p>1. Beggining on a recipe.</p></b>
<b><p>1. Beggining on a recipe.</p></b>
<p>1.1 add.</p>
<p>1.1 add. </p>
<p>1.2 modify.</p>
<p>1.2 modify. Done in oe-selftest</p>
<p>1.3 upgrade.</p>
<p>1.3 upgrade. Done in oe-selftest</p>




Line 130: Line 211:
<p>3.1 edit-recipe.</p>
<p>3.1 edit-recipe.</p>
<p>3.2 configure-help.</p>
<p>3.2 configure-help.</p>
<p>3.3 build.</p>
<p>3.3 build. - Done in oe-selftest</p>
<p>3.4 update recipe.</p>
<p>3.4 update recipe. - Done in oe-selftest</p>
<p>3.5 reset.</p>
<p>3.5 reset. - Done in oe-selftest</p>






<b><p>4. Testing changes on target.</p></b>
<b><p>4. Testing changes on target.</p></b>
<p>4.1 deploy-target.</p>
<p>4.1 deploy-target. - Done in oe-selftest</p>
<p>4.2 undeploy-target.</p>
<p>4.2 undeploy-target.</p>
<p>4.3 package.</p>
<p>4.3 package.</p>
<p>4.4 build-image.</p>
<p>4.4 build-image.- Done in oe-selftest</p>
<p>4.5 runqemu.</p>
<p>4.5 runqemu. - Done in oe-selftest </p>




<b><p>5. Advanced.</p></b>
<b><p>5. Advanced.</p></b>
<p>5.1 build-sdk.</p>
<p>5.1 build-sdk.</p>
<p>5.2 extract.</p>
<p>5.2 extract. Done in oe-selftest</p>
<p>5.3 sync.</p>
<p>5.3 sync.</p>


Line 164: Line 245:


<b><p> 9. Node.js dependencies installation. </p></b>
<b><p> 9. Node.js dependencies installation. </p></b>
<p>9.1 TBD. </p>
<p>9.1 Add Node.js modules. </p>
<p>9.2 TBD.</p>


= Schedule =
= Schedule =
Line 173: Line 253:
{| class="wikitable"
{| class="wikitable"
! Build
! Build
! Weekly Test
! Automated Test
! Full Pass
! Manual Test
|-
|-
| Daily Master
| Daily Master
| Y
| Y
| Y
| NO
| NO
|-
|-
| Weekly Build
| Weekly Build
| In progress
| Y
| Y
| Y
| NO
|-
|-
| Milestone Build
| Milestone Build
| Y
| Y
| Y
| Y
| Y
Line 199: Line 276:


=Risk Assumptions=
=Risk Assumptions=
 
* The complete features will not be tested now, as per the automation is in process now.


=Tools=
=Tools=
 
<p> Software Development Kit </p>


= Release Criteria/ Exit Criteria =
= Release Criteria/ Exit Criteria =

Latest revision as of 19:50, 26 September 2017

This article is the test plan for eSDK.

About eSDK

Extensible SDK makes it easy to add new applications and libraries to an image, edit the source for an existing component, test the changes on the target hardware, and also allow you to integrate into the rest of OpenEmbedded build system. In order to Setting up the extensible SDK please go to Setting Up to Use the Extensible SDK.

Objectives

Verify all Extensible SDK components to be fully functional. Components to be verified:

Team members

QA Team involved in eSDK testing

Francisco Pedraza

Scope

eSDK
  • eSDK Installation.
  • eSDK Setup.

Devtool features to be tested:
  • Add a new recipe.
  • Modify the source for an existing recipe.
  • Upgrade an existing recipe.
  • Show work-space status.
  • Search available recipes.
  • Edit a recipe file in your work-space.
  • Get help on configure script options.
  • Build a recipe.
  • Apply changes from external source tree to recipe.
  • Remove a recipe from your work-space.
  • Deploy recipe output files to live target machine.
  • Un-deploy recipe output files in live target machine.
  • Build packages for a recipe.
  • Add Native Tools.
  • Build image including work-space recipe packages.
  • Run QEMU on the specified image.
  • Build a derivative SDK of this one.
  • Extract the source for an existing recipe.
  • Synchronize the source tree for an existing recipe.
  • Update SDK components.
  • Install additional SDK components.
  • Locating Pre-Built SDK Installers.

Node.js

  • Add Node.js Modules

Test Strategy

There are several test approaches for Extensible SDK, such as: Perform test cases agreed upon the development during periodic Manual (full pass) according to the Schedule. Write new test cases based on developer request or new features as required. Maintain current test cases and update them accordingly the new changes. Perform exploratory testing on existing functionalities.


Test automation

Currently the execution of the cases is using "testsdkext" automation below are the three scenarios that should be tested

  • Test Building the eSDK installer:
  1. Clone poky and source the environment 
     $ source oe-init-build-env 

  2. Add proper configuration to local.conf
     $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf 
     $ echo 'INHERIT += "testsdk"' >> conf/local.conf

  3. Build eSDK intaller
     $ bitbake core-image-minimal  -c populate_sdk_ext -v 

  4. Execute the tests 
     $ bitbake core-image-minimal -c testsdkext

  5. All test should be PASSED


  • Test using Pre-Built SDK Installers:
  1. Under poky/ do another build folder different from the previous scenario 
     $ source oe-init-build-env build-pre-built

  2. Add proper configuration to local.conf
     $ echo 'SDK_EXT_TYPE = "minimal"' >> conf/local.conf 
     $ echo 'INHERIT += "testsdk"' >> conf/local.conf

  3. Create sdk directory
     $ mkdir -p tmp/deploy/sdk

  4. Download artifacts form AB under new folder created 
     $ cd  tmp/deploy/sdk  
     $ wget <AB_LINK>/toolchain/x86_64/<ARTIFACTS_NEEDED>
        
         Example:
         $ wget https://autobuilder.yocto.io/pub/releases/yocto-2.4_M3.rc1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest   
         
         NOTE: All the artifacts that should be downloaded are:
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.target.manifest
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.host.manifest
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.sh
           - poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.3.testdata.jason

   5. Give execution permissions to sh files. Rename files, only if it asks.
     $ chmod +x *.sh
     $ for file in *<RELEASE>*; do mv "$file" "${file/<RELEASE>/<RELEASE>+snapshot}"; done
       Example execute below command: 
        $ for file in *2.3*; do mv "$file" "${file/2.3/2.3+snapshot}"; done

   6. Go back to poky build folder "poky/build-pre-built"
     $ cd - 

   7. Prepare the environment to execute tests     
     $ bitbake libxml2

   8. Execute the test   
     $ bitbake core-image-minimal -c testsdkext

   9. All test should be PASSED

   NOTE: There are 4 combinations of tests that should be executed 
        - qemux86 + minimal --> (poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext*)
        - qemux86 + sato    --> (poky-glibc-x86_64-core-image-sato-i586-toolchain-ext*)
        - qemux86-64 + minimal  --> (poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext*)
        - qemux86-64 + sato     --> (poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext*)
  • Test eSDK as part of the oe-selftest suite
  1. Under poky/ do another build folder different from the previous scenario 
     $ source oe-init-build-env build-esdk-selftest

  2. Avoid Sanity Distro validation 
     $  echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf 

  3. Execute the tests 
     $ oe-selftest -r eSDK

  4. All test cases should be PASS

Test Approach

Sanity testing

  • Not covered at this moment
  • TBD following DEV discussions

Performance and Stress

  • Usability

Regression

  • Regression testing will be covered on Automation execution.

Maintaining the test cases

Submitting Bugs

Being part of the Yocto Project, eSDK follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. eSDK bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

This set of requirements is listed in a two column format, Bugzilla entries vs. requirement description in the form of a user story.

8137 As a developer I want to be able to install needed development libraries and headers from published sstate feeds. - Test Case 1471

8136 As a developer I want to be able to generate target packages in SDK - Test Case 1473

8135 As a developer I want to be able to generate images out of binary feeds. - Test Case 1472

8136 As a developer I want the ability to Public eSDK with modifications/addons. - Test Case 1474

8974 As a developer I want to be able to create eSDK image benchmark. - Test Case 1475

8974 As a developer I want to be able to develop eSDK software benchmark. - Test Case 1476

8960 As a developer I want to be able to install node.js modules' in bitbake output. - Test Case 1477

8690 As a developer I want to be able to create proper recipes for Node.js modules for devtool. - Test Case 1478

7635 As a developer I want to be able to extend cmake recipe creation on recipetool. - Test Case 1479

6658 8999As a developer I want to be able to create a kernel recipe with custom .config for devtool. - Test Case 1480

8138As a developer I want to be able to store additional detailed information about builds history. - Test Case 1481

7634 As a developer I want to be able to extend autotools recipe creation. - Test Case 1482

9040 As a developer I want to be able to list all the content of bundles using Bitbake. - Test Case 1483

8982 As a developer I want to be able to add support out-of-tree kernel modules. - Test Case 1484

8975 As a developer I want to be able to validate the way how to detect recipe problems where it can become host-dependant. - Test Case 1485

6658 As a developer I want to be able to enable kernel development from devtool. - Test Case 1486

8883 As a developer I want to be able to update eSDK. - Test Case 1487

HW Requirements

  • Any machine able to build.

Software Requirements

  • eSDK.

Environment Requirements

    • YP 2.1 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2

Features

Features to be Tested

1. Beggining on a recipe.

1.1 add.

1.2 modify. Done in oe-selftest

1.3 upgrade. Done in oe-selftest


2. Getting Information.

2.1 Status.

2.2 Check.


3. Working on a recipe in the workspace.

3.1 edit-recipe.

3.2 configure-help.

3.3 build. - Done in oe-selftest

3.4 update recipe. - Done in oe-selftest

3.5 reset. - Done in oe-selftest


4. Testing changes on target.

4.1 deploy-target. - Done in oe-selftest

4.2 undeploy-target.

4.3 package.

4.4 build-image.- Done in oe-selftest

4.5 runqemu. - Done in oe-selftest


5. Advanced.

5.1 build-sdk.

5.2 extract. Done in oe-selftest

5.3 sync.


6. SDK maintenance.

6.1 sdk-update.

6.2 sdk-install.

7. sstate relevant features.

7.1 Ability to generate target packages in SDK.

7.2 Ability to install needed development libraries and headers from published package feeds.

8. Kernel relevant recipe and build.

8.1 Devtool: add: support out-of-tree kernel modules.

8.2 Devtool: enable kernel development.


9. Node.js dependencies installation.

9.1 Add Node.js modules.

Schedule

Test execution Cycle

Build Automated Test Manual Test
Daily Master Y NO
Weekly Build In progress Y
Milestone Build Y Y

Dependencies

  • YP Dependencies
    • Devtool.
    • poky-glibc script.
    • toolchain

Risk Assumptions

  • The complete features will not be tested now, as per the automation is in process now.

Tools

Software Development Kit

Release Criteria/ Exit Criteria

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Extensible_SDK_Test_Plan_(eSDK)&oldid=31947"