Yocto 1.5 Overall Test Plan

Yocto 1.5 Overall Test Plan reversion history

Version	Modifier	Comments
1.0	Alexandru Georgescu	Initial Version

Yocto 1.5 Test Execution Plan

This test plan defines test targets/components, scope, strategy, configurations as well as test execution cycles for 1.5 version of Yocto.

Targets / Components to Be Tested

• Core OS: The core OS feature included Yocto kernel, distribution components(like connman, smart updater & zypper), file system. [TODO]
• Core build system: includes build system tests also using additional features and tweaks for it.
• Hob2: It includes functional tests for both frontend and backend of Hob2.
• Yocto ADT: It includes cross-toolchain, Eclipse plugin tests.
• Performance: Checks the CPU power behavior by software level tool, such as powertop. Real Power consumption when idle is collected. The performance of the build system is also recorded. [TODO]
• Distribution support: Current versions for supported distributions are tested along with one previous version and the beta release for the next one (the supported distributions are Ubuntu, Fedora, CentOS and OpenSUSE).[TODO]
• Compliance testing: LSB, LTP and POSIX tests are ran on the the selected targets. [TODO]
• Stress testing: Helltest + Crashme test suites are ran.[TODO]

What will not Be Tested in Yocto v1.5

Following feature categories won't be tested by QA team in Yocto v1.5:

• Documentation: QA will not validate the correctness of each documentation.
• License file: license files and legal process are owned by Distro team.

[TODO]

Test Environment

Test Platform matrix

[TODO] - add the test platform matrix with all the core bsps that tests will be performed on

Notes:

• Fullpass testing defined in execution plan will be performed against sato-sdk image.
• The atom images are both emgd and no-emgd images

Test Strategy and Approach

QA will cover following tests for each target in each round fullpass testing.

[TODO] - define the matrix for what Yocto component tests will be performed on each target.

Note:

• Compliance Test on Beagleboard and routerstation will only cover LTP and POSIX Test, since LSB test suite does not support ARM and MIPS.
• Jasperforest is selected for stress test, because it is powerful with 12 CPUs and 2GB memory. Moreover, it is a server, which needs more stress tests.

Test Levels

The Yocto 1.5 will be tested from the following different test execution levels.

Sanity Test

Sanity Test is a brief and quick test. It is integrated with each build process (including incremental build) in autobuilder system. It checks image basic functionalities, such as booting, network, zypper etc. Its results will be published in build web page and referred by the other testing. It is fully automation test and its execution time should be less than 10 minutes. Additional test cases can be defined and integrated.

Frequency: Sanity test can be run on autobuilder system against the built images or on the local development machines for the locally built images.

Scope: Sanity test checks basic functionality in QEMU and its execution time depends on the test cases scheduled to run

Weekly Test

Weekly test is a test cycle against the weekly image released through distribution team.