Oe-selftest

2024-02-19T06:48:56Z

Vyacheslav Yurkov:

=Introduction=
Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:
* based on Python [http://docs.python.org/2/library/unittest.html unittest]
* designed to simulate normal usage patterns
* does not cover image run-time testing
* implements a new layer that contains generic/specific metadata used only by tests

=How to use=

In order to use oe-selftest, you need the following(in this specific order):
# Download poky / openembedded-core (dora or newer branch)
# source the build environment setup script
# add the meta-selftest layer in conf/bblayers.conf
# add SANITY_TESTED_DISTROS="" to local.conf

Some selftests require qemu support, which in turns sets up internal network to perform the tests. This is a privileged operation, so make sure the user you run the tests with has required privileges. Or for simplicity extend your ''/etc/sudoers'' to allow running following commands:
* poky/scripts/runqemu-ifup
* poky/scripts/runqemu-ifdown
* poky/scripts/runqemu-gen-tapdevs

== Running all (except hidden) tests ==
To run all oe-selftest tests simply issue:
$ oe-selftest --run-all-tests

==Listing tests==
#--list-modules: lists all available modules.
#--list-classes: lists all available classes and the tests they contain.
#--list-tests-by <name|class|module|id|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.

Listing used tags:
--list-tags

Listing all tests:
--list-tests

== Running specific tests ==
First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name.
Look for the test modules inside /meta/lib/oeqa/selftest/ .
After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .

There are 2 methods for running specific tests:

Note:
*Keep in mind this structure: module.class.test
*The order the tests are running is alphabetical.
#--run-tests:
#;-–run-tests moduleA moduleB: will run all tests from this two modules
#;–-run-tests moduleA.classA module.classB: will run all tests from this two classes
#;–-run-tests moduleA.classA.test_A module.classB.test_B: will run these two tests
#:You can mix and match modules, classes and test names:
#;-–run-tests moduleA moduleB.classB moduleC.classC.testC: will run all tests from moduleA, all tests from moduleB.classB and testC from moduleC.classC
#-–run-tests-by <name|class|module|id|tag> <br>
#;name: --run-tests-by name testA testB testC: simply specify test names and it will determine automatically where they are located.
#;class: --run-tests-by class classA classB classC: simply specify class names and it will determine automatically what tests they contain and where they are located.
#;module: --run-tests-by module moduleA moduleB: simply specify module names and it will determine automatically what tests they contain.
#;id: --run-tests-by id id1 id2 id3: simply specify test ids and it will determine automatically what tests the ids point to.
#;tag: --run-tests-by tag tag1 tag2: if tests are tagged with @tag(feature=<>) you can run them like this also.

For the previous running methods you can specify the –-machine option to tell oe-selftest to run tests with custom MACHINE:
*-–machine random: will run tests switching to a random MACHINE for each test.
*-–machine all: will run every test from the list against all available MACHINES.

== Running hidden tests ==
Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above.
Example: 'oe-selftest _test_hidden_module.Class_1.test_1'

=Important file locations=
The following files and locations are relevant to oe-selftest:
* the script itself: oe-selftest
is located in the /scripts directory under the main OE-Core tree.
* the case.py module
is located in the meta/lib/oeqa/selftest directory and contains the base OESelftestTestCase class that extends unittest.TestCase (via OETestCase) and adds extra functionality.
* the test modules
are located in the meta/lib/oeqa/selftest/cases directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.
* the 'utils' library
is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.

=Basic architecture information=

When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:
== appending to the build's global configuration ==
In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.
== adding test files(like test classes) to the build ==
In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.
== appending to a recipe ==
In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.

== Cleaning up ==

We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:
* add_command_to_tearDown(command)
Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).
* track_for_cleanup(path)
Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).

Please be careful when using these methods as they have no safety checks.

=You should be aware of the following=
* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
* running all tests can take a long time! If you're modifying a particular feature/script then you may want to run just the related test module(s) or case(s) to save time.

=How to contribute=
You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes].

Project Users

2022-06-26T17:58:28Z

Vyacheslav Yurkov: /* Others */

It's hard to know which companies or projects use the Yocto Project since there is no requirement to tell us. This list is here to informally collate the companies, projects and products that use the Yocto Project in some way. This helps the project since it means we can show at least some cross section of which companies are using it and how/where. There is more information about this here: https://lists.yoctoproject.org/g/yocto/topic/82722441

= Companies using the Yocto Project =

== Semiconductor Vendors ==
* AMD (Silver Member)
* ARM (Platinum Member)
* Intel (Platinum Member)
* Microchip
* NXP (Silver Member)
* Qualcomm
* Renesas (Gold Member)
* STMicroelectronics (Silver Member)
* Texas Instruments (Gold Member)
* Xilinx (Platinum Member)

== Operating System Vendors ==
* ENEA (Silver Member)
* Linaro (Silver Member)
* Lineo (Silver Member)
* Mentor Graphics (Gold Member)
* Montavista (Silver Member)
* Wind River (Gold Member)

== Others ==
* [https://alladin.at/ alladin-IT GmbH]
* Atlas Copco
* AWS (Platinum Member)
* [https://www.axis.com/ Axis Communications]
* [https://www.balena.io/ Balena]
* BMW
* BMW Car IT
* [https://www.bruker.com/ Bruker BioSpin GmbH]
* [https://bootlin.com/ Bootlin]
* Cisco (Platinum Member)
* Comcast (Platinum Member)
* [https://www.cytera.bio/ Cytera] (via Belena)
* Dell (Silver Member)
* Dynamic Devices
* Facebook (Platinum Member)
* [https://formlabs.com/ Formlabs]
* Fujitsu
* [https://www.gardena.com/ GARDENA] (Husqvarna Group)
* [https://www.garmin.com/ Garmin]
* General Electric
* [https://genielifesciences.com/ Genie Life Sciences]
* Gigaset Communications GmbH
* Juniper
* [https://koansoftware.com/ KOAN sas]
* Kodak
* [https://lawo.com/ LAWO]
* Lexmark
* LG (Silver Member)
* Microsoft (Platinum Member)
* National Instruments
* [https://www.ovo.auto/ OVO Automotive]
* [https://www.pengutronix.de/ Pengutronix]
* reMarkable
* [https://www.smile.eu/en/offers/embedded-iot Smile ECS]
* StreamUnlimited Engineering GmbH
* [https://www.taitradio.com Tait Communications]
* [https://www.witekio.com/values-expertise/ Witekio]
* [https://www.veobot.com/ Veo Robotics]

= Products that use the Yocto Project =

* BMW cars
* Comcast set top boxes
* [https://www.gardena.com/int/products/smart/ GARDENA smart garden] (The smart garden gateway runs Yocto/OpenEmbedded. Open source parts can be found [https://github.com/husqvarnagroup/smart-garden-gateway-public on GitHub.])
* Some Garmin [https://www.garmin.com/marine/ Marine] Products
* Gigaset DSPG-SOCs based products (e.g N870, N870 Integrator, Marine 2/3/4)
* Go Pro (https://gopro.com/content/dam/help/open-source/2020-09-28%20-%20HERO9%20Black%20-GoPro%20Open%20Source%20Software%20Notice.pdf)
* LAWO [https://lawo.com/products/a__uhd-core/ A__UHD Core] [https://lawo.com/products/a__line/ A__ Line]
* Lexmark Printers
* LG webOS TVs
* Mellanox Bluefield SmartNIC
* Sky Glass (clear from sources tarball [http://oss.sky.com/SkyHD/SkyGlass/skyglass_spk.tar.bz2])
* [https://www.streamunlimited.com/hardware-modules/ StreamUnlimited hardware modules for voice assistants and connected speakers]
* Vernier LabQuest

= Projects that use the Yocto Project =

* [https://www.automotivelinux.org/ Automotive Grade Linux (AGL)]
* [https://www.balena.io/ Balena]
* Comcast RDK
* Nvidia vibrante SDK [https://docs.nvidia.com/drive/archive/4.1.8.0L/nvoss_docs/index.html]
* OpenBMC
* [https://oryx-linux.org/ Oryx Linux]
* [https://www.streamunlimited.com/stream-sdk/ StreamSDK for voice assistants and connected speakers]
* Windows Subsystem Linux (v1+v2)
* webOS

Yocto Project - User contributions [en]

Oe-selftest

Project Users