Distro Testing Plan

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

About Distro Testing

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project Autobuilder. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

Test Objectives

• Verify that Distro executes oe-selftest
• Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
• Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
• Verify that Distro is able to build different image types (LSB, non LSB)
• Verify that Distro is able to build per architecture (arm, x86)
• Verify that Distro is able to build per bootloader (systed, init)
• Verify that Distro is able to build poky-tiny
• Verify that Distro executes multilib

Test Strategy

The strategy will be divided in two groups:

• Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
• SWAT monitoring at the Autobuilders

GDC Autobuilder

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by QA, this is a one-time testing activity, not executed in every milestone o release test cycle.

Steps

Public Autobuilder

Once the distro was first time validated by QA team it will be running on public autobuilder different build set already defined [[]], after that it going to be under the focus of SWAT team

Process

This section describes the process to add a new Distro as supported on Yocto Project

Install Distro

To Be Updated

Execute Build Sets

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS

RPM	DEB	IPK	Component	Multilib	World	oe-selftest
ARM	x86_64	x86_32	Eclipse	x86 64/32	x86_64
init	systemd	systemd	Toaster	x86 64/x32	systemd
poky_lsb	poky	poky_tiny	buildtools	core-image-sato	poky
core-image-lsb	core-image-sato-sdk	core-image-minimal	read_only rootfs	poky	core-image-sato-sdk
			pam	init/systemd
			Log rotate	RPM,DEB,IPK

Results

• The QA team is with the responsibility to fill the the appropriate bugs if there is any failure on the Distro.

Update Supported Distro List

Once all the build sets were executed and all were PASS at least once, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

Sanity Tested Distro File (Snapshot)

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
             poky-1.7 \n \
             poky-1.8 \n \
             poky-2.0 \n \
             Ubuntu-14.04 \n \
             Ubuntu-14.10 \n \
             Ubuntu-15.04 \n \
             Ubuntu-15.10 \n \
             Ubuntu-16.04 \n \
             Fedora-21 \n \
             Fedora-22 \n \
             Fedora-23 \n \
             CentOS-6.* \n \
             CentOS-7.* \n \
             Debian-7.* \n \
             Debian-8.* \n \
             openSUSE-project-13.2 \n \

Test execution Cycle

Public Autobuilder

• There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

Exit Criteria for Enabling an OS Distribution

• All the defined build steps are PASS Execute Build Sets
• Distro was added to the supported list Update Supported Distro

Supported OS Distributions

The distributions used are Fedora, Ubuntu, CentOS, OpenSuse, Debian with their latest update. Below are the details

Supported OS Distributions

List of supported OS Distributions should be the same at [1] and in variable mentioned in []

OS Version	Status
Ubuntu-14.04	PASS
Ubuntu-14.10	PASS
Ubuntu-15.04	PASS
Ubuntu-15.10	PASS
Ubuntu-16.04	PASS
Fedora-22	PASS
Fedora-23	PASS
CentOS-7	PASS
Debian-8	PASS
openSUSE-13.2	PASS