Distro Testing Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 24: Line 24:




== GDC Autobuilder ==
== Staging AutoBuilder ==


When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.
When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by [[QA]], this is a one-time testing activity, not executed in every milestone o release test cycle.

Revision as of 23:40, 2 February 2017

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder

About Distro Testing

Distro Testing is part of the process of enabling an OS distribution at the Yocto Project Autobuilder. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.

Test Objectives

  • Verify that Distro executes oe-selftest
  • Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
  • Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
  • Verify that Distro is able to build different image types (LSB, non LSB)
  • Verify that Distro is able to build per architecture (arm, x86)
  • Verify that Distro is able to build per bootloader (systed, init)
  • Verify that Distro is able to build poky-tiny
  • Verify that Distro executes multilib

Test Strategy

The strategy will be divided in two groups:

  • Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing)
  • SWAT monitoring at the Autobuilders


Staging AutoBuilder

When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by QA, this is a one-time testing activity, not executed in every milestone o release test cycle.

Public Autobuilder

Once the distro was first time validated by QA team it will be running on public autobuilder different build sets already defined , after that it going to be under the focus of SWAT team

Process

This section describes the process to add a new Distro as supported on Yocto Project

Distro Steps v2.PNG


Open a Bug

When a new Distro is identified to be added as a supported candiate a bug should be raised on Bugzilla and have to be assigned to the QA Responsible of the Distro Testing.

Install Distro

QA responsible should ensure that the new Distro is installed properly on the local AB

Process To Install New Distro

1. Install the new distro 2. Intsall the requirements from the YP Quickstart 3. Start YP Autobuilder 

  $ git clone ssh://git@git.yoctoproject.org/yocto-autobuilder
  $ cd yocto-autobuilder
  
  # Generates password for AB Web
  $ ./bin/htpasswd -c -b .htpasswd distrouser distropass

  # Load buildset configuration with nightly-qa-distro $ ln -sf buildset-config.yocto-qa/ buildset-config

  # start the autobuilder
  $ source yocto-autobuilder-setup
  $ ./yocto-start-autobuilder both

4. Open a browser and enter to http://localhost:8010 

  4.1 If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log

5. Log in with the distrouser and distropass 6. Enter to http://localhost:8010/builders/nightly-qa-distro and Force a build 7. Wait to the buildsets to end and review the output, load bugs if necessary 8. Stops autobuilder 

  $ ./yocto-stop-autobuilder both

Execute Build Sets

Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB

RPM DEB IPK Component Multilib World oe-selftest
ARM x86_64 x86_32 Eclipse x86 64/32 x86_64
init systemd systemd Toaster x86 64/x32 systemd
poky_lsb poky poky_tiny buildtools core-image-sato poky
core-image-lsb core-image-sato-sdk core-image-minimal read_only rootfs poky core-image-sato-sdk
pam init/systemd
Log rotate RPM,DEB,IPK

Results

The QA Responsible should ensure that all the defined Build Sets are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.

Update Supported Distro List

Once all the build sets were executed and all were PASS, then it can be added to the list of supported Distros, also the dropped Distros should be removed from the list.

Sanity Tested Distro File (Snapshot)

meta-poky/conf/distro/poky.conf
-----
SANITY_TESTED_DISTROS ?= " \
             poky-1.7 \n \
             poky-1.8 \n \
             poky-2.0 \n \
             Ubuntu-14.04 \n \
             Ubuntu-14.10 \n \
             Ubuntu-15.04 \n \
             Ubuntu-15.10 \n \
             Ubuntu-16.04 \n \
             Fedora-21 \n \
             Fedora-22 \n \
             Fedora-23 \n \
             CentOS-6.* \n \
             CentOS-7.* \n \
             Debian-7.* \n \
             Debian-8.* \n \
             openSUSE-project-13.2 \n \

Requirements

HW Requirements

To be updated

Software Requirements

To Be Updated

Environment Requirements

To Be Updated

Test execution Cycle

QA Responsible

TBD meanwhile you can contact José Pérez Carranza

Public Autobuilder

  • There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team

Exit Criteria for Enabling an OS Distribution

Supported OS Distributions

The distributions used are:

  • Poky
  • Fedora
  • Ubuntu
  • CentOS
  • OpenSuse
  • Debian

Supported OS Distributions Criteria

List of supported OS Distributions should be the same at Mega Manual and in variable mentioned in Distro File if is not equal should a bug should be filled.

The following criteria is used to see if a Distro and his specific version is supported

  • Poky
    • Most 2 recent releases
  • Fedora
    • Most 2 recent releases
  • Ubuntu
    • Most recent release
    • Most recent LTS release
    • When above are the same, use most recent 2 releases
  • CentOS
    • Most recent release
  • OpenSuse
    • Most recent release
    • Most recent LEAP release
  • Debian
    • Most recent release

BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list.

List

OS Version Status
Ubuntu-16.04 LTS PASS
Ubuntu-16.10 PASS
Fedora-24 PASS
Fedora-25 NOT YET TESTED
CentOS-7 PASS
Debian-8 PASS
openSUSE-13.2 PASS
openSUSE-42.2 PASS

Supported OS Distributions Release Calendar

This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Distro_Testing_Plan&oldid=23624"