Distro Testing Plan

From Yocto Project
Jump to navigationJump to search

This article is the test plan for Distro testing

About Distro Testing

Distro Testing is intended to catch bugs that are distribution specific using the yocto-autobuilder.

Objectives

  • Verify that Distro executes oe-selftest
  • Verify that Dsitro is able to execute componenets (Eclipse, Toaster, WIC)
  • Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
  • Verify that Distro is able to build different image types (LSB, non LSB)
  • Verify that Distro is able to build per architecture (arm, x86)
  • Verify that Distro is able to build per bootloader (systed, init)
  • Verify that Distro is able to build poky-tiny
  • Verify that Distro executes multilib

Test Strategy

The strategy will be divided in two groups one executed once in GDC autobuilders when the Distro is released and another tested continuously on public autobuilders


GDC Autobuilder

The first time that Distro is released it will be installed on GDC autobuilder and a series of build steps going to be applied to that Distro, once very build set is PASS that distro will be added to the list of supported sitros by QA team

Steps

Distro Steps.PNG

Public Autobuilder

Once the distro was first time validated by QA team it will be running on public autobuilder different build set already defined [[]], after that it going to be under the focus of SWAT team


Process

This section describes the process to add a new Distro as supported on Yocto Project


Enable Distro

Update Supported Distros

Requirements

HW Requirements

A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

Software Requirements

Environment Requirements

  • Supported Browsers – no official browser selected yet, usually Firefox is used
  • Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
    • YP 2.0 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2
  • Latest version of Selenium installed as python module
  • YP QS (Maybe, depending where Toaster server is installed)
  • SQLITE 3 Client


Features

Features to be tested

Backend

  • Toaster installation and first time configuration
    • Download poky
    • Checkout on desired version
    • Start Toaster with “bitbake/bin/Toaster” command
  • Data integrity
    • Data presence - the information is being recorded into the database
    • Data consistency - the information is being recorded properly when modified from UI
    • Build data - all the information from bitbake process is being transmitted properly to the database
  • Data transfer between Toaster and Bitbake
  • Toaster and Bitbake complete cycle communication
The test runs templates for Toaster backend are are found in Testopia, as following

Frontend

  • Verifying if all view elements work (buttons, tables, sort buttons, etc.)
  • Configurations
    • Communication between UI-BackEnd
    • Browse and build any layers published in OpenEmbedded Metadata Index
    • Import your own layers for building
    • Add/Remove layers from configuration
    • Set configurations variables
    • Select a target or multiple targets to build
  • Building images
    • Build base images(core-image-minimal, core-image-sato etc)
    • Build images with added layers
    • Build images after modifying project settings
  • Data Presentation
    • All builds report table
    • View what was built and what packages were installed into final image
    • Browse the directory structure of the image
    • View the value of all variables form build configuration
    • Examine errors, warnings and trace messages
    • View information about the Bitbake task executed and reused during a build
    • View dependency relations between recipes, packages and tasks
    • View performance informations such as build time, task time, CPU usage and disk I/O
    • Build details
      • Build configuration page
        • Build artifacts
        • Bitbake variables table
      • Tasks report table
      • Metrics report tables
        • CPU usage
        • Disk I/O
        • Time
      • Included packages report table
        • Package details page
      • Recipes used during build
        • Recipe details page
    • Definition data from JSON file
      • file path: “/poky/meta-yocto/conf”
      • contains default settings and informations for:
        • configuration
        • layer source
        • releases
        • bitbake
The test runs templates for Toaster frontend are are found in Testopia, as following

New Features to be tested 2.1

Image Customization

  • Create, edit, delete custom image
  • Add/remove packages
  • Build image
  • Image details page
  • Downloading the image custom recipe

Setup covered by QA tests

  • SQLITE database
    • add MySQL database testing
    • add running Toaster as a service with Apache (e.g. production instance)
  • YP 2.0 Release:
    • Ubuntu-14.04
    • Ubuntu-14.10
    • Ubuntu-15.04
    • Fedora-21
    • CentOS-6.*
    • CentOS-7.*
    • Debian-7.*
    • Debian-8.*
    • openSUSE-project-13.2
  • Firefox browser
  • Add Chrome as well

We don’t cover yet

  • Layer integrity (covering all layers is impossible)
    • AR Mihai to get Brian’s supported layer list
  • Toaster-next branch
    • what’s the easiest way to run the automated tests for a person.
    • holy grail - make a single framework solution.
  • Safari browser
  • TBD - to check with DEV what it’s not covered yet.

Schedule

  • Toaster is tested only at Full Pass by QA Team
    • the plan is to run it according to the Test execution Cycle, after automation is in place.
  • Toaster is tested on Toaster-next branch before every merge into Master by Dev team


Test execution Cycle

Build Sanity Test/E2E Weekly Test Full Pass
Daily Master Y Y NO
Weekly Build Y Y NO
Milestone Build Y Y Y


Sanity Test is described here.
  • run on Toaster-next or poky-contrib.
    • TBD
      • need info from DEV owners on what Sanity testing is done by them
      • detail the set of tests. e.g. django unit tests, Toaster oe-selftest

Weekly test

  • Weekly testing is performed on a weekly basis on the poky weekly build.
  • E2E consists of all automated tests that can be run on Toaster
    • oe-selftest Toaster tests
  • selenium UI tests
  • TBD
    • add E2E integration information
    • add more information

Full Pass

  • On full pass all available tests will be run:
    • Automated tests:
      • Django Unit Tests
      • oe-selftest Toaster tests
      • Selenium automated UI tests
    • Manual tests
      • The non-automated tests from the Toaster test runs.
  • Full pass is scheduled to be run on every YP release candidate. The YP is divided in four release milestone structure as described in the YP Schedule.

Dependencies

  • YP Dependencies
    • Bitbake works
  • Some changes in the Toaster UI might affect running the UI automated tests using Selenium
    • e.g. Changing the element ID’s in the Toaster pages reflected in failing all UI related automated TC’s
  • recipe image we’re testing to get build data (metadata broken)


Risk Assumptions

  • TBD (AlexG)
  • Project Risks
    • QA and DEV to define this
    • Toaster doesn’t affect atm any other YP component (confirmed)
    • Changes in Bitbake, bitbake logging might affect Toaster functionality
    • Is reliant by host/browser environment
      • TBD - add opensuse as well at some point
      • TBD - expand the Toaster test framework to the supported host distributions of YP
  • Testing Risks
    • Toaster quality might be affected

Tools

  • Refer to Requirements section
  • TBD (need more detailed info of required tools)

Release Criteria/ Exit Criteria

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Distro_Testing_Plan&oldid=19335"