Testopia

From Yocto Project
Jump to: navigation, search

Contents

About this wiki page

This page is designed to help testers and community members accommodate to the Yocto Project new test case management platform, Testopia. While I will try to explain in as much detail without going too much offtrack, please consult the Testopia home page (http://www.mozilla.org/projects/testopia/) and the Testopia manual (http://landfill.mozilla.org/testopia2/testopia/doc/Manual.pdf) for more detailed information about Testopia itself.

This documentation assumes you have a minimum knowledge about testing procedures and Bugzilla.

Introduction

Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. The reason this test case management platform was chosen is that is facilitates involvement from the community by associating itself with the already community-driven defect reporting platform Bugzilla. We hope that interested community members already active or not in Bugzilla will start using Testopia as a means of aiding and communicating with the Yocto Project QA teams.

As mentioned above, the main goal of Testopia is to involve the community in the testing of the Yocto project. Using this platform, community members can follow test plans from the QA teams, propose test plans, test cases and test runs, aid in the testing of features provided the interest and ultimately develop their own test plans and test cases to run and maintain themselves.

CONTACT: Please contact me for any questions, issues, ideas. You can use email corneliux.stoicescu@intel.com or on Yocto IRC chat: Corneliu

Why would I want to use Testopia?

Like the reasons for using Bugzilla, there are several reasons for using a common test case management environment:

  • Continuous support from the QA teams
  • You can track the progression of weekly or release-based testing done by various QA teams. If you are interested in one or a number of specific test case results you will be able to get the results as soon as they are introduced in Testopia.
  • You can add your own test cases. This way you have a solid foundation for managing and reporting your test case results. Either by using the Community test plan, your own test plan or being given access to the official branches test plans (master, danny, etc.) you can add your own test cases and conduct test runs. Also because the results are placed in a common public environment, everybody can see your progress and the status of your tested features.
  • You can aid the QA teams in order to speed up the QA process for features you have interest in. The sooner a feature has all its issues cleared, the sooner it will move forward and be enhanced.
  • You can check all the issues that have been reported related to a specific feature. Testopia has a history of related defects for all of its test cases.
  • Your testing results can be published to the community. Using the upcoming MediaWiki extension for Testopia automated testing reports will be generated real-time. You can create your own testing reports or you can have them added to the QA teams reports.
  • Keep your testing time to a minimum using the results obtained from other QA teams.
  • Ultimately your test cases may be automated using our automated testing infrastructure.

All in all, the main advantage of using Testopia is that you can be a part of the global Yocto Project QA Team and you can do it in relationship with your interest and at your own pace.

What can Testopia do?

  • Test case management and organization
  • Relate to Bugzilla elements
  • Make connections between bugs and test cases
  • Keep a history for test cases and test runs
  • Automated test cases using scripts
  • Automated bug reporting
  • Automated test reporting using MediaWiki extension (now implemented)
  • Test case linking using dependencies
  • Easy reporting using the built-in reporting tool (tables, graphs)
  • Testing results export to CSV
  • Improve communication between the testing teams, the development teams and each other

Who is using Testopia?

The following companies are helping with the testing of the Yocto Project:

  • Intel OTC
  • Wind River
  • Huawei
  • Enea

If you would like to join this list, either as a company or as an individual, please use the contact details at the end of this wiki.

What do I need to start using Testopia?

All you need is a Bugzilla user account. Get one here: https://bugzilla.yoctoproject.org/ .

Testopia and the Yocto Project

Testopia is an open source software designed for a general test case management purpose; because of this, we needed to choose a way to use the features given by it without compromising functionality and adapting them to the Yocto Project QA needs. Following is some information needed to aid community members as well as new dedicated testers in utilizing Testopia in a minimally standardized way. The reason behind the standardized elements is that the communication between Testopia users is limited due to physical distance and timezone difference, because we will be working on the same testing platform and we will be sharing resources and information. Despite all this we need to be efficient and productive, thus collaboration is the key.

Accessing Testopia

Any Bugzilla user can access Testopia by clicking on the "Product Dashboard" inside the main Bugzilla page https://bugzilla.yoctoproject.org/ .

Product dashboard.png

User permissions

As any user-based software, we developed a user permissions management system aimed at controlling "who can do what". By default all Bugzilla accounts have the right to see (read) all test plans and their content, as the Yocto project is an open source project. Any further permissions will be given on a request-need basis from the Testopia administrators or the Bugzilla administrator.

Please note that "be default" does not mean any newly created Test Plan has public read access. This permission must be given in the Test Plan permissions tab using the .* regex.

Bugzilla and Testopia interaction applied to the Yocto Project needs

Below is a diagram aimed at showing how Testopia interacts with elements in Bugzilla as well as how the common Bugzilla elements interact with each other and how the Testopia elements interact with eachother. Even though it contains many details, it is however a complete diagram meaning it contains all the elements you will likely encounter while using Testopia.

Testopia diagram.png

Following are details about the above diagram. Please note that all the following information is presented phrasing the above diagram from left to right (example: when talking about test plans I will present their relationship with all the elements taken from left to right according to the above diagram)


Products

The products in Testopia are imported from Bugzilla and are basically the products of the Yocto Project. These products can be seen in the left column of Testopia main page:

Products.png

By clicking on a product you will see detailed description about it like the Test Plans it contains, the test cases and so forth.


Test Plans

  • test plans are the core of Test Case organization.
  • test plans are unique to the product they belong to and can not be shared between products.
  • test plans contain all the test cases relevant to the product they belong to.
  • test plans within the same product can share test cases.
  • test plans do not share test runs.
  • test plans within the same product share builds.
  • test plans within the same product share environments.
Naming

 We choose to name test plans similar to the branches they test. Thus, for starters, we will have a Test Plan named "master".
 We also introduced a Test Plan named "Community" in which all members have read-write access and can use it to submit Test Cases 
 and Test Runs. These can also be used to conduct their own Test Runs until by request-need basis they can receive their own 
 Test Plan or introduce them in an already-existing one.

Test Cases

  • test cases can not be shared between products.
  • test cases can be shared between test plans within the same product.
  • a test case can not exist outside at least 1 test plan.
  • test cases are closely linked to the components defined in Bugzilla.
  • each test case should have at least one component assigned to it.
  • test cases have 2 special fields assigned to them: priority and category.
  • test case category can be: weekly, full pass pre-release, full-pass post-release, sanity.
  • test case priorities are linked to Bugzilla and affect bugs filed from the test case. They can be: Undecided, Low, Medium, Medium+, High.
Naming

 Test cases should have a short name that describes its purpose and functionality.

Test Runs

Test runs hold the results of how certain test cases fared in a specific build and environment. They are composed of Case Runs.

  • test runs are unique to a specific product.
  • test runs can not be shared between products.
  • test runs are unique to the test plan they belong to.
  • test runs can not be shared between test plans.
  • test runs contain case runs.
  • test runs have a unique build and all case runs within it are bound to it.
  • test runs have a default environment but the case runs within in are not bound to it.
Naming  starting in 2.2

 Test runs are of 2 categories: templates and actual usable test runs. 

- templates have the following naming format:
    Spec:

       <TEMPLATE-NAME>_<release-version>_<execution-type>_<component>_[image-hardware] 
    
    Example: 
       
       TRTEMP_2.1_AUTO_BSP/QEMU_core-image-sato-sdk_ANYYQEMU

    Details :

        -  < > are mandatory fields 
	-  [ ] are optional fields
	-  TEMPLATE-NAME : Type of the template, could be 3 different types of templates 
	    -  TRTEMP: Templates for test runs 
	    -  TCTEMP: Templates for test cases
	    -  TPTEMP: Templates for test plans 
	-  release-version: Release version 
	-  execution-type:  how the test cases inside of test run are executed could be MANUAL or AUTO 
	-  component: The component as is listed in Testopia 
	-  image_hardware: This only applies to BSPs and is the image that is tested along with the hardware tested
   
  
- usable test runs are clones of the templates and replace the TEMPLATE_NAME field with the date of the clone
      
       Example: 
       
           2016-05-15_2.1_AUTO_ BSP/QEMU_ core-image-sato-sdk_ ANYYQEMU

Naming until 2.1

Test runs are of 2 categories: templates and actual usable test runs. 
- templates have the folowing naming format: TEMPLATE - <name>
- usable test runs are clones of the templates and replace the TEMPLATE field with the date of the clone. Format: <date> - <name>

Case Runs

Each Case Run is a link to a Test Case and also contains specific information like how the test case fared in a specific build and environment. Multiple Case Runs can be associated to a Test Case but no more than 1 in the same Test Run.

  • case runs have no direct connection to products.
  • case runs have no direct connection to test plans.
  • each case run is linked to a unique test case.
  • case runs compose test runs.
  • case runs are unique to a specific test run.
  • case runs within the same test run have a unique build.
  • case runs within the same test run can have multiple environments.
  • case runs provide information regarding how test cases fared related to a specific build and environment.

Builds

Builds are directly related to Yocto Project commits to the Git repository. Each build is associated with a unique combination of branch and commit.

  • builds are unique to a specific product, though they can be copied to another.
  • builds are shared to test plans within the same product.
  • builds have no direct connection to test cases.
  • builds can be shared between test runs.
  • builds are unique to case runs within the same test run.
  • each build has a unique milestone imported from Bugzilla.
  • builds have no direct connection to environments.

Note: Builds contain an "active" field. Only active builds can be used for new test runs. All builds that are not currently tested on should be deactivated.

Naming

 Builds have 2 fields: name and description
 - build name format: <branch>:<commit>
 - build description can contain any relevant information like date of the build or a link to the autobuilder images.

Environments

Environments represent the machine the testing is done on. Please note that it does not represent the target build architecture(ex: qemux86, qemuarm, etc.); however this information can be set either in the Test Run's name or the Test Case itself.

  • environments are unique to a specific product, though they can be copied to another.
  • environments are shared to test plans within the same product.
  • environments have no direct connection to test cases.
  • environments can be shared between test runs.
  • environments can be shared between case runs within the same test run.
  • environments have no direct connection to builds.
Naming

 Environments are named after the name + version of the testing environment.
 - environment naming format: <distribution> <version> <architecture> . Example: Ubuntu 12.04 x86_64

Testing workflow

Following is a testing workflow diagram that any tester should be able to use. The key points are marked with a red contour while the most common path is marked with green arrows.

  • squares represent actions
  • circles represent decision points
  • this diagram does not take into account mistakes made by the tester

If you feel more states should be added, please contact this wiki's maintainer.

Testing workflow.png

Need help? Contact us.

This wiki's maintainer: corneliux.stoicescu@intel.com

If you have any questions, issues, ideas about Testopia, please contact me using email corneliux.stoicescu@intel.com or IRC chat on the Yocto channel, http://webchat.freenode.net/, nick Corneliu.

You can also consult the Testopia HOWTO wiki: https://wiki.yoctoproject.org/wiki/Testopia_HOWTO

Personal tools