Contribute to SRTool

From Yocto Project
Jump to navigationJump to search

This page summarizes the Security Response Tool (SRTool) development process. We hope this will help you start contributing to the project. The SRTool is based on the Yocto Project Toaster codebase (Django,Python,Javascript), so many of the process and debugging techniques apply.


Published content: General CVE management issues and the SRTool


Set up the local repository and SRTool instance

1) Host requirements

The SRTool should in principle work on any host that supports Yocto Project. It requires "Python3", "Django >= 1.11", and "Sqlite3". A SQL GUI tool like 'sqlitebrowser' is recommended.

The required host package installation instructions are the same as Toaster, and the instructions can be found here: Toaster documentation

It was specifically developed and tested with:

Item Host 1 Host 2 Host 3
Host Ubuntu 16.04 Ubuntu 16.04
Python3 3.5.2 3.5.2
Django 1.11 LTS 2.2
Sqlite3 3.11 3.11


2) Cloning the SRTool 

  $ git clone ssh://git@push.yoctoproject.org/srtool && cd srtool

3) Starting the SRTool

Option #1: Restrict to the local browser: 

  $ ./bin/srtool start webport=localhost:9000

Option #2: Enable remote browsers: 

  $ ./bin/srtool start webport=0.0.0.0:9000

NOTE: The first time you run the SRTool, there will be a delay (30 to 60 minutes) as the default CVE repositories (NIST, Mitre, ...) are scanned, scored, and loaded into the database. After that, the updates will be incremental.

4) Create a superuser

You will need to create at least one super user account to promote users to higher permission levels: 

  $ ./bin/srt manage createsuperuser

5) Open browser to <IPADDR>:9000

This will bring up the SRTool page.

6) Read the "Guided Tour"

Click on the "Guided Tour" button on the home page to see the on-line tool and user information.

7) Create your guest account

Click on the "Login" button in the top bar, select "Request Account", fill in the information, Click "Sign Up", and then log in.

8) Promote your account to Admin level

Login as the superuser, click "Management > Manage Users", click the edit icon for your guest account row, change the "Group" to "Admin", and click "Submit Changes".

You can now log out of the superuser account and into your own account.


Sanity Test Bring-up

You can use these steps to start a minimal instance of the SRTool to help validate that the host and the code base are working on your system. This will for example set the environment variable "SRTDBG_MINIMAL_DB=1" which will only fetch a few representative records from each data source. 

 $ git pull
 $ # Copy the dev tools to the base directory
 $ cp bin/dev_tools/* .
 $ # Source the debug environment
 $ . ./srt_env.sh debug
 $ # Start the system from scratch
 $ ./recreate.sh
 $ # Run the sanity test
 $ ./bin/common/srtool_sanity_test.py -i
 $

Sample output: 

 $ ./bin/common/srtool_sanity_test.py -i
 Uname       = #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64
 Django      = (1, 11, 8, 'final', 0)
 Python3     = Python 3.5.2
 Sqlite3     = 3.11.0 2016-02-15
 Server PID  = 10234
 Server Port = [10234](/opt/srtool/lib/manage.py runserver --noreload 0.0.0.0:9123)
 CVEs        = 64
 Users       = 5
 Data source = 22
 $

To re-start as the full instance, do this: 

 $ # Source the standard environment
 $ . ./srt_env.sh
 $ # Start the system from scratch
 $ ./recreate.sh
 $


First Time Bring-up

  • When you first start, you will have the CVEs from 2015 to 2018
  • You will not have any Vulnerabilities, Investigations, or Defects until you (a) create them, and/or (b) integrate your defect system or other sustaining data.
  • All CVEs will by default get the status "Historical".
    • The CVEs that were created in the previous 30 days will instead get the status "New"
    • This will allow you to immediately test the CVE triage features
    • The status settings are normally preset by your defect system or other sustaining data
  • You can create a Vulnerability by:
    • Selecting a CVE
    • Click on "Create Vulnerability"
    • Follow the new link to the new Vulnerability
  • You can create an Investigation by:
    • Open a Vulnerability record
    • Click on "Add Product"
    • Select one or more Products, and click "Submit"
    • Observe that the products are added together with new respective Investigations
    • Follow the new link to the new Investigations
  • You can create a Defect by:
    • Open an Investigation record
    • Click on "Create Defect"
    • With the sample defect integration script, a simulated defect will be created and attached to the investigation


SRTool Primary Goals and Workflows

  • Goals
    • A common system to track and share security issues, combining community CVE's
    • A simple yet flexible interface for reporting and exploring the security issues
    • A place to upload and share attachments, including patches, fixes, emails, and documents
    • The ability to generate accurate and up-to-date reports and exports
    • A modular design for easy extension and adoption


  • Primary Workflows
    • Guide and manage the incoming CVE triage process
    • Attach CVEs to specific company products and defects
    • Automated updates from upstream CVEs and internal defect system
    • Ability to generate reports
    • Simple customization for organizations
    • Correlate CVEs with products and release (IN PROGRESS)
    • Correlate CVEs with specific customer builds (IN PROGRESS)


SRTool CVE Lifecycle

  • CVE are imported
    • New CVEs are imported and triaged from upstream (NIST/MITRE, 1000+/month)
    • Specific CVEs are requested by customers
  • CVEs are managed
    • CVEs are investigated
    • Vulnerable CVEs are assigned to products via defects
    • SRTool manager tracks the progress of open CVEs
    • SRTool manager produces reports as required by customers, field, and management
  • CVEs are resolved
    • CVEs are either fixed or declared not vulnerable
    • Releases are updated with the CVE fixes
    • Resolved CVEs are reported to customers, public website


Testing the Alternate Sample Organization 'ACME'

  • An example alternate organization (master application) called "acme" is included. To enable it do:

   $ # Include the quick development tools
   $ cp bin/dev_tools/* .
   $ # Reset 'acme' as the master app/organization
   $ ./stop.sh
   $ ./master_app.sh acme
   $ ./start.sh

  • When you browse the SRTool pages, will now observe several differences
    • In the top bar the text "[ACME]" will appear in several places. This shows how files like "lib/acme/templates/base.html" can take precedence over the default template files in "lib/srtgui".
    • In the top left corner the "ACME" logo will appear. This is from "SRTOOL_LOCAL_LOGO" being defined in "datasource.json".
    • If you click on the ACME logo you will be taken to an ACME hello page. This shows that "lib/acme/urls.py" has extended the URL map, plus how you can implement basic HTML pages in ACME's "lib/acme/views.py" and "lib/acme/templates/acme_hello.html".
    • In the ACME hello page, there are links to custom table pages for Products and Defects. This shows how to implement custom Toaster Tables via "lib/acme/urls.py" and "lib/acme/tables.py"
    • In the Products page, when you click "Export" a new report type "ACME Products Table" is appended. This shows how to extend and customize existing or new reports.
  • To restore the default "yp" master application, do the following:

   $ ./stop.sh
   $ ./master_app.sh yp
   $ ./start.sh

Adapting SRTool to your Organization

  • You can use the sample alternate organization "acme" as a guide to adapt SRTool to your organization
  • In this example, we will instantiate a new organization called "yoyodyne"
  • You should be able to place all of your local content into these two directories:
    • "./bin/yoyodyne": this is your master datasource directory
    • "./lib/yoyodyne": this is your master Django App directory
  • If you find you need to make changes in the core SRTool code, we recommend (in preferential order):
    • See if you can make it a generic feature so that it can become part of the mainline codebase
    • See if you can make an easy hook for the mainline code, so that your custom content can still reside in your directories
    • Talk to us on how to easily have managed custom patches to the mainline code (we do this in Toaster)

1) Instantiate your Organization

You can create a starting customization of the SRTool for your organization with these simple steps.

  • In this example, we will use the name "yoyo" as a shorthand for a mythical "Yoyodyne Corporation".

 $ cd /path_to/srtool_dir
 $ copy bin/dev_tools/* .
 $ ./stop.sh
 $ ./master_app.sh create yoyo
 $ ./start.sh

  • This script will perform the following actions:
    • Copy the sample 'bin/acme' directory to 'bin/yoyo'
    • Copy the sample 'lib/acme' directory to 'lib/yoyo'
    • Rename the respective Acme/ACME/acme labels and filenames to Yoyo/YOYO/yoyo
    • Set 'yoyo' as the new main application
  • You now have access to all the preset custom features described above for the ACME sample organization, now for Yoyodyne.

2) Add your defect system integration

You will need to implement a defect integration backend script to scan your defect system and instantiate the CVE defects in SRTool database, and then keep them up to date.

  • An example implementation template for Jira is provided here.

 bin/common/srtool_jira_template.py

  • You can use this to guide your integration to Jira or to other defect systems.
  • To see how to extend this template to your organization and yet maintain compatibility with upstream, see this section: Extending and customizing common/template code files
  • Please contact the SRTool developers for assistance in your defect system integration effort.

The main features of this example code, in addition to the Jira system access, is:

  • The "--init" command
    • This command that will initialize the SRTool database with your SRT related defects. Specifically, this command can used to bootstrap the SRTool database for your company's data
    • It will create an "Investigation" for each defect, and attach it to the respective product
    • It will create a "Vulnerability" to wrap the respective "Investigation"
    • If should find the respective CVE record if it exists, and attach that to the created "Vulnerability"
    • If a "Vulnerability" record already exists for the defect's CVE, then this script should attach the new "Investigation" to that "Vulnerability" record.
  • The "--update" command
    • This will scan your CVE-related defects
    • If the matching defect record in the SRTool database does not exist, then treat it like the "--init" command.
    • It will update the SRTool defect record with the current defect system state
    • If the defect has changed priority (or any other important state information), it will create a "Notification" event to the respective manager so that they can know that they need to act on the change.
  • The example code assumes that the CVE names are available via a text-based search, for example when the respective CVE name is in the defect title or comments. If your system uses a different marker, then you will need to code for that.


High Level Development Plan

Release Features Status/date
0.80 Multi-host testing, Django user model, Django 2.x support Released (December 2018)
0.90 Mapping of CVE-CPE > Packages > Recipes > Product/Release CPEs, Add Releases to Products, modular report definitions Under development (January 2019)
1.00 General Release Under development (February 2019)
1.10 Improved CPE/package mapping hueristics, advanced reports Under development (March+ 2019)


SRTool Bugzilla Page

Here is the link to the SRTool Bugzilla Page: SRTool Bugzilla Page



SRTool CGIT Page

Here is the link to the CGIT Page: SRTool CGIT Page



SRTool User Page

Here is the link to the User's Documentation page: SRTool UserPage



SRTool Developer Page

Here is the link to the Developer's page: SRTool Developer Page

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Contribute_to_SRTool&oldid=47304"