SRTool Developer Page

From Yocto Project
Jump to navigationJump to search


This page summarizes the Security Response Tool (SRTool) development process. We hope this will help you start contributing to the project. The SRTool is based on the Toaster codebase, so many of the process and debugging techniques apply.

Parent Wiki page: Contribute to SRTool


Design Philosophy

  • Make the tool easy to use
    • Support a web-based interface
    • Support backend scripts
    • Make the next-steps easy to find and understand
    • On-line documentation
  • Make the design modular
    • It must be easy to add new CVE upstream data sources
    • It must be easy to adapt the SRTool to a new orgainization
    • It must be easy to add new reports and exports
  • Make the design adaptable
    • Use the Django migration feature for on-the-fly updates
    • Make the data/interface table driven, remove hard coding as much as possible
  • Minimize the data in the database
    • Only put CVE summary information in the database
    • Extract CVE details on-the-fly from the source data files
    • Cache CVE data detail lookups for fast response
    • Only put Defect summary information in the database
    • Provide links to the Defect system, with its full defect data
  • Use scripts to bulk load/update the database
    • Scripts can update the database 10x faster than the Django DB interface
    • Scripts support background cron-job driven updates
  • Support data recovery
    • The system will fail, so be prepared
    • Provide data backup scripts, cron-job supportable
    • Provide backup data recovery
    • Make it easy to re-build the system from scratch
  • Support audit trails
    • Every user change should be trackable
    • It must be easy to generate a report on a CVE's action history
  • Support protected data
    • There must be a validated user model so that changes can be tracked
    • There must be a way to mark individual CVEs and there releated data private to specific users


SRTool Startup

This section describes how the SRTool gets started and initialized.

SRTool Start-up High-Level

High level actions from the "bin/srt" startup script.

Description Call Data file Comments
Check if the webport is already used "manage checksocket ..." ./.srtmain.pid If already used, warn and halt
Check the database and schema is up to date "manage migrate ..." ./srt.sqlite Create database if needed, assert migrations
Check database has fixtures and datasources "manage checksettings ..." Django/SRTool migration files Apply fixtures, apply un-read datasource data
Start the server "manage runserver ..." ./.srtmain.pid Start the server thread, open the webport

SRTool Start-up Details

Detailed actions from Django and "lib/*" startup scripts.

Description Code file Data file Comments
Check the database
Check if the webport is already used lib/srtmain/management/commands/checksocket.py .srtmain.pid If not, 'bin/srt' warns and halts
Check if the database exists ./bin/srt: "manage migrate" (Django) srt.sqlite If not, create empty database
Assert the Migrations ./bin/srt: "manage migrate" (Django) lib/*/migrations/* Process migration files (not already added)
Setup Django Environment
Django default settings ./bin/srt: "manage checksettings" (Django) lib/srtmain/settings.py Set the plug-ins, database connection, web connection, security, app priority
Apply the default fixture lib/orm/management/commands/checksettings.py lib/orm/fixtures/common.xml Set the default system database records
Apply the 'custom' fixture (if any) lib/orm/management/commands/checksettings.py lib/orm/fixtures/custom.xml Set the optional custom override system database records
Gather and sort the datasources lib/orm/management/commands/checksettings.py bin/*/datasource.json Common datasource is first, main app is last, rest sorted by 'key' value
Process the Data Sources
Process each datasource in order ... lib/orm/management/commands/lsupdates.py bin/*/datasource.json Add the datasource records, execute the 'init' function
1. process the initial 'common' datasource data lib/orm/management/commands/lsupdates.py bin/common/datasource.json Add the common datasource records, execute the 'init' function
2. process the 'NIST' datasources lib/orm/management/commands/lsupdates.py bin/nist/datasource.json Fetch and add the NIST CVEs
3. process the 'MITRE' datasources lib/orm/management/commands/lsupdates.py bin/mitre/datasource.json Fetch and add the MITRE CVEs
4. process the other datasources lib/orm/management/commands/lsupdates.py bin/*/datasource.json Fetch and add the other data sources
5. process the master app datasources lib/orm/management/commands/lsupdates.py bin/yp/datasource.json Fetch and add the master app data source (e.g. 'yp')
5a. Define the external Defect Manager bin/yp/srtool_yp.py bin/yp/srtool_defect.py The 'init' function can read CVE defects and pre-populate the Vulnerabilities and Investigations
5b. Define the products bin/yp/srtool_yp.py bin/yp/yocto-project-products.json The 'init' function can pre-populate the organization's Products
5c. Define the default users bin/yp/srtool_yp.py bin/yp/yocto-project-users.json The 'init' function can pre-populate the organization's users
6. process the final common datasource data lib/orm/management/commands/lsupdates.py bin/common/datasource.json Fetch and add the final data source (e.g. 'local cve', pre-score the 'new' CVEs)
Start the server
Start the server ./bin/srt: "manage runserver" (Django) .srtmain.pid Start the server thread, open the webport

SRTool Start and Restart Notes:

  • You can have as many SRTool instances as you wish (e.g. production, test, development, ...) as long as they use separate webports and separate directories (to avoid a collision on "./.srtmain.pid")
  • Fixture values are always applied during starts and restarts. They use absolute record ID locations, and should be used for rock-solid bootstrap values that are fixed for the life time of the database.
  • Datasource "srtsetting" values are like fixture values but use dynamic record locations, and are preferred for flexible and/or changeable data over the life of the database.
  • The datasource "Init" functions are only executed once. This happens at the first startup-up, and any new data sources get processed at the next re-start.



Major Components

Data Sources

The Data Source model is used to cleanly and dynamically connect external content (e.g. CVE sources) and managers (e.g. Defect Systems) to the SRTool.

Specifically, they define:

  • Upsteam/external data
  • Initialization actions
  • Update actions
  • Management actions
  • Backup actions


Here is the datasource schema:

name description
srtsetting (normally defined in master organization datasource)
SRTOOL_DOCUMENATION_URL The SRTool top bar documentation link
SRTOOL_LOCAL_LOGO The optional path to the top bar logo for the orgainization
SRTOOL_DEFECT_ADD Script: Add an existing defect to SRTool defect database, '--add defect'
SRTOOL_DEFECT_DEL Script: Delete an existing defect from SRTool defect database, '--del defect'
SRTOOL_DEFECT_NEW Script: create defect '--new program summary description priority components urllink'
SRTOOL_DEFECT_SAMPLENAME Displayed text schema of an example defect
SRTOOL_DEFECT_TOOL The registered script to manage defects
SRTOOL_DEFECT_URLBASE The url base lookup for the external defect tool
datasource
key Sort key for this datasource entry
data Type of data (e.g. 'cve', 'database_schema', 'package_keywords', 'notify_categories', ...)
source Soure of the data (e.g. 'common', 'nist' ,'mitre', ...)
name Name for this Data Source
description Description of this Data Source
cve_filter If CVE datasource, prefix of included CVEs (e.g. "CVE-", "CVE-2018", ...)
init The init function for STARTUP, either a script call or a file path
update The updated function for UPDATES, either a script call or a file path
lastModifiedDate Date the last time this source was updated
update_frequency How often the UPDATE function should be called
update_time Dictionary to refine the update time (see below)
srtuser ('common' only)
name Name for this built-in SRTool user
email Email for this built-in SRTool user (typically the admin's or empty)
role The role for this user
is_staff This user is built-in (i.e. set to 'True')
srtuser_groups (Django user model, 'common' only)
srtuser Name of above 'srtuser'
group Name of below 'group'
permissions (Django user model, 'common' only)
content_type_id Unique record ID for this permission
codename Code Name for this permission (e.g. "Reader", "Admin", ...)
name Name for this permission
groups (Django user model, 'common' only)
name Name for this group (e.g. "Reader", "Admin", ...)
group_permissions (Django user model, 'common' only)
group Group for this group (e.g. "Reader", "Admin", ...)
permission Above 'permission' name


Defect System Integration

  • Ability to import CVE affected defects from Defect system
  • Ability to create new CVE affected defects to Defect system
  • Ability to update CVE status from Defect system


Report/Export System

  • Ability to export respective data from any page
    • The "Export" button is available for all pages in the top bar.
    • Most pages already have a default report setup, as seen in "lib/srtgui/reports.py"
    • Additional pages can have a report defined via the class ReportManager() in "lib/srtgui/reports.py"
    • The page's respective report is passed via "request.resolver_match.url_name"
  • Ability to generate reports in multiple formats (e.g. text, CSV)
    • Most pages already have a default "Text" and "CSV" format definded in "lib/srtgui/reports.py"
  • Ability to easily add new reports and new output formats
    • Examine "lib/acme/reports.py" to see how to easily extend and/or replace existing report implementations for custom master applications


Automatic and manual data source updates

  • The SRTool has a facility to regularly refresh the upstream data, for example CVEs and Defect status
  • The update job runs in the background, and is started and stopped together with the SRT web server
  • Each data source can (a) select the update frequency, and (b) define the desired action in the 'update' field, typically a script call
  • The update time can be refined with the 'update_time' filter value dictionary:

 update_frequency update_time (optional)
 ================ ======================================================
 MINUTELY = 0     "{\"minutes\":\"10\"}"  # every ten minutes
 HOURLY = 1       "{\"minute\":\"10\"}"   # at ten minutes past the hour
 DAILY = 2        "{\"hour\":\"2\"}"      # at 2 hours after midnight
 WEEKLY = 3       "{\"weekday\":\"6\",\"hour\":\"2\"}"  # day of week, hour
 MONTHLY = 4      "{\"day\":\"1\"\"hour\":\"2\"}"       # day of month
 ONDEMAND = 5     "{}"                    # only on demand
 ONSTARTUP = 6    "{}"                    # on every SRTool start up

  • The most recent status of the update scan is available in this file:

 $ cat update_logs/update_status.log 
 ====================
 Update: Date=2019-01-14 14:46:36,Filter='all',Force=False
   Skip 'Weekly archive database backup': update time not reached (2019-01-26 14:13:43)
   Skip 'Daily archive database backup': update time not reached (2019-01-15 14:13:43)
   Skip 'Score CVEs': update time not reached (2019-01-14 14:48:23)
   Skip 'MITRE 2016': update time not reached (2019-01-26 13:36:45)
   Skip 'MITRE 2019': update time not reached (2019-01-26 13:36:45)
 ...

  • The developer can also perform direct updates
  • Here is the list of manual update options

 $ # Display options (with the 'manage' options removed for clarity)
 $ ./bin/srt manage update -h
 SRT_MAIN_APP=yp
 usage: manage.py update [-h] [-v {0,1,2,3}]
     [--cron-start] [--cron-stop] [--list] [--run-updates]
     [--force]
     [--name-filter NAME_FILTER [NAME_FILTER ...]]
     [--verbose] [--trial]
 optional arguments:
     -h, --help            show this help message and exit
     -v {0,1,2,3}, --verbosity {0,1,2,3}
                           Verbosity level; 0=minimal output, 1=normal output,
                           2=verbose output, 3=very verbose output
     --cron-start          Start the SRTool background updater
     --cron-stop           Stop the SRTool background updater
     --list, -l            List data sources
     --run-updates, -u     update scheduled data sources
     --force, -f           Force the update
     --name-filter NAME_FILTER [NAME_FILTER ...], -n NAME_FILTER [NAME_FILTER ...]
                           Filter for datasource name
     --verbose             Debugging: verbose output
     --trial, -t           Debugging: trial run

  • See the list of available update data sources

 $ ./bin/srt manage update --list
 SRTool Update List:
            Data  Source           Name  Frequency                       Offset Description
 =============== ======= ============== ========== ============================ ===========================================
 database_schema  common     SRT Schema  OnStartup                           {} SRT database schema for command line tools
   backup_weekly  common  Weekly Backup     Weekly   {"weekday":"5","hour":"2"} Weekly archive database backup
    backup_daily  common   Daily Backup      Daily                 {"hour":"7"} Daily archive database backup
      score_cves  common          Score     Minute             {"minutes":"10"} Score CVEs
             cve   mitre          MITRE     Weekly   {"weekday":"5","hour":"2"} MITRE 2016
             cve   mitre          MITRE     Weekly   {"weekday":"5","hour":"2"} MITRE 2019
 ...

  • Update all sources (with optional "--force" option to update now)

 $ ./bin/srt manage update --run-updates [--force]
 BEGINNING UPDATING DATASOURCES... this MAY take a long time
 SRTool Update: time_now = 2019-01-14 20:18:12
 Update required	...	executing 'bin/common/srtool_common.py --score-new-cves NEW'
 0030:        CVE-2019-0008
 ...

  • The '--name-filter' can do matches again any of the 'Description','Name','Source','Data' fields

 $ # Update all 'nist' source types
 $ ./bin/srt manage update --run-updates -n nist
 $ # Update all 'cve' data types
 $ ./bin/srt manage update --run-updates -n cve
 $ # Update the specific "NIST 2019" source by its description
 $ ./bin/srt manage update --run-updates -n "NIST 2019"

  • You can disable the automatic updater job, and start/stop it manually

 $ export SRT_SKIP_AUTOUPDATE=1
 $ ./bin/srt start
--- or ---
 $ ./bin/srt start noautoupdate

 $ # Manually Start the update server job
 $ ... do work ...
 $ ./bin/srt manage update --cron-start
 $ ... do work ...
 $ ./bin/srt manage update --cron-stop
 $ ./bin/srt stop


Data backup and Recovery

  • Ability to regularly backup data via scripts and a CRON job
  • Ability to restore data


SRTool Code Organization

  • All of the scripts are under the "/bin" directory
  • All of the Django files are under the "/lib" directory
  • All of the GUI files are under the "/lib/srtgui" directory
  • All of the database management files are under the "/lib/orm" directory (object-relationhip-model)
  • All of the user management files are under "/lib/user" and "lib/srtmain/templates"
  • The GUI starts from "lib/srtmain/urls.py"


Debugging the SRTool

1. Debugging Techniques

The same basic techniques for debugging Toaster also apply to the SRTool. See this link for details Debugging_Toaster.


2. Debugging Log Files

Debugging output:

Type Output Location
Start up errors All progress and error messages appear in the terminal shell
Django template and processing errors ./srt_web.log
Normal SRTool exceptions and general execution messages ./srt.log 
(via 'logger.[warning|debug|info](msg)')
Short term debugging and trace messages ./srt_dbg.log 
(via '_log(msg)')


3. Environment Variables

There are several environment settings that can be used to help the debug cycle and to present functional values.

To set the productions values, you can run the helper script: 

 $ source ./srt_env.sh

To set the debugging values, you can run the helper script: 

 $ source ./srt_env.sh debug

Optional Environment Setting Overrides:

Name Purpose Default Production Debugging
SRT_PORT Default webport value 9000 9000 9900
SRTDBG_LOG Log file for quick debug messages ("_log(...)") /tmp/srt_dbg.log `pwd`/srt_dbg.log `pwd`/srt_dbg.log
SRTDBG_MINIMAL_DB On start up, only load 10 records from each source, useful for fast schema and functional testing 0 0 1
SRTDBG_SKIP_DEFECT_IMPORT Skip loading the Defect data, useful for fast startup testing 0 0 1
SRTDBG_SKIP_CVE_IMPORT Skip loading the CVE data, useful for fast startup testing 0 0 0
SRT_SKIP_AUTOUPDATE Skip starting the background database updater job 0 0 1
SRT_SMTP Email setup <empty> Example: smtp.acme.com "
SRT_USER User name for SMTP and Defect server <empty> Example: srt_user "
SRT_PASSWD User password for SMTP and Defect server <empty> Example: srt_rocks! "

Notes:

  • Holding the user name and password as environment variables is a good way to hide them from external users that may have read access to the SRTool file system


Running Pylint on the SRTool Codebase

Is is recommended/required to run pylint on all new code.

Here is an example on how to install and run pylint on an Ubuntu host: 

 $ sudo apt install pylint3
 $ pip3 install pylint_django
 $ cd /path/to/srtool/clone
 $ PYTHONPATH=bin/:lib/ pylint3 --load-plugins pylint_django -E lib


Quick Development Tools

A set of quick development assistance tools are available under "bin/dev_tools". To make them available, run: 

 $ cp bin/dev_tools/* .

Here is a summary of their functions:

Name Function
./start.sh Script to start the SRTool. Edit the webport value for different directories/instances (production, development, ...)
./stop.sh Script to stop the SRTool
./restart.sh Script to restart the SRTool, useful for asserting run time code changes (models, views, tables, basically any change to a python file)
./recreate.sh Script to force a new SRTool from scratch, useful for startup/schema testing (especially when combined with "export SRTDBG_MINIMAL_DB=1")
./tail.sh [line_count] Script to dump the tail from the many log files, useful in finding exception messages
source ./srt_env.sh Script to source the SRTool environment values, useful for seeing and updating the development overrides
./super.sh Script to set up a superuser account ("./bin/srt manage createsuperuser")
./lssrt.sh List of running SRTool instances, including their PID and webport
./showdb.sh Script to start a SQL GUI (e.g. 'sqlitebrowser'), useful for quick database content checking and minor adjustments
./masterapp.sh appname Script to reset the master app (for example './masterapp.sh acme' and reset with './masterapp.sh yp')

There is also a basic sanity script to test the host settings, check that the database passed the startup and was initialized: 

 $ ./bin/common/srtool_sanity_test.py -i
 Uname       = #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64
 Django      = (2, 2, 0, 'alpha', 0)
 Python3     = Python 3.5.2
 Sqlite3     = 3.11.0 2016-02-15
 Server PID  = 8654
 Server Port = [8654](python3 /opt/dreyna_srt/srt_django20/lib/manage.py runserver --noreload 0.0.0.0:9993)
 CVEs        = 64
 Users       = 15
 Data sources= 25
 $



Submitting Patches to the SRTool via GitLab

GitLab's hosted service has automatic repository mirroring, and it is very easy to fetch the lastest SRTool updates and share patch branches with the SRTool developers.

1) Open your gitlab.com account. You can create one if you do not have one already (they're free!).

2) Do the following:

This ensures that the master branch in the project is in sync with master at "git://git.yoctoproject.org/srtool", and then create your own branches for work.

3) Start your patch branch off of master with this suggested branch naming style: 

  username/srtool/FeatureOrBug

4) Do Work

5) Test the changes.

6) Rebase on master. It has probably changed while you were working (unless you are really really fast!)

7) Commit your change

NOTE: The format of the commit message should be like this 

    srtool: <module> <short one line summary>

    long(er) description

    [YOCTO #0000]

    Signed-off-by: First Last <name@domain.com>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your "git commit -s" credentials.

8) Share your branch's link with the SRTool maintainers, e.g.:

  • david.reyna@windriver.com
  • yocto-security@yoctoproject.org

Note: to use "yocto-security" you will first need to subscribe here: "https://lists.yoctoproject.org/listinfo/yocto-security"

9) NOTE: when the patch has been accepted upstream, you can refresh your master branch and delete your patch branch.

Submitting Patches to the SRTool via poky-contrib

To contribute to the SRTool with this method you will also need authorization to write to the upstream yocto project repository "poky-contrib". Contact a member of the SRTool team for details.

1) Download the master branch of the SRTool 

  git pull ssh://git@push.yoctoproject.org/srtool && cd srtool

2) Add poky-contrib to the local repository you set up above 

  git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib

3) Fetch the poky-contrib branches 

  git fetch --all

4) Start your feature branch off of master, name style of branch is convention, but suggested. 

  git checkout -b username/srtool/FeatureOrBug origin/master

5) Do Work

6) Test the changes.

7) Rebase on master. It has probably changed while you were working (unless you are really really fast!) 

  git rebase origin/master

8) Commit your change

NOTE: The format of the commit message should be like this 

    srtool: <module> <short one line summary>

    long(er) description

    [YOCTO #0000]

    Signed-off-by: First Last <name@domain.com>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.

9) Push your feature branch to poky-contrib 

 git push -u poky-contrib username/srtool/FeatureOrBug:username/srtool/FeatureOrBug

10) Send to the srtool-mailing list:

We accept patches on the Security mailing list ( yocto-security@yoctoproject.org ) by "git send-email".

e.g. 

   $ git send-email HEAD^

11) NOTE: when the patch has been accepted upstream, you can clean up your poy-contrib branch with: 

  git push -u poky-contrib :username/srtool/FeatureOrBug
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=SRTool_Developer_Page&oldid=46075"