SRTool Developer Page

This page summarizes the Security Response Tool (SRTool) development process. We hope this will help you start contributing to the project. The SRTool is based on the Toaster codebase, so many of the process and debugging techniques apply.

Design Philosophy

• Make the tool easy to use
  • Support a web-based interface
  • Support backend scripts
  • Make the next-steps easy to find and understand
  • On-line documentation
• Make the design modular
  • It must be easy to add new CVE upstream data sources
  • It must be easy to adapt the SRTool to a new orgainization
  • It must be easy to add new reports and exports
• Make the design adaptable
  • Use the Django migration feature for on-the-fly updates
  • Make the data/interface table driven, remove hard coding as much as possible
• Minimize the data in the database
  • Only put CVE summary information in the database
  • Extract CVE details on-the-fly from the source data files
  • Cache CVE data detail lookups for fast response
  • Only put Defect summary information in the database
  • Provide links to the Defect system, with its full defect data
• Use scripts to bulk load/update the database
  • Scripts can update the database 10x faster than the Django DB interface
  • Scripts support background cron-job driven updates
• Support data recovery
  • The system will fail, so be prepared
  • Provide data backup scripts, cron-job supportable
  • Provide backup data recovery
  • Make it easy to re-build the system from scratch
• Support audit trails
  • Every user change should be trackable
  • It must be easy to generate a report on a CVE's action history
• Support protected data
  • There must be a validated user model so that changes can be tracked
  • There must be a way to mark individual CVEs and there releated data private to specific users

SRTool Startup

This section describes how the STool gets started and initialized.

SRTool Start-up High-Level

High level actions from the "bin/srt" startup script.

Description	Call	Data file	Comments
Check if the webport is already used	"manage checksocket ..."	.srtmain.pid	If already used, warn and halt
Check the database and schema is up to date	"manage migrate ..."	srt.sqlite	Create database if needed, assert migrations
Check database has fixtures and datasources	"manage checksettings ..."	Django/SRTool migration files	Apply fixtures, apply un-read datasource data
Start the server	"manage runserver ..."	.srtmain.pid	Start the server thread, open the webport

SRTool Start-up Details

Detailed actions from Django and "lib/*" startup scripts.

Description	Code file	Data file	Comments
Check the database
Check if the webport is already used	lib/srtmain/management/commands/checksocket.py	.srtmain.pid	If not, 'bin/srt' warns and halts
Check if the database exists	"manage migrate" (Django)	srt.sqlite	If not, create empty database
Assert the Migrations	"manage migrate" (Django)	lib/*/migrations/*	Process migration files (not already added)
Setup Django Environment
Django default settings	"manage checksettings" (Django)	lib/srtmain/settings.py	Set the plug-ins, database connection, web connection, security, app priority
Apply the default fixture	lib/orm/management/commands/checksettings.py	lib/orm/fixtures/common.xml	Set the default system database records
Apply the 'custom' fixture (if any)	lib/orm/management/commands/checksettings.py	lib/orm/fixtures/custom.xml	Set the optional custom override system database records
Gather and sort the datasources	lib/orm/management/commands/checksettings.py	bin/*/datasource.json	Common datasource is first, main app is last, rest sorted by 'key' value
Process the Data Sources
Process each datasource in order ...	lib/orm/management/commands/lsupdates.py	bin/*/datasource.json	Add the datasource records, execute the 'init' function
1. process the initial 'common' datasource data	lib/orm/management/commands/lsupdates.py	bin/common/datasource.json	Add the common datasource records, execute the 'init' function
2. process the 'NIST' datasources	lib/orm/management/commands/lsupdates.py	bin/nist/datasource.json	Fetch and add the NIST CVEs
3. process the 'MITRE' datasources	lib/orm/management/commands/lsupdates.py	bin/mitre/datasource.json	Fetch and add the MITRE CVEs
4. process the other datasources	lib/orm/management/commands/lsupdates.py	bin/*/datasource.json	Fetch and add the other data sources
5. process the master app datasources	lib/orm/management/commands/lsupdates.py	bin/yp/datasource.json	Fetch and add the master app data source (e.g. 'yp')
5a. Define the external Defect Manager	bin/yp/srtool_yp.py	bin/yp/srtool_defect.py	The 'init' function can read CVE defects and pre-populate the Vulnerabilities and Investigations
5b. Define the products	bin/yp/srtool_yp.py	bin/yp/yocto-project-products.json	The 'init' function can pre-populate the organization's Products
5c. Define the default users	bin/yp/srtool_yp.py	bin/yp/yocto-project-users.json	The 'init' function can pre-populate the organization's users
6. process the final common datasource data	lib/orm/management/commands/lsupdates.py	bin/common/datasource.json	Fetch and add the final data source (e.g. 'local cve', pre-score the 'new' CVEs)
Start the server
Start the server	"manage runserver" (Django)	.srtmain.pid	Start the server thread, open the webport

SRTool Start and Restart Note

• You can have as many SRTool instances as you wish (e.g. production, test, development, ...) as long as they use separate webports and separate directories (to avoid a collision on "./.srtmain.pid")
• Fixture values are always applied during starts and restarts. They use absolute record ID locations, and should be used for rock-solid bootstrap values that are fixed for the life time of the database.
• Datasource "srtsetting" values are like fixture values but use dynamic record locations, and are preferred for flexible and/or changeable data over the life of the database.
• The datasource "Init" functions are only executed once. This usually happens at the first startup-up, but new datasources get processed at the next re-start.

Major Components

Data Sources

The Data Source model is used to cleanly and dynamically connect external content (e.g. CVE sources) and managers (e.g. Defect Systems) to the SRTool.

Specifically, they define:

• Upsteam/external data
• Initialization actions
• Update actions
• Management actions
• Backup actions

Here is the datasource schema:

name	description
srtsetting (default value)
datasource_org_default (would be overwritten by master organization datasource 'datasource_org')	Default bin directory for the Organization (by default is "bin/yp")
srtsetting (normally defined in master organization datasource)
datasource_org (force defines the master organization)	The bin directory for the master organization datasource
SRTOOL_DOCUMENATION_URL	The SRTool top bar documentation link
SRTOOL_LOCAL_LOGO	The optional path to the top bar logo for the orgainization
SRTOOL_DEFECT_ADD	Script: Add an existing defect to SRTool defect database, '--add defect'
SRTOOL_DEFECT_DEL	Script: Delete an existing defect from SRTool defect database, '--del defect'
SRTOOL_DEFECT_NEW	Script: create defect '--new program summary description priority components urllink'
SRTOOL_DEFECT_SAMPLENAME	Displayed text schema of an example defect
SRTOOL_DEFECT_TOOL	The registered script to manage defects
SRTOOL_DEFECT_URLBASE	The url base lookup for the external defect tool
datasource
key	Sort key for this datasource entry
data	Type of data (e.g. 'cve', 'database_schema', 'package_keywords', 'notify_categories', ...)
source	Soure of the data (e.g. 'common', 'nist' ,'mitre', ...)
name	Name for this Data Source
description	Description of this Data Source
cve_filter	If CVE datasource, prefix of included CVEs (e.g. "CVE-", "CVE-2018", ...)
init	The init function for STARTUP, either a script call or a file path
update	The updated function for UPDATES, either a script call or a file path
lastModifiedDate	Date for the source data (e.g. for NIST data files)
update_frequency	How often the UPDATE function should be called
update_time	Last time the UPDATE function was called
srtuser (typically 'common' only)
name	Name for this built-in SRTool user
email	Email for this built-in SRTool user (typically the admin's or empty)
role	The role for this user
is_staff	If this user is staff and thus built-in (typically 'True')
srtuser_groups (Django user model, 'common' only)
srtuser	Name of above 'srtuser'
group	Name of below 'group'
permissions (Django user model, 'common' only)
content_type_id	Unique record ID for this permission
codename	Code Name for this permission (e.g. "Reader", "Admin", ...)
name	Name for this permission
groups (Django user model, 'common' only)
name	Name for this group (e.g. "Reader", "Admin", ...)
group_permissions (Django user model, 'common' only)
group	Group for this group (e.g. "Reader", "Admin", ...)
permission	Above 'permission' name

Defect System Integration

• Ability to import CVE affected defects from Defect system
• Ability to create new CVE affected defects to Defect system
• Ability to update CVE status from Defect system

Report/Export System

• Ability to export respective data from any page
• Ability to generate reports in multiple formats (e.g. text, CSV)
• Ability to easily add new reports and new output formats

Data backup and Recovery

• Ability regularly backup data via scripts and a CRON job
• Ability to restore data

SRTool Code Organization

• All of the scripts are under the "/bin" directory
• All of the Django files are under the "/lib" directory
• All of the GUI files are under the "/lib/srtgui" directory
• All of the database management files are under the "/lib/orm" directory (object-relationhip-model)
• All of the user management files are under "/lib/user" and "lib/srtmain/templates"
• The GUI starts from "lib/srtmain/urls.py"