SRTool Developer Page
From Yocto Project
Jump to navigationJump to search
This page summarizes the Security Response Tool (SRTool) development process. We hope this will help you start contributing to the project. The SRTool is based on the Toaster codebase, so many of the process and debugging techniques apply.
Design Philosophy
- Make the tool easy to use
- Support a web-based interface
- Support backend scripts
- Make the next-steps easy to find and understand
- On-line documentation
- Make the design modular
- It must be easy to add new CVE upstream data sources
- It must be easy to adapt the SRTool to a new orgainization
- It must be easy to add new reports and exports
- Make the design adaptable
- Use the Django migration feature for on-the-fly updates
- Make the data/interface table driven, remove hard coding as much as possible
- Minimize the data in the database
- Only put CVE summary information in the database
- Extract CVE details on-the-fly from the source data files
- Cache CVE data detail lookups for fast response
- Only put Defect summary information in the database
- Provide links to the Defect system, with its full defect data
- Use scripts to bulk load/update the database
- Scripts can update the database 10x faster than the Django DB interface
- Scripts support background cron-job driven updates
- Support data recovery
- The system will fail, so be prepared
- Provide data backup scripts, cron-job supportable
- Provide backup data recovery
- Make it easy to re-build the system from scratch
- Support audit trails
- Every user change should be trackable
- It must be easy to generate a report on a CVE's action history
- Support protected data
- There must be a validated user model so that changes can be tracked
- There must be a way to mark individual CVEs and there releated data private to specific users
SRTool Startup
This section describes how the SRTool gets started and initialized.
SRTool Start-up High-Level
High level actions from the "bin/srt" startup script.
| Description | Call | Data file | Comments |
|---|---|---|---|
| Check if the webport is already used | "manage checksocket ..." | ./.srtmain.pid | If already used, warn and halt |
| Check the database and schema is up to date | "manage migrate ..." | ./srt.sqlite | Create database if needed, assert migrations |
| Check database has fixtures and datasources | "manage checksettings ..." | Django/SRTool migration files | Apply fixtures, apply un-read datasource data |
| Start the server | "manage runserver ..." | ./.srtmain.pid | Start the server thread, open the webport |
SRTool Start-up Details
Detailed actions from Django and "lib/*" startup scripts.
| Description | Code file | Data file | Comments |
|---|---|---|---|
| Check the database | |||
| Check if the webport is already used | lib/srtmain/management/commands/checksocket.py | .srtmain.pid | If not, 'bin/srt' warns and halts |
| Check if the database exists | ./bin/srt: "manage migrate" (Django) | srt.sqlite | If not, create empty database |
| Assert the Migrations | ./bin/srt: "manage migrate" (Django) | lib/*/migrations/* | Process migration files (not already added) |
| Setup Django Environment | |||
| Django default settings | ./bin/srt: "manage checksettings" (Django) | lib/srtmain/settings.py | Set the plug-ins, database connection, web connection, security, app priority |
| Apply the default fixture | lib/orm/management/commands/checksettings.py | lib/orm/fixtures/common.xml | Set the default system database records |
| Apply the 'custom' fixture (if any) | lib/orm/management/commands/checksettings.py | lib/orm/fixtures/custom.xml | Set the optional custom override system database records |
| Gather and sort the datasources | lib/orm/management/commands/checksettings.py | bin/*/datasource.json | Common datasource is first, main app is last, rest sorted by 'key' value |
| Process the Data Sources | |||
| Process each datasource in order ... | lib/orm/management/commands/lsupdates.py | bin/*/datasource.json | Add the datasource records, execute the 'init' function |
| 1. process the initial 'common' datasource data | lib/orm/management/commands/lsupdates.py | bin/common/datasource.json | Add the common datasource records, execute the 'init' function |
| 2. process the 'NIST' datasources | lib/orm/management/commands/lsupdates.py | bin/nist/datasource.json | Fetch and add the NIST CVEs |
| 3. process the 'MITRE' datasources | lib/orm/management/commands/lsupdates.py | bin/mitre/datasource.json | Fetch and add the MITRE CVEs |
| 4. process the other datasources | lib/orm/management/commands/lsupdates.py | bin/*/datasource.json | Fetch and add the other data sources |
| 5. process the master app datasources | lib/orm/management/commands/lsupdates.py | bin/yp/datasource.json | Fetch and add the master app data source (e.g. 'yp') |
| 5a. Define the external Defect Manager | bin/yp/srtool_yp.py | bin/yp/srtool_defect.py | The 'init' function can read CVE defects and pre-populate the Vulnerabilities and Investigations |
| 5b. Define the products | bin/yp/srtool_yp.py | bin/yp/yocto-project-products.json | The 'init' function can pre-populate the organization's Products |
| 5c. Define the default users | bin/yp/srtool_yp.py | bin/yp/yocto-project-users.json | The 'init' function can pre-populate the organization's users |
| 6. process the final common datasource data | lib/orm/management/commands/lsupdates.py | bin/common/datasource.json | Fetch and add the final data source (e.g. 'local cve', pre-score the 'new' CVEs) |
| Start the server | |||
| Start the server | ./bin/srt: "manage runserver" (Django) | .srtmain.pid | Start the server thread, open the webport |
SRTool Start and Restart Notes:
- You can have as many SRTool instances as you wish (e.g. production, test, development, ...) as long as they use separate webports and separate directories (to avoid a collision on "./.srtmain.pid")
- Fixture values are always applied during starts and restarts. They use absolute record ID locations, and should be used for rock-solid bootstrap values that are fixed for the life time of the database.
- Datasource "srtsetting" values are like fixture values but use dynamic record locations, and are preferred for flexible and/or changeable data over the life of the database.
- The datasource "Init" functions are only executed once. This usually happens at the first startup-up, but new datasources get processed at the next re-start.
Major Components
Data Sources
The Data Source model is used to cleanly and dynamically connect external content (e.g. CVE sources) and managers (e.g. Defect Systems) to the SRTool.
Specifically, they define:
- Upsteam/external data
- Initialization actions
- Update actions
- Management actions
- Backup actions
Here is the datasource schema:
| name | description |
|---|---|
| srtsetting (default value) | |
| datasource_org_default (would be overwritten by master organization datasource 'datasource_org') | Default bin directory for the Organization (by default is "bin/yp") |
| srtsetting (normally defined in master organization datasource) | |
| datasource_org (force defines the master organization) | The bin directory for the master organization datasource |
| SRTOOL_DOCUMENATION_URL | The SRTool top bar documentation link |
| SRTOOL_LOCAL_LOGO | The optional path to the top bar logo for the orgainization |
| SRTOOL_DEFECT_ADD | Script: Add an existing defect to SRTool defect database, '--add defect' |
| SRTOOL_DEFECT_DEL | Script: Delete an existing defect from SRTool defect database, '--del defect' |
| SRTOOL_DEFECT_NEW | Script: create defect '--new program summary description priority components urllink' |
| SRTOOL_DEFECT_SAMPLENAME | Displayed text schema of an example defect |
| SRTOOL_DEFECT_TOOL | The registered script to manage defects |
| SRTOOL_DEFECT_URLBASE | The url base lookup for the external defect tool |
| datasource | |
| key | Sort key for this datasource entry |
| data | Type of data (e.g. 'cve', 'database_schema', 'package_keywords', 'notify_categories', ...) |
| source | Soure of the data (e.g. 'common', 'nist' ,'mitre', ...) |
| name | Name for this Data Source |
| description | Description of this Data Source |
| cve_filter | If CVE datasource, prefix of included CVEs (e.g. "CVE-", "CVE-2018", ...) |
| init | The init function for STARTUP, either a script call or a file path |
| update | The updated function for UPDATES, either a script call or a file path |
| lastModifiedDate | Date for the source data (e.g. for NIST data files) |
| update_frequency | How often the UPDATE function should be called |
| update_time | Last time the UPDATE function was called |
| srtuser ('common' only) | |
| name | Name for this built-in SRTool user |
| Email for this built-in SRTool user (typically the admin's or empty) | |
| role | The role for this user |
| is_staff | This user is built-in (i.e. set to 'True') |
| srtuser_groups (Django user model, 'common' only) | |
| srtuser | Name of above 'srtuser' |
| group | Name of below 'group' |
| permissions (Django user model, 'common' only) | |
| content_type_id | Unique record ID for this permission |
| codename | Code Name for this permission (e.g. "Reader", "Admin", ...) |
| name | Name for this permission |
| groups (Django user model, 'common' only) | |
| name | Name for this group (e.g. "Reader", "Admin", ...) |
| group_permissions (Django user model, 'common' only) | |
| group | Group for this group (e.g. "Reader", "Admin", ...) |
| permission | Above 'permission' name |
Defect System Integration
- Ability to import CVE affected defects from Defect system
- Ability to create new CVE affected defects to Defect system
- Ability to update CVE status from Defect system
Report/Export System
- Ability to export respective data from any page
- Ability to generate reports in multiple formats (e.g. text, CSV)
- Ability to easily add new reports and new output formats
Data backup and Recovery
- Ability regularly backup data via scripts and a CRON job
- Ability to restore data
SRTool Code Organization
- All of the scripts are under the "/bin" directory
- All of the Django files are under the "/lib" directory
- All of the GUI files are under the "/lib/srtgui" directory
- All of the database management files are under the "/lib/orm" directory (object-relationhip-model)
- All of the user management files are under "/lib/user" and "lib/srtmain/templates"
- The GUI starts from "lib/srtmain/urls.py"
Debugging the SRTool
1. Debugging Techniques
The same basic techniques for debugging Toaster also apply to the SRTool. See this link for details Debugging_Toaster.
2. Debugging Log Files
Debugging output:
| Type | Output Location |
|---|---|
| Start up errors | All progress and error messages appear in the terminal shell |
| Django template and processing errors | ./srt_web.log |
| Normal SRTool exceptions and general execution messages | ./srt.log (via 'logger.[warning|debug|info](msg)') |
| Short term debugging and trace messages | ./srt_dbg.log (via '_log(msg)') |
3. Environment Variables
There are several environment settings that can be used to help the debug cycle and to present functional values.
To set the productions values, you can run the helper script:
$ source ./srt_env.sh
To set the debugging values, you can run the helper script:
$ source ./srt_env.sh debug
Environment Settings:
| Name | Purpose | Production | Debugging |
|---|---|---|---|
| SRT_PORT | Default webport value | 9000 | 9900 |
| SRTDBG_LOG | Log file for quick debug messages ("_log(...)") | `pwd`/srt_dbg.log | `pwd`/srt_dbg.log |
| SRTDBG_MINIMAL_DB | On start up, only load 10 records from each source, useful for fast schema and functional testing | 0 | 1 |
| SRTDBG_SKIP_DEFECT_IMPORT | Skip loading the Defect data, useful for fast startup testing | 0 | 1 |
| SRTDBG_SKIP_CVE_IMPORT | Skip loading the CVE data, useful for fast startup testing | 0 | 0 |
| SRT_SMTP | Email setup | Example: smtp.acme.com | " |
| SRT_USER | User name for SMTP and Defect server | Example: srt_user | " |
| SRT_PASSWD | User password for SMTP and Defect server | Example: srt_rocks! | " |
Notes:
- Holding the user name and password as environment variables is a good way to hide them external users that may have read access to the SRTool file system
Running Pylint on the SRTool Codebase
Is is recommended/required to run pylint on all new code.
Here is an example on how to install and run pylint on an Ubuntu host:
$ sudo apt install pylint3
$ pip3 install pylint_django
$ cd /path/to/srtool/clone
$ PYTHONPATH=bin/:lib/ pylint3 --load-plugins pylint_django -E lib
Quick Development Tools
A set of quick development assistance tools are available under "bin/dev_tools". To make them available, run:
$ cp bin/dev_tools/* .
Here is a summary of their functions:
| Name | Function |
|---|---|
| ./start.sh | Script to start the SRTool. Edit the webport value for different directories/instances (production, development, ...) |
| ./stop.sh | Script to stop the SRTool |
| ./recreate.sh | Script to force a new SRTool from scratch, useful for startup/schema testing |
| ./restart.sh | Script to restart the SRTool, useful for asserting run time code changes (models, views, tables) |
| ./tail.sh [line_count] | Script to dump the tail from the many log files, useful to find exception messages |
| source ./srt_env.sh | Script to source the SRTool environment values, useful for seeing and updating the development overrides |
| ./lssrt.sh | List of running SRTool instances, including their PID and webport |
| ./showdb.sh | Script to start a SQL GUI (e.g. 'sqlitebrowser'), useful for quick database content checking and minor adjustments |
There is also a basic sanity script to test the host settings, check that the database passed the startup and was initialized:
$ ./bin/common/srtool_sanity_test.py -i
Uname = #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64
Django = (2, 2, 0, 'alpha', 0)
Python3 = Python 3.5.2
Sqlite3 = 3.11.0 2016-02-15
Server PID = 8654
Server Port = [8654](python3 /opt/dreyna_srt/srt_django20/lib/manage.py runserver --noreload 0.0.0.0:9993)
CVEs = 64
Users = 15
Data sources= 25
$
Submitting Patches to the SRTool
To contribute to the SRTool you will also need authorization to write to the upstream yocto project repository. Contact a member of the SRTool team for details.
1) Download the master branch of the SRTool
git pull ssh://git@push.yoctoproject.org/srtool && cd srtool
2) Add poky-contrib to the local repository you set up above
git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib
3) Fetch the poky-contrib branches
git fetch --all
4) Start your feature branch off of master, name style of branch is convention, but suggested.
git checkout -b username/srtool/FeatureOrBug origin/master
5) Do Work
6) Test the changes.
7) Rebase on master. It has probably changed while you were working (unless you are really really fast!)
git rebase origin/master
8) Commit your change
NOTE: The format of the commit message should be like this
srtool: <module> <short one line summary>
long(er) description
[YOCTO #0000]
Signed-off-by: First Last <name@domain.com>
Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.
9) Push your feature branch to poky-contrib
git push -u poky-contrib username/srtool/FeatureOrBug:username/srtool/FeatureOrBug
10) Send to the srtool-mailing list:
We accept patches on the Security mailing list ( yocto-security@yoctoproject.org ) by "git send-email".
e.g.
$ git send-email HEAD^
11) NOTE: when the patch has been accepted upstream, you can clean up your poy-contrib branch with:
git push -u poky-contrib :username/srtool/FeatureOrBug
