Yocto Project - User contributions [en]

Webhob REST API Tasks

2013-08-02T13:25:15Z

Calin Dragomir:

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /tasks || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || -
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || "<FIELD>:<ORDER_DIRECTION>"
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].model || string || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.build || number || 2 || YES || - || Primary key for builds table || -
|-
| [Object Root].list[].fields.cpu_usage || number || - || YES || - || Utilisation (%) as collected by buildstats || -
|-
| [Object Root].list[].fields.disk_io || number || - || YES || - || Number of ms spent doing I/Os || -
|-
| [Object Root].list[].fields.elapsed_time || string || - || YES || - || How long it took for the task to complete. || -
|-
| [Object Root].list[].fields.line_number || number || - || YES || - || The starting line in the file that applies to the task || -
|-
| [Object Root].list[].fields.order || number || - || YES || - || Shows the order of execution || -
|-
| [Object Root].list[].fields.outcome || number || - || YES || - || Identifies the result of a task. || OUTCOME_SUCCESS = 0 OUTCOME_COVERED = 1 OUTCOME_SSTATE = 2 OUTCOME_EXISTING = 3 OUTCOME_FAILED = 4 OUTCOME_NA = 5 
|-
| [Object Root].list[].fields.path_to_sstate_obj || File Path || - || YES || - || Shows the path BitBake searched for files for a certain task || -
|-
| [Object Root].list[].fields.py_stack_trace || string || - || YES || - || The Python stack trace || -
|-
| [Object Root].list[].fields.script_type || number || - || YES || - || This item specifies if it is a Python task or a shell task || CODING_PYTHON = 0 CODING_SHELL = 1
|-
| [Object Root].list[].fields.source_url || File Path || - || YES || - || The path to the sstate object || -
|-
| [Object Root].list[].fields.sstate_checksum || string || - || YES || - || Signature of a task's inputs to support incremental builds || -
|-
| [Object Root].list[].fields.sstate_result || number || - || YES || - || We'll use this to generate information for the task history || SSTATE_NA = 0 SSTATE_MISS = 1 SSTATE_FAILED = 2 SSTATE_RESTORED = 3 
|-
| [Object Root].list[].fields.task_executed || boolean || - || YES || - || Identifies if a task was executed or not || true means Executed, false means Prebuilt
|-
| [Object Root].list[].fields.task_name || string || - || YES || - || The task name || -
|-
| [Object Root].list[].fields.message || string || - || YES || - || Message returned by task || -
|-
| [Object Root].list[].fields.logfile || File Path || - || YES || - || Log for the task || -
|-
| [Object Root].list[].fields.work_directory || File Path || - || YES || - || The WORKDIR for the recipe to which the task applies || -
|-
| [Object Root].list[].fields.recipe || number || - || YES || - || Primary key for the Recipes model of which the task belongs to || -
|}

Webhob REST API Tasks

2013-08-02T12:54:55Z

Calin Dragomir:

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /tasks || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || -
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || "<FIELD>:<ORDER_DIRECTION>"
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].model || string || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.build || number || 2 || YES || - || - || -
|-
| [Object Root].list[].fields.cpu_usage || string || - || YES || - || - || -
|-
| [Object Root].list[].fields.disk_io || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.elapsed_time || number representation of date || - || YES || - || - || -
|-
| [Object Root].list[].fields.line_number || number || - || YES || - || - || -
|-
| [Object Root].list[].fields.order || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.outcome || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.path_to_sstate_obj || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.py_stack_trace || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.script_type || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.source_url || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.sstate_checksum || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.sstate_result || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.task_executed || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.task_name || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.message || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.logfile || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.work_directory || string array || - || YES || - || - || -
|}

Webhob REST API Tasks

2013-08-02T12:01:11Z

Calin Dragomir: /* Response */

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /tasks || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || -
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || "<FIELD>:<ORDER_DIRECTION>"
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].model || string || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.build || number || 2 || YES || - || - || -
|-
| [Object Root].list[].fields.cpu_usage || string || - || YES || - || - || -
|-
| [Object Root].list[].fields.disk_io || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.elapsed_time || number representation of date || - || YES || - || - || -
|-
| [Object Root].list[].fields.line_number || number || - || YES || - || - || -
|-
| [Object Root].list[].fields.log_file || string || - || YES || - || Path to log file || -
|-
| [Object Root].list[].fields.order || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.outcome || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.path_to_sstate_obj || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.py_stack_trace || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.script_type || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.source_url || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.sstate_checksum || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.sstate_result || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.task_executed || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.task_name || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.message || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.logfile || string array || - || YES || - || - || -
|-
| [Object Root].list[].fields.work_directory || string array || - || YES || - || - || -
|}

Webhob REST API Builds

2013-07-26T12:16:21Z

Calin Dragomir:

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /builds || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || The only argument is the string searched in the database.
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || "<FIELD>:<ORDER_DIRECTION>"
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].model || string || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.outcome || number || 2 || YES || - || Signals successful or failed build || 0 - the build has failed, 1 - successful build
|-
| [Object Root].list[].fields.target || string || - || YES || - || The name of the build target(s) || -
|-
| [Object Root].list[].fields.machine || string array || - || YES || - || The selected hardware || -
|-
| [Object Root].list[].fields.started_on || number representation of date || - || YES || - || Marks the moment the process is started || -
|-
| [Object Root].list[].fields.completed_on || number representation of date || - || YES || - || Marks the moment the process is completed || -
|-
| [Object Root].list[].fields.errors_no || number || - || YES || - || Number of errors thrown by the build || -
|-
| [Object Root].list[].fields.warnings_no || number || - || YES || - || Number of warnings thrown by the build || -
|-
| [Object Root].list[].fields.cooker_log_path || string || - || YES || - || Path to log file || -
|-
| [Object Root].list[].fields.image_fstypes || string array || - || YES || - || The extensions of the root file systems produced by the build || -
|}

Webhob REST API Builds

2013-07-23T11:04:47Z

Calin Dragomir:

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /builds || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || -
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || -
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].model || string || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.outcome || number || 2 || YES || - || Signals successful or failed build || 0 - the build has failed, 1 - successful build
|-
| [Object Root].list[].fields.target || string || - || YES || - || The name of the build target(s) || -
|-
| [Object Root].list[].fields.machine || string array || - || YES || - || The selected hardware || -
|-
| [Object Root].list[].fields.started_on || number representation of date || - || YES || - || Marks the moment the process is started || -
|-
| [Object Root].list[].fields.completed_on || number representation of date || - || YES || - || Marks the moment the process is completed || -
|-
| [Object Root].list[].fields.errors_no || number || - || YES || - || Number of errors thrown by the build || -
|-
| [Object Root].list[].fields.warnings_no || number || - || YES || - || Number of warnings thrown by the build || -
|-
| [Object Root].list[].fields.cooker_log_path || string || - || YES || - || Path to log file || -
|-
| [Object Root].list[].fields.image_fstypes || string array || - || YES || - || The extensions of the root file systems produced by the build || -
|}

Webhob REST API Builds

2013-07-23T10:41:05Z

Calin Dragomir:

[[Category:REST API Contracts]]

=== Endpoints ===

{| class="wikitable"
|-
! Method !! Endpoint !! Body !! Description !! Comments
|-
| GET || /builds || JSON || Returns page size limited and search criteria filtered amount of builds from the data base || -
|}

=== Parameters ===

{| class="wikitable"
|-
! Name !! Type !! Description !! Comments
|-
| limit || number || The amout of builds to be displayed in one page || -
|-
| offset || number || Represents the position of the first record in the page || If limit is 1, end point will target the findOne functionality and the offset will act as the actual build id
|-
| search || string || Search string for all fields || -
|-
| filter || string || Consists of a string used to filter results || The general form: "<FIELD>:<VALUE>".
|-
| orderby || string || the name of the field that determens the soring of the resuls. || -
|}

=== Response ===

{| class="wikitable"
|-
! Name !! Type !! Dimension !! Required !! Default value(s) !! Description !! Comments
|-
| [Object Root] || object || - || YES || - || - || -
|-
| [Object Root].count || number || - || YES || - || Total amount of builds to be displayed (according to filter parameter) || -
|-
| [Object Root].list || object array || - || YES || - || - || To describe a generic object element of the array, the refference to it's root is "list[]"
|-
| [Object Root].list[].pk || number || - || YES || - || - || -
|-
| [Object Root].list[].string || number || - || YES || - || - || -
|-
| [Object Root].list[].fields || object || - || YES || - || - || -
|-
| [Object Root].list[].fields.outcome || number || 2 || YES || - || Signals successful or failed build || 0 - the build has failed, 1 - successful build
|-
| [Object Root].list[].fields.target || string || - || YES || - || The name of the build target(s) || -
|-
| [Object Root].list[].fields.machine || string array || - || YES || - || The selected hardware || -
|-
| [Object Root].list[].fields.started_on || number representation of date || - || YES || - || Marks the moment the process is started || -
|-
| [Object Root].list[].fields.completed_on || number representation of date || - || YES || - || Marks the moment the process is completed || -
|-
| [Object Root].list[].fields.errors_no || number || - || YES || - || Number of errors thrown by the build || -
|-
| [Object Root].list[].fields.warnings_no || number || - || YES || - || Number of warnings thrown by the build || -
|-
| [Object Root].list[].fields.cooker_log_path || string || - || YES || - || Path to log file || -
|-
| [Object Root].list[].fields.image_fstypes || string array || - || YES || - || The extensions of the root file systems produced by the build || -
|}

Event information model for Toaster

2013-06-21T12:31:25Z

Calin Dragomir:

[[Category:WebHob]]

This is the information we need for completing the tables of the WebHOB database.

==== BUILD ====

:uuid : DONE : unique task id
:target : N/A : # OneToMany to the targets table
:machine : DONE : # index into all the machines that we support ? # we can ask bitbake for the MACHINE variable
:distro : DONE : # variable - DISTRO
:distro_version : DONE : # variable - DISTRO_VERSION
:started_on : DONE # use BuildStarted event
:completed_on DONE # use BuildCompleted event
:outcome : DONE # We can grab this at the end of all commands by checking the errors number or the return value.
:number_of_errors DONE # total number of log events with level of ERROR (see knotty)
:number_of_warnings DONE # total number of log events with level of WARNING (see knotty)
:cpu_usage : N/A # it's in the build stats; we don't have a meaning
:disk_io : N/A # it's in the build stats; we don't have a meaning
:cooker_log_path : DONE # main cooker log - composed of all log events over the set log level (see knotty)
:build_name : DONE # variable - BUILDNAME
:bitbake_version : DONE # variable - BB_VERSION

! This has been removed for the moment -> output : N/A # list of generated files; downloads - please check how Hob does this

==== TASK ====
:uuid : DONE -> generated at the beginning of each build.
:order : DONE -> counted on each TaskStarted event
:task_executed : DONE -> set to True for each TaskStarted event
:outcome : DONE? -> retrieved from event.getDisplayName() # is this OK ?
:sstate_checksum : N/A
:path_to_sstate_obj : N/A
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
*index on recipe / task_name
:source_url : N/A
:log_file: N/A
:work_directory : N/A
:script_type : N/A
:file_path : N/A
:line_number : N/A
:py_stack_trace : N/A
:disk_io : N/A
:cpu_usage : N/A
:elapsed_time : DONE -> difference between started and finished time
:errors_no : N/A
:warnings_no : N/A
:error : N/A
:warning : N/A
:sstate_result : N/A
:diffsigs : N/A

==== TASK DEPENDENCY ====
:task : N/A
:depends_on : N/A # still a task_id

==== TARGET ====
:build : N/A # of build that generated the image
:is_image : N/A
! TBC

==== ARTIFACT ====
:build : N/A
:target_id : N/A
:file_name : N/A
:file_size : N/A

==== PACKAGE IN IMAGE ====
:package : N/A
:target : N/A

==== PACKAGE ====
:recipe : N/A # recipe that generated this package
:name : N/A
:version : N/A
:size : N/A

==== PACKAGE DEPENDENCY ====
:package : N/A
:depends_on : N/A

==== FILELIST ====
:package : N/A # id in the list of packages
:complete_file_path : N/A
:file_size : N/A

==== RECIPE ====
:name : N/A
:version : N/A
:layer : N/A # point to a layer index
:summary : N/A
:description : N/A
:section : N/A ?
:license : N/A
:licensing info: N/A ?
:homepage : N/A ?
:bugtracker : N/A ?
:author : N/A ?
:file_path : N/A ?

==== RECIPE DEPENDENCY ====
:recipe : N/A
:depends_on : N/A # also a recipe id

==== LAYER ====
:name : N/A
:local_path : N/A
:layer_index_url: N/A
! TBC

==== BUILD_LAYER ====
:build : N/A
:layer : N/A
:branch : N/A
:commit : N/A
:priority : N/A

==== VARIABLE ====
:build : N/A
:variable_name : N/A # a list of `bitbake -e`
:variable_value : N/A
:file : N/A # a single file name where the variable is set
:changed : N/A
:human_readable_name : N/A
:description : N/A

==== MACHINE ====
:name : N/A
:description : N/A

Event information model for Toaster

2013-06-20T10:46:39Z

Calin Dragomir:

[[Category:WebHob]]

This is the information we need for completing the tables of the WebHOB database.

==== BUILD ====

:uuid : DONE : unique task id
:target : N/A : # OneToMany to the targets table
:machine : DONE : # index into all the machines that we support ? # we can ask bitbake for the MACHINE variable
:distro : DONE : # variable - DISTRO
:distro_version : DONE : # variable - DISTRO_VERSION
:started_on : DONE # use BuildStarted event
:completed_on DONE # use BuildCompleted event
:outcome : DONE # We can grab this at the end of all commands by checking the errors number or the return value.
:number_of_errors DONE # total number of log events with level of ERROR (see knotty)
:number_of_warnings DONE # total number of log events with level of WARNING (see knotty)
:cpu_usage : N/A # it's in the build stats; we don't have a meaning
:disk_io : N/A # it's in the build stats; we don't have a meaning
:cooker_log_path : DONE # main cooker log - composed of all log events over the set log level (see knotty)
:build_name : DONE # variable - BUILDNAME
:bitbake_version : DONE # variable - BB_VERSION

! This has been removed for the moment -> output : N/A # list of generated files; downloads - please check how Hob does this

==== TASK ====
:uuid : DONE -> generated at the beginning of each build.
:task_id :N/A
:order : DONE -> counted on each TaskStarted event
:task_executed : DONE -> set to True for each TaskStarted event
:outcome : DONE? -> retrieved from event.getDisplayName() # is this OK ?
:sstate_checksum : N/A
:path_to_sstate_obj : N/A
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
*index on recipe / task_name
:source_url : N/A
:log_file: N/A
:work_directory : N/A
:script_type : N/A
:file_path : N/A
:line_number : N/A
:py_stack_trace : N/A
:disk_io : N/A
:cpu_usage : N/A
:elapsed_time : DONE -> difference between started and finished time
:errors_no : N/A
:warnings_no : N/A
:error : N/A
:warning : N/A
:sstate_result : N/A
:diffsigs : N/A

==== TASK DEPENDENCY ====
:task : N/A
:depends_on : N/A # still a task_id

==== TARGET ====
:build : N/A # of build that generated the image
:is_image : N/A
! TBC

==== ARTIFACT ====
:build : N/A
:target_id : N/A
:file_name : N/A
:file_size : N/A

==== PACKAGE IN IMAGE ====
:package : N/A
:target : N/A

==== PACKAGE ====
:recipe : N/A # recipe that generated this package
:name : N/A
:version : N/A
:size : N/A

==== PACKAGE DEPENDENCY ====
:package : N/A
:depends_on : N/A

==== FILELIST ====
:package : N/A # id in the list of packages
:complete_file_path : N/A
:file_size : N/A

==== RECIPE ====
:name : N/A
:version : N/A
:layer : N/A # point to a layer index
:summary : N/A
:description : N/A
:section : N/A ?
:license : N/A
:licensing info: N/A ?
:homepage : N/A ?
:bugtracker : N/A ?
:author : N/A ?
:file_path : N/A ?

==== RECIPE DEPENDENCY ====
:recipe : N/A
:depends_on : N/A # also a recipe id

==== LAYER ====
:name : N/A
:local_path : N/A
:layer_index_url: N/A
! TBC

==== BUILD_LAYER ====
:build : N/A
:layer : N/A
:branch : N/A
:commit : N/A
:priority : N/A

==== VARIABLE ====
:build : N/A
:variable_name : N/A # a list of `bitbake -e`
:variable_value : N/A
:file : N/A # a single file name where the variable is set
:changed : N/A
:human_readable_name : N/A
:description : N/A

==== MACHINE ====
:name : N/A
:description : N/A

Event information model for Toaster

2013-06-20T10:46:09Z

Calin Dragomir:

[[Category:WebHob]]

This is the information we need for completing the tables of the WebHOB database.

==== BUILD ====

:uuid : DONE : unique task id
:target : N/A : # OneToMany to the targets table
:machine : DONE : # index into all the machines that we support ? # we can ask bitbake for the MACHINE variable
:distro : DONE : # variable - DISTRO
:distro_version : DONE : # variable - DISTRO_VERSION
:started_on : DONE # use BuildStarted event
:completed_on DONE # use BuildCompleted event
:outcome : DONE # We can grab this at the end of all commands by checking the errors number or the return value.
:number_of_errors DONE # total number of log events with level of ERROR (see knotty)
:number_of_warnings DONE # total number of log events with level of WARNING (see knotty)
:cpu_usage : N/A # it's in the build stats; we don't have a meaning
:disk_io : N/A # it's in the build stats; we don't have a meaning
:cooker_log_path : DONE # main cooker log - composed of all log events over the set log level (see knotty)
:build_name : DONE # variable - BUILDNAME
:bitbake_version : DONE # variable - BB_VERSION

: !This has been removed for the moment -> output : N/A # list of generated files; downloads - please check how Hob does this

==== TASK ====
:uuid : DONE -> generated at the beginning of each build.
:task_id :N/A
:order : DONE -> counted on each TaskStarted event
:task_executed : DONE -> set to True for each TaskStarted event
:outcome : DONE? -> retrieved from event.getDisplayName() # is this OK ?
:sstate_checksum : N/A
:path_to_sstate_obj : N/A
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
*index on recipe / task_name
:source_url : N/A
:log_file: N/A
:work_directory : N/A
:script_type : N/A
:file_path : N/A
:line_number : N/A
:py_stack_trace : N/A
:disk_io : N/A
:cpu_usage : N/A
:elapsed_time : DONE -> difference between started and finished time
:errors_no : N/A
:warnings_no : N/A
:error : N/A
:warning : N/A
:sstate_result : N/A
:diffsigs : N/A

==== TASK DEPENDENCY ====
:task : N/A
:depends_on : N/A # still a task_id

==== TARGET ====
:build : N/A # of build that generated the image
:is_image : N/A
! TBC

==== ARTIFACT ====
:build : N/A
:target_id : N/A
:file_name : N/A
:file_size : N/A

==== PACKAGE IN IMAGE ====
:package : N/A
:target : N/A

==== PACKAGE ====
:recipe : N/A # recipe that generated this package
:name : N/A
:version : N/A
:size : N/A

==== PACKAGE DEPENDENCY ====
:package : N/A
:depends_on : N/A

==== FILELIST ====
:package : N/A # id in the list of packages
:complete_file_path : N/A
:file_size : N/A

==== RECIPE ====
:name : N/A
:version : N/A
:layer : N/A # point to a layer index
:summary : N/A
:description : N/A
:section : N/A ?
:license : N/A
:licensing info: N/A ?
:homepage : N/A ?
:bugtracker : N/A ?
:author : N/A ?
:file_path : N/A ?

==== RECIPE DEPENDENCY ====
:recipe : N/A
:depends_on : N/A # also a recipe id

==== LAYER ====
:name : N/A
:local_path : N/A
:layer_index_url: N/A
!!TBC

==== BUILD_LAYER ====
:build : N/A
:layer : N/A
:branch : N/A
:commit : N/A
:priority : N/A

==== VARIABLE ====
:build : N/A
:variable_name : N/A # a list of `bitbake -e`
:variable_value : N/A
:file : N/A # a single file name where the variable is set
:changed : N/A
:human_readable_name : N/A
:description : N/A

==== MACHINE ====
:name : N/A
:description : N/A

Event information model for Toaster

2013-06-13T16:22:40Z

Calin Dragomir:

[[Category:WebHob]]

This is the information we need for completing the tables of the WebHOB database.

==== BUILDS ====

:uuid : DONE : unique task id
:target : N/A : # OneToMany to the targets table
:machine : DONE : # index into all the machines that we support ? # we can ask bitbake for the machine variable
:distro : DONE : # we can ask bitbake for the distro variable
:distro_version : DONE : # we can ask bitbake for the distro version variable
:host_system : N/A
:host_distribution : N/A
:target_system : N/A
:layers : : DONE # index into the layer table ? # we can ask bitbake for the list (BBLAYERS)
:started_on : N/A # Is there a build started event we can use ?
:completed_on N/A # Is there a build success event or should we grab this when events processing has ended ?
:outcome : DONE # We can grab this at the end of all commands by checking the errors number or the return value.
:number_of_errors N/A
:number_of_warnings N/A
:cpu_usage : N/A # it's in the build stats; we don't have a meaning
:disk_io : N/A # it's in the build stats; we don't have a meaning
:log : N/A # main cooker log
:output : N/A # list of generated files; downloads
:build_name : N/A # bitbake generates a name
:bitbake_version : N/A

==== TASKS ====
:uuid : DONE -> generated at the beginning of each build.
:task_id :N/A
:order : DONE -> counted on each TaskStarted event
:task_executed : DONE -> set to True for each TaskStarted event
:outcome : DONE? -> retrieved from event.getDisplayName() # is this OK ?
:sstate_checksum : N/A
:path_to_sstate_obj : N/A
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
*index on recipe / task_name
:source_url : N/A
:log_file: N/A
:work_directory : N/A
:script_type : N/A
:file_path : N/A
:line_number : N/A
:py_stack_trace : N/A
:disk_io : N/A
:cpu_usage : N/A
:elapsed_time : DONE -> difference between started and finished time
:errors_no : N/A
:warnings_no : N/A
:error : N/A
:warning : N/A
:sstate_result : N/A
:diffsigs : N/A

==== TASK DEPENDENCY ====
:task_id : N/A
:depends_on : N/A # still a task_id

==== TARGETS ====
:uuid : N/A # of build that generated the image
:is_image : N/A
# TBC

==== ARTIFACTS ====
:uuid : N/A
:target_id : N/A
:file_name : N/A
:file_size : N/A

==== PACKAGES IN IMAGES ====
:package_id : N/A
:target_id : N/A

==== PACKAGES ====
:package_id : N/A
:recipe : N/A # recipe that generated this package
:name : N/A
:version : N/A
:size : N/A

==== PACKAGE DEPENDENCIES ====
:package_id : N/A
:depends_on : N/A

==== FILELIST ====
:package_id : N/A # id in the list of packages
:complete file path : N/A
:file size : N/A

==== RECIPES ====
:name : N/A
:version : N/A
:layer : N/A # point to a layer index
:summary : N/A
:description : N/A
:section : N/A ?
:license : N/A
:licensing info: N/A ?
:homepage : N/A ?
:bugtracker : N/A ?
:author : N/A ?
:file_path : N/A ?

==== RECIPE DEPENDENCY ====
:recipe_id : N/A
:depends_on : N/A # also a recipe id

==== LAYERS ====
:name : N/A
:branch : N/A
:commit : N/A
:priority : N/A
:link to oecore layer index: N/A ?
# TBC

==== VARIABLES ====
:uuid : N/A
:variable_name : N/A # a list of `bitbake -e`
:variable_value : N/A
:file : N/A # a single file name where the variable is set
:changed : N/A
:human_readable_name : N/A
:description : N/A

==== MACHINES ====
:name : N/A
:description : N/A

Event information model for Toaster

2013-06-10T07:39:46Z

Calin Dragomir:

This is the information we need for completing the tables of the WebHOB database.

==== TASKS ====

:uuid : DONE -> generated at the beginning of each build.
:task_id : NOT AVAILABLE
:order : DONE -> counted on each TaskStarted event
:task_executed : DONE -> set to True for each TaskStarted event
:outcome : DONE? -> retrieved from event.getDisplayName() #is this OK ?
:sstate_checksum : NOT AVAILABLE
:path_to_sstate_obj : NOT AVAILABLE
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
:source_url : NOT AVAILABLE
:log_file: NOT AVAILABLE
:work_directory : NOT AVAILABLE
:script_type : NOT AVAILABLE
:file_path : NOT AVAILABLE
:line_number : NOT AVAILABLE
:py_stack_trace : NOT AVAILABLE
:disk_io : NOT AVAILABLE
:cpu_usage : NOT AVAILABLE
:elapsed_time : DONE -> difference between started and finished time
:dependent_tasks : NOT AVAILABLE
:errors_no : NOT AVAILABLE
:warnings_no : NOT AVAILABLE
:error : NOT AVAILABLE
:warning : NOT AVAILABLE
:sstate_result : NOT AVAILABLE

Django models

2013-06-10T07:25:25Z

Calin Dragomir: moved Django models to Web Hob Django Models: This page relates to the Web Hob project.

#REDIRECT [[Web Hob Django Models]]

Event information

2013-06-10T07:23:44Z

Calin Dragomir: moved Event information to Web Hob Event information: This page relates to the Web Hob project.

#REDIRECT [[Web Hob Event information]]

Event information model for Toaster

2013-06-10T07:23:44Z

Calin Dragomir: moved Event information to Web Hob Event information: This page relates to the Web Hob project.

This is the information we need for completing the tables of the WebHOB database.

==== TASKS ====

:uuid : DONE -> generated at the beginning of each build.
:task_id : NOT AVAILABLE
:order : DONE -> counted on each TaskStarted event
:task_executed : NOT AVAILABLE
:outcome : DONE? -> retrieved from event.getDisplayName() #is this OK ?
:sstate_checksum : NOT AVAILABLE
:path_to_sstate_obj : NOT AVAILABLE
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
:source_url : NOT AVAILABLE
:log_file: NOT AVAILABLE
:work_directory : NOT AVAILABLE
:script_type : NOT AVAILABLE
:file_path : NOT AVAILABLE
:line_number : NOT AVAILABLE
:py_stack_trace : NOT AVAILABLE
:disk_io : NOT AVAILABLE
:cpu_usage : NOT AVAILABLE
:elapsed_time : DONE -> difference between started and finished time
:dependent_tasks : NOT AVAILABLE
:errors_no : NOT AVAILABLE
:warnings_no : NOT AVAILABLE
:error : NOT AVAILABLE
:warning : NOT AVAILABLE
:sstate_result : NOT AVAILABLE

Event information model for Toaster

2013-06-07T15:03:47Z

Calin Dragomir:

This is the information we need for completing the tables of the WebHOB database.

==== TASKS ====

:uuid : DONE -> generated at the beginning of each build.
:task_id : NOT AVAILABLE
:order : DONE -> counted on each TaskStarted event
:task_executed : NOT AVAILABLE
:outcome : DONE? -> retrieved from event.getDisplayName() #is this OK ?
:sstate_checksum : NOT AVAILABLE
:path_to_sstate_obj : NOT AVAILABLE
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
:source_url : NOT AVAILABLE
:log_file: NOT AVAILABLE
:work_directory : NOT AVAILABLE
:script_type : NOT AVAILABLE
:file_path : NOT AVAILABLE
:line_number : NOT AVAILABLE
:py_stack_trace : NOT AVAILABLE
:disk_io : NOT AVAILABLE
:cpu_usage : NOT AVAILABLE
:elapsed_time : DONE -> difference between started and finished time
:dependent_tasks : NOT AVAILABLE
:errors_no : NOT AVAILABLE
:warnings_no : NOT AVAILABLE
:error : NOT AVAILABLE
:warning : NOT AVAILABLE
:sstate_result : NOT AVAILABLE

Event information model for Toaster

2013-06-07T14:59:54Z

Calin Dragomir:

This is the information we need for completing the tables of the WebHOB database.

==== TASKS ====

:uuid : DONE -> generated at the beginning of each build.
:task_id : NOT AVAILABLE
:order : DONE -> counted on each TaskStarted event
:task_executed : NOT AVAILABLE
:outcome : DONE? -> retrieved from event.getDisplayName() #is this OK ?
:sstate_checksum : NOT AVAILABLE
:path_to_sstate_obj : NOT AVAILABLE
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
:source_url : NOT AVAILABLE
:log_file: NOT AVAILABLE
:work_directory : NOT AVAILABLE
:script_type : NOT AVAILABLE
:file_path : NOT AVAILABLE
:line_number : NOT AVAILABLE
:py_stack_trace : NOT AVAILABLE
:disk_io : NOT AVAILABLE
:cpu_usage : NOT AVAILABLE
:elapsed_time : DONE -> difference between started and finished time
:dependent_tasks : NOT AVAILABLE
:errors_no : NOT AVAILABLE
:warnings_no : NOT AVAILABLE
:error : NOT AVAILABLE
:warning : NOT AVAILABLE
:sstate_result : NOT AVAILABLE

Event information model for Toaster

2013-06-07T14:45:44Z

Calin Dragomir: Created page with "This is the information we need for completing the tables of the WebHOB database. ==== TASKS ==== :uuid : DONE -> generated at the beginning of each build. :task_id : NOT AVAI..."

This is the information we need for completing the tables of the WebHOB database.

==== TASKS ====

:uuid : DONE -> generated at the beginning of each build.
:task_id : NOT AVAILABLE
:order : DONE -> counted on each TaskStarted event
:task_executed : NOT AVAILABLE
:outcome : NOT AVAILABLE
:sstate_checksum : NOT AVAILABLE
:path_to_sstate_obj : NOT AVAILABLE
:recipe : DONE -> retrieved from event._package
:task_name : DONE -> retrieved from event._task
:source_url : NOT AVAILABLE
:log_file: NOT AVAILABLE
:work_directory : NOT AVAILABLE
:script_type : NOT AVAILABLE
:file_path : NOT AVAILABLE
:line_number : NOT AVAILABLE
:py_stack_trace : NOT AVAILABLE
:disk_io : NOT AVAILABLE
:cpu_usage : NOT AVAILABLE
:elapsed_time : DONE -> difference between started and finished time
:dependent_tasks : NOT AVAILABLE
:errors_no : NOT AVAILABLE
:warnings_no : NOT AVAILABLE
:error : NOT AVAILABLE
:warning : NOT AVAILABLE
:sstate_result : NOT AVAILABLE

Event Logger Data Interfaces

2013-06-07T14:33:24Z

Calin Dragomir: /* interface with the Data Store system */

[[Category:WebHob]]

Documenting the event logger data interfaces and action models.
* The EVT is the Event Translation Layer, which receive a set of events from a Bitbake build run, and builds a in-memory record set describing the build.
* The DSI is the Data Store Interface that will poll new records from each of the EVTs assigned to it, and write them in permanent record.
* The AAM is the Artifacts Analysis Module. This module is run on build complete, scans the build/ directory for artifacts, and generates records sets about the build artifacts that will be stored in the data store.

== EVT ==

Bitbake uses a push-event model where it sends pickled Python objects over XMLRPC to an event sink. EVT receives Python objects describing the progress of a build run, and updates its internal state to match the state of the Bitbake server. It will generate a set of records describing the statistics and results of the build run, e.g. task records for all the tasks that did run, build records for all targets, etc.

=== interface with bitbake ===

The bitbake server runs at IP:PORT location. A event server controller lives at IP:(PORT+2) accepting a XMLRPC interface with two commands: registerEventHandler and unregisterEventHandler.

EVT will run resident in memory, connecting to the bitbake server and accepting events. It will have an internal state that will follow the internal state of the bitbake server. This allows the EVT to infer information about the server state, and export this information to the DSI.

The event list that it receives from Bitbake is documented here, together with standard actions being taken in Knotty and what should happen in EVT.

{| class="wikitable"
|-
! Event object type !! action taken in Knotty !! action in EVT ||
|-
|bb.runqueue.runQueueExitWait ||
* Assigns 1 for main.shutdown if not already set
||
|-
|bb.event.LogExecTTY ||
*If the stdin and stdout are terminals -> while there are still tries:
::if the event finishes successfully, the loop exists, and bb continues with other events.
::else it waits for sleep_delay amount of time and retries.
*A warning message is written in the logger file.
||
|-
|logging.LogRecord ||
*If there is an error or warning, count them
*Log every event
||
|-
|bb.build.TaskFailed ||
* If a logfile is found: display error by reading the logfile
||
|-
|bb.build.TaskBase ||
* Log the event message in the logger file
||
|-
|bb.event.ParseStarted ||
* Start a new parseprogress if event.total is not zero
|-
|bb.event.ParseProgress ||
* Update the parse progress based on the current event
||
|-
|bb.event.ParseCompleted ||
* If there is a parseprogress object, finish it and print to the console the results of the parse
||
|-
|bb.event.CacheLoadStarted ||
* Create a cacheprogress object
||
|-
|bb.event.CacheLoadProgress ||
* Update the cacheprogress based on the current event
||
|-
|bb.event.CacheLoadCompleted ||
* Finish the current cacheprogress and print the number of events grabbed from dependency cache
||
|-
|bb.command.CommandFailed ||
* Mark this as an error and exit the main loop by putting shutdown into state 2
||
|-
|bb.command.CommandExit ||
* If a returned value is not set, grab the event exit code and set it as a return value
||
|-
|bb.command.CommandCompleted ||
* Put shutdown into state 2
||
|-
|bb.cooker.CookerExit ||
* Put shutdown into state 2
||
|-
|bb.event.MultipleProviders ||
* Log info about multiple providers available and instruct the user to define a PREFERRED_PROVIDER
||
|-
|bb.event.NoProvider ||
* Mark this as an error
* Check dependees and log the error
* List the error reasons in the log file
||
|-
|bb.runqueue.sceneQueueTaskStarted ||
* Marks the start of a scene queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskStarted ||
* Marks the start of a queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskFailed ||
* Grab failed task, append it to task failures and log the error together with the exit code
||
|-
|bb.runqueue.sceneQueueTaskFailed ||
* Log a warning about the setscene task that failed together with the exit code and report that the real task will be ran instead
||
|-
| bb.event.BuildBase || ignored || ignored
|-
| bb.event.StampUpdate || ignored || ignored
|-
| bb.event.ConfigParsed || ignored || timestamp entry
|-
| bb.event.RecipeParsed || ignored || timestamp entry
|-
| bb.event.RecipePreFinalise || ignored || ignored
|-
| bb.runqueue.runQueueEvent || ignored
|-
| bb.runqueue.runQueueExitWait || ignored
|-
| bb.event.OperationStarted || ignored || timestamp
|-
| bb.event.OperationCompleted || ignored || timestamp, operation complete, timing
|-
| bb.event.OperationProgress || ignored || ignored
|-
| bb.event.DiskFull || ignored || error
|}

=== interface with DSI ===

== DSI ==

=== interface with EVT ===

=== interface with the Data Store system ===

The data interface is based on the Django ORM.
A definition of [[Django models]] for build and task information is available.

A list with the [[Event information]] we currently have and the things we need to have is also available.

== Artifacts analysis module ==

Once a Bitbake build is completed, it's artifacts are available for usage, inspection and download. The artifacts analysis module (AAM) will be launched by EVT at the end of a build, parse the generated artifacts and insert relevant data into the DSI.

Event Logger Data Interfaces

2013-06-07T12:16:09Z

Calin Dragomir:

[[Category:WebHob]]

Documenting the event logger data interfaces and action models.
* The EVT is the Event Translation Layer, which receive a set of events from a Bitbake build run, and builds a in-memory record set describing the build.
* The DSI is the Data Store Interface that will poll new records from each of the EVTs assigned to it, and write them in permanent record.
* The AAM is the Artifacts Analysis Module. This module is run on build complete, scans the build/ directory for artifacts, and generates records sets about the build artifacts that will be stored in the data store.

== EVT ==

Bitbake uses a push-event model where it sends pickled Python objects over XMLRPC to an event sink. EVT receives Python objects describing the progress of a build run, and updates its internal state to match the state of the Bitbake server. It will generate a set of records describing the statistics and results of the build run, e.g. task records for all the tasks that did run, build records for all targets, etc.

=== interface with bitbake ===

The bitbake server runs at IP:PORT location. A event server controller lives at IP:(PORT+2) accepting a XMLRPC interface with two commands: registerEventHandler and unregisterEventHandler.

EVT will run resident in memory, connecting to the bitbake server and accepting events. It will have an internal state that will follow the internal state of the bitbake server. This allows the EVT to infer information about the server state, and export this information to the DSI.

The event list that it receives from Bitbake is documented here, together with standard actions being taken in Knotty and what should happen in EVT.

{| class="wikitable"
|-
! Event object type !! action taken in Knotty !! action in EVT ||
|-
|bb.runqueue.runQueueExitWait ||
* Assigns 1 for main.shutdown if not already set
||
|-
|bb.event.LogExecTTY ||
*If the stdin and stdout are terminals -> while there are still tries:
::if the event finishes successfully, the loop exists, and bb continues with other events.
::else it waits for sleep_delay amount of time and retries.
*A warning message is written in the logger file.
||
|-
|logging.LogRecord ||
*If there is an error or warning, count them
*Log every event
||
|-
|bb.build.TaskFailed ||
* If a logfile is found: display error by reading the logfile
||
|-
|bb.build.TaskBase ||
* Log the event message in the logger file
||
|-
|bb.event.ParseStarted ||
* Start a new parseprogress if event.total is not zero
|-
|bb.event.ParseProgress ||
* Update the parse progress based on the current event
||
|-
|bb.event.ParseCompleted ||
* If there is a parseprogress object, finish it and print to the console the results of the parse
||
|-
|bb.event.CacheLoadStarted ||
* Create a cacheprogress object
||
|-
|bb.event.CacheLoadProgress ||
* Update the cacheprogress based on the current event
||
|-
|bb.event.CacheLoadCompleted ||
* Finish the current cacheprogress and print the number of events grabbed from dependency cache
||
|-
|bb.command.CommandFailed ||
* Mark this as an error and exit the main loop by putting shutdown into state 2
||
|-
|bb.command.CommandExit ||
* If a returned value is not set, grab the event exit code and set it as a return value
||
|-
|bb.command.CommandCompleted ||
* Put shutdown into state 2
||
|-
|bb.cooker.CookerExit ||
* Put shutdown into state 2
||
|-
|bb.event.MultipleProviders ||
* Log info about multiple providers available and instruct the user to define a PREFERRED_PROVIDER
||
|-
|bb.event.NoProvider ||
* Mark this as an error
* Check dependees and log the error
* List the error reasons in the log file
||
|-
|bb.runqueue.sceneQueueTaskStarted ||
* Marks the start of a scene queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskStarted ||
* Marks the start of a queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskFailed ||
* Grab failed task, append it to task failures and log the error together with the exit code
||
|-
|bb.runqueue.sceneQueueTaskFailed ||
* Log a warning about the setscene task that failed together with the exit code and report that the real task will be ran instead
||
|-
| bb.event.BuildBase || ignored || ignored
|-
| bb.event.StampUpdate || ignored || ignored
|-
| bb.event.ConfigParsed || ignored || timestamp entry
|-
| bb.event.RecipeParsed || ignored || timestamp entry
|-
| bb.event.RecipePreFinalise || ignored || ignored
|-
| bb.runqueue.runQueueEvent || ignored
|-
| bb.runqueue.runQueueExitWait || ignored
|-
| bb.event.OperationStarted || ignored || timestamp
|-
| bb.event.OperationCompleted || ignored || timestamp, operation complete, timing
|-
| bb.event.OperationProgress || ignored || ignored
|-
| bb.event.DiskFull || ignored || error
|}

=== interface with DSI ===

== DSI ==

=== interface with EVT ===

=== interface with the Data Store system ===

The data interface is based on the Django ORM.
A definition of [[Django models]] for build and task information is available.

== Artifacts analysis module ==

Once a Bitbake build is completed, it's artifacts are available for usage, inspection and download. The artifacts analysis module (AAM) will be launched by EVT at the end of a build, parse the generated artifacts and insert relevant data into the DSI.

Event Logger Data Interfaces

2013-06-05T08:21:22Z

Calin Dragomir:

[[Category:WebHob]]

Documenting the event logger data interfaces and action models.
* The EVT is the Event Translation Layer, which receive a set of events from a Bitbake build run, and builds a in-memory record set describing the build.
* The DSI is the Data Store Interface that will poll new records from each of the EVTs assigned to it, and write them in permanent record.
* The AAM is the Artifacts Analysis Module. This module is run on build complete, scans the build/ directory for artifacts, and generates records sets about the build artifacts that will be stored in the data store.

== EVT ==

Bitbake uses a push-event model where it sends pickled Python objects over XMLRPC to an event sink. EVT receives Python objects describing the progress of a build run, and updates its internal state to match the state of the Bitbake server. It will generate a set of records describing the statistics and results of the build run, e.g. task records for all the tasks that did run, build records for all targets, etc.

=== interface with bitbake ===

The bitbake server runs at IP:PORT location. A event server controller lives at IP:(PORT+2) accepting a XMLRPC interface with two commands: registerEventHandler and unregisterEventHandler.

EVT will run resident in memory, connecting to the bitbake server and accepting events. It will have an internal state that will follow the internal state of the bitbake server. This allows the EVT to infer information about the server state, and export this information to the DSI.

The event list that it receives from Bitbake is documented here, together with standard actions being taken in Knotty and what should happen in EVT.

{| class="wikitable"
|-
! Event object type !! action taken in Knotty !! action in EVT ||
|-
|bb.runqueue.runQueueExitWait ||
* Assigns 1 for main.shutdown if not already set
||
|-
|bb.event.LogExecTTY ||
*If the stdin and stdout are terminals -> while there are still tries:
::if the event finishes successfully, the loop exists, and bb continues with other events.
::else it waits for sleep_delay amount of time and retries.
*A warning message is written in the logger file.
||
|-
|logging.LogRecord ||
*If there is an error or warning, count them
*Log every event
||
|-
|bb.build.TaskFailed ||
* If a logfile is found: display error by reading the logfile
||
|-
|bb.build.TaskBase ||
* Log the event message in the logger file
||
|-
|bb.event.ParseStarted ||
* Start a new parseprogress if event.total is not zero
|-
|bb.event.ParseProgress ||
* Update the parse progress based on the current event
||
|-
|bb.event.ParseCompleted ||
* If there is a parseprogress object, finish it and print to the console the results of the parse
||
|-
|bb.event.CacheLoadStarted ||
* Create a cacheprogress object
||
|-
|bb.event.CacheLoadProgress ||
* Update the cacheprogress based on the current event
||
|-
|bb.event.CacheLoadCompleted ||
* Finish the current cacheprogress and print the number of events grabbed from dependency cache
||
|-
|bb.command.CommandFailed ||
* Mark this as an error and exit the main loop by putting shudown into state 2
||
|-
|bb.command.CommandExit ||
* If a returned value is not set, grab the event exit code and set it as a return value
||
|-
|bb.command.CommandCompleted ||
* Put shutdown into state 2
||
|-
|bb.cooker.CookerExit ||
* Put shutdown into state 2
||
|-
|bb.event.MultipleProviders ||
* Log info about multiple providers available and instruct the user to define a PREFERRED_PROVIDER
||
|-
|bb.event.NoProvider ||
* Mark this as an error
* Check dependees and log the error
* List the error reasons in the log file
||
|-
|bb.runqueue.sceneQueueTaskStarted ||
* Marks the start of a scene queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskStarted ||
* Marks the start of a queue task and logs info about it
||
|-
|bb.runqueue.runQueueTaskFailed ||
* Grab failed task, append it to task failures and log the error together with the exit code
||
|-
|bb.runqueue.sceneQueueTaskFailed ||
* Log a warning about the setscene task that failed together with the exit code and report that the real task will be ran instead
||
|-
| bb.event.BuildBase || ignored || ignored
|-
| bb.event.StampUpdate || ignored || ignored
|-
| bb.event.ConfigParsed || ignored || timestamp entry
|-
| bb.event.RecipeParsed || ignored || timestamp entry
|-
| bb.event.RecipePreFinalise || ignored || ignored
|-
| bb.runqueue.runQueueEvent || ignored
|-
| bb.runqueue.runQueueExitWait || ignored
|-
| bb.event.OperationStarted || ignored || timestamp
|-
| bb.event.OperationCompleted || ignored || timestamp, operation complete, timing
|-
| bb.event.OperationProgress || ignored || ignored
|-
| bb.event.DiskFull || ignored || error
|}

=== interface with DSI ===

== DSI ==

=== interface with EVT ===

=== interface with the Data Store system ===

The data interface is based on the Django ORM.
A definition of [[Django models]] for build and task information is available.

== Artifacts analysis module ==

Once a Bitbake build is completed, it's artifacts are available for usage, inspection and download. The artifacts analysis module (AAM) will be launched by EVT at the end of a build, parse the generated artifacts and insert relevant data into the DSI.

Toaster future release planning

2013-05-29T14:23:53Z

Calin Dragomir: /* Technology choice */

[[Category:WebHob]]

In order to select a data store technology, we investigated several possible backends for storing data generated by a build. These are the results of the investigation documenting the final choice.

== Criteria and investigation technique ==

We selected the following criteria, in descending importance order, as we target optimization in normal data usage. The pattern we expect is to insert for a run a set of records in the range on 10.000 entries, and then repeatedly read them in order to compute statistics and display information to users.

The criteria selected was:
# Time to read records (seconds/1000 records)
# Time to write records (seconds/1000 records)
# Space on disk (Mb/1000 records)
# Ease to install / deploy (subjective, 5 stars best, 1 star worst)

The investigation was done by generating around 8000 records from actual builds and then replicating these records to a high number in order to do simulate a high load on the data store.

Timings were performed on the client-side access of the data store, using timestamps to mark start and end of a batch run for read/writes. Disk space measurement was taken manually at the end of data generation.

The technologies measured were:
* CouchDB
* MySQL
* MongoDB
* sqlite

== Results from investigation ==

1. Time to write
(1000 records) - time is measured in seconds.
Ran 3 times.
* MySQL: 35.4514319897 / 30.2873010635 / 30.2174918652
* CouchDB: 83.7538969517 / 83.9673860073 / 86.0426850319
* SQLite: 194.561517954 / 176.219516039 / 172.443398952

2. Time to read
(1000 records) - time is in seconds
Ran 3 times => Command SELECT * FROM Events LIMIT 1000
* MySQL: 0.00492382049561 / 0.0043830871582 / 0.00462698936462
* SQLite: 0.000476837158203 / 0.000610828399658 / 0.000468015670776

3. Space on Disk:
* MySQL: 40.6 MB (3044814 records)
* CouchDB: 2.3 GB (13737 documents)
* SQLlite: 2,2 MB (24772 records)

4. Ease to install /deploy
* MySQL: 5 stars
* CouchDB: 3 stars
* SQLite: 4 stars

== Technology choice ==
sqlite was very slow on inserting new data, inserting 3 mil records would have taken too long, so a smaller ammount of data was chosen (~ 25k).
CouchDB is also slow on inserting data, and occupies a large space. It shows that it has a 2.3 GB size for only 13k of records, so this option was dropped from the list.

Analyzing the results obtained, it seems that sqlite is faster at reading than MySQL and this is very good for WebHOB, as reads are done more often than writes.

We're going to use MySQL for storing data in development and go for sqlite in Production.
If tests show that this is ineffective, we will make the necessary changes to achive the best results.

Toaster future release planning

2013-05-29T13:57:19Z

Calin Dragomir: /* Technology choice */

[[Category:WebHob]]

In order to select a data store technology, we investigated several possible backends for storing data generated by a build. These are the results of the investigation documenting the final choice.

== Criteria and investigation technique ==

We selected the following criteria, in descending importance order, as we target optimization in normal data usage. The pattern we expect is to insert for a run a set of records in the range on 10.000 entries, and then repeatedly read them in order to compute statistics and display information to users.

The criteria selected was:
# Time to read records (seconds/1000 records)
# Time to write records (seconds/1000 records)
# Space on disk (Mb/1000 records)
# Ease to install / deploy (subjective, 5 stars best, 1 star worst)

The investigation was done by generating around 8000 records from actual builds and then replicating these records to a high number in order to do simulate a high load on the data store.

Timings were performed on the client-side access of the data store, using timestamps to mark start and end of a batch run for read/writes. Disk space measurement was taken manually at the end of data generation.

The technologies measured were:
* CouchDB
* MySQL
* MongoDB
* sqlite

== Results from investigation ==

1. Time to write
(1000 records) - time is measured in seconds.
Ran 3 times.
* MySQL: 35.4514319897 / 30.2873010635 / 30.2174918652
* CouchDB: 83.7538969517 / 83.9673860073 / 86.0426850319
* SQLite: 194.561517954 / 176.219516039 / 172.443398952

2. Time to read
(1000 records) - time is in seconds
Ran 3 times => Command SELECT * FROM Events LIMIT 1000
* MySQL: 0.00492382049561 / 0.0043830871582 / 0.00462698936462
* SQLite: 0.000476837158203 / 0.000610828399658 / 0.000468015670776

3. Space on Disk:
* MySQL: 40.6 MB (3044814 records)
* CouchDB: 2.3 GB (13737 documents)
* SQLlite: 2,2 MB (24772 records)

4. Ease to install /deploy
* MySQL: 5 stars
* CouchDB: 3 stars
* SQLite: 4 stars

== Technology choice ==
sqlite was very slow on inserting new data, so I could not insert 3 mil records because it took too long.
CouchDB is also slow on inserting data, and occupies a large space. It shows that it has a 2.3 GB size for only 13k of records.

Analyzing the results obtained, it seems that sqlite is faster at reading than MySQL and this is very good for WebHOB, as reads done more often than writes.

We're going to use MySQL for storing data in development and go for sqlite in production.
If tests show that this is ineffective, we will make the necessary changes.

Toaster future release planning

2013-05-29T13:32:35Z

Calin Dragomir: /* Results from investigation */

Toaster future release planning

2013-05-29T13:30:19Z

Calin Dragomir: /* Results from investigation */

[[Category:WebHob]]

In order to select a data store technology, we investigated several possible backends for storing data generated by a build. These are the results of the investigation documenting the final choice.

== Criteria and investigation technique ==

We selected the following criteria, in descending importance order, as we target optimization in normal data usage. The pattern we expect is to insert for a run a set of records in the range on 10.000 entries, and then repeatedly read them in order to compute statistics and display information to users.

The criteria selected was:
# Time to read records (seconds/1000 records)
# Time to write records (seconds/1000 records)
# Space on disk (Mb/1000 records)
# Ease to install / deploy (subjective, 5 stars best, 1 star worst)

The investigation was done by generating around 8000 records from actual builds and then replicating these records to a high number in order to do simulate a high load on the data store.

Timings were performed on the client-side access of the data store, using timestamps to mark start and end of a batch run for read/writes. Disk space measurement was taken manually at the end of data generation.

The technologies measured were:
* CouchDB
* MySQL
* MongoDB
* sqlite

== Results from investigation ==

1. Time to write
(1000 records) - time is measured in seconds.
Ran 3 times.
MySQL: 35.4514319897 / 30.2873010635 / 30.2174918652
CouchDB: 83.7538969517 / 83.9673860073 / 86.0426850319
SQLite: 194.561517954 / 176.219516039 / 172.443398952

2. Time to read
(1000 records) - time is in seconds
Ran 3 times => Command SELECT * FROM Events LIMIT 1000
MySQL: 0.00492382049561 / 0.0043830871582 / 0.00462698936462
SQLite: 0.000476837158203 / 0.000610828399658 / 0.000468015670776

3. Space on Disk:
MySQL: 40.6 MB (3044814 records)
CouchDB: 2.3 GB (13737 documents)
SQLlite: 2,2 MB (24772 records)

4. Ease to install /deploy
MySQL: 5 stars
CouchDB: 3 stars
SQLite: 4 stars

== Technology choice ==