PR Service

PR Service

Overview

One of the challenges the build system and its users face is how to maintain a package feed which is compatible with existing package manager applications like rpm, apt and opkg. The main requirement is that versions increase in a linear fashion and there are a number of version components that facilitate this, namely in order of decreasing priority PE, PV and PR (epoch, version and revision).

The values these fields take is highly dependent on the policies and procedures of a given distribution and package feed. For a long time we've tried to maintain these values in the metadata but this is error prone, inaccurate and causes problems for people submitting recipes. The alternative is to autogenerate increasing numbers, particularly the revision field and this is what the PR service aims to do.

It should be noted that the build system itself knows when things need to rebuild by using "signatures" which represent any given build. All the inputs into a given task are represented by this signature and if it changes, the output can potentially change so it is rebuilt. The build system itself therefore doesn't rely on the PR numbers. We can use these signatures to generate PR values though. The service works with both OEBasic and OEBasicHash generators, with the understanding that PR bumps happen when the checksum changes and the different generator mechanisms change signatures under different circumstances.

As implemented, values from the PR service are included into the PR field as an addition of the form ".X" so r0 becomes r0.1, r0.2 and so on. This allows existing PR values to be used for whatever reasons allowing manual PR bumps should it be necessary.

Basic Operation

By default, there is no PR server and the packages generated are just "self consistent". The build system will add/remove any packages and there are no guarantees about upgrade paths.

In its simplest form with only a single build generating a package feed, the PR service can be enabled with the following in local.conf:

PRSERV_HOST = "localhost:0"

which will then also mean that packages have increasing PR values. Bitbake itself will take care of starting and stopping the server.

For more complex setups where multiple builds are working against a common package feed, you'd have one PR service running and connect each builder to it. This is done with a command like "bitbake-prserv --host <ip> --port <port> --start" and then the configuration would point at the server address and port number as needed.

Its recommended the server be used in conjunction with buildhistory which adds some sanity checks to package versions. This is enabled with:

# It is recommended to activate "buildhistory" for testing the PR service
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

Interaction with sstate

It should be noted that the PR information is not maintained as part of the sstate packages. If you maintain a sstate feed, its expected that either you run builders contributing to the sstate feed with a shared PR service, or you don't run one. Mixing the two will lead to problems.

Commands

bitbake-prserv

This is the main server application. It takes the following options:

--start
       start the server
--stop
       stop the server
--file
       the database file (default prserv.sqlite3)
--log
       the logfile (default prserv.log)
--loglevel
       the log level
--host
       ip address to bind
--port
       port to bind

bitbake-prserv-tool

This tool is used to export the PRs to a file and to import them back. It can be used in the forms:

bitbake-prserv-tool import <file>
bitbake-prserv-tool export <file>

In addition, bitbake-prserv-tool can be used to migrate git LOCALCOUNTs to AUTOINCs:

bitbake-prserv-tool migrate_localcount

(this needs to be executed once per MACHINE)

Technical Details

The basic scenario of using the network based PR service is that the client asks the server for a value (i.e. PRAUTO) by including the task checksum as part of the search index. The server returns the stored value if the search index is found in its backend database. If the search index is not found, the server bumps the value, stores the new index into the database, and returns the incremented value to the client. The client then includes the PRAUTO as part of the 'package revision' field value in the created package feed. The service is network based so a central database can be used to provide multiple build clients with the same value for the same index. The revision is injected into the metadata using the PKGR variable at package generation time.

• bitbake-prserv utility locks using the /tmp/PRServer_hostaddr_hostport.pid file
• uses a sqlite3 database file called prserv.sqlite3 by default
• database table used: (VERSION, PKGARCH, CHECKSUM, VALUE) with (VERSION, PKGARCH, CHECKSUM) as PRIMARY KEY
• database wrapper object supports two working modes, "hist: Preserve PR history" and "no_hist: Don't use history (delete old entries from the database)"
• currently the server forces the no_hist=True value, forcing no_hist mode.
• bitbake-prserv presents an XML RPC API:

     getPR(version, pkgarch, checksum)
     quit
     ping
     export(version=None, pkgarch=None, checksum=None, colinfo=True)
     importone(version, pkgarch, checksum, value)

Other Observations

bitbake-prserv

• needs the following variable set: PRSERV_HOST
• the PR Service can export the PR info to a file (PRSERV_DUMPFILE)
• PRSERV_LOCKDOWN can be used for releases (the PRs don't change anymore)
• cooker.py automatically starts/stops the PR server using the pre_serve and post_serve methods
• you can set the cachedir by setting the PERSISTENT_DIR OR the CACHE variables

prserv.bbclass

• defines the prserv_get_pr_auto which fetches the PR from the PR Server
• version <- PRAUTOINX; pkgarch <- PACKAGE_ARCH; checksum <- BB_TASKHASH;
• uses the __PRSERV_CONN to store the connection
• uses PRSERV_HOST_pn-<recipename> to set the HOST for a specific recipe

bitbake.conf - hook into the metadata

• PKGR ?= "${PR}${EXTENDPRAUTO}"
• EXTENDPRAUTO = "${@['.${PRAUTO\x7d',][d.getVar('PRAUTO',1) is None]}"

Export Functionality

• Used to lockdown the PRs (PRSERV_LOCKDOWN = "1" will be inserted into the file)
• When one exports the database to a file using the script, the bitbake -R conf/prexport.conf -p command runs and triggers prexport_handler (defined in the prexport.bbclass)
• ParseStarted Event: Establish the connection with the PRServer
• RecipeParsed Event: export all data for version = PRSERV_DUMPOPT_VERSION = "${PRAUTOINX}" to PRSERV_DUMPFILE
• Basically, for every recipe we export the associated info from the database table info by calling: oe.prservice.prserv_export_tofile(e.data, None, datainfo, False)
• ParseCompleted Event: export all meta-information (the structure of the database table used) to PRSERV_DUMPFILE by calling: oe.prservice.prserv_export_tofile(d, metainfo, None, True) is called

Example exported metadata

#Table: PRMAIN_nohist
#Columns:
#name        type        notn        dflt        pk
#----------  --------    --------    --------    ----
#   version      TEXT           1        None       1
#   pkgarch      TEXT           1        None       1
#  checksum      TEXT           1        None       1
#     value   INTEGER           0        None       0

# Exported Per-recipe data example
PRAUTO$zlib-1.2.7-r0$i586$fae8ba2c781fd378307f4a90c021a798 = "0"
PRAUTO_zlib-1.2.7-r0_i586 = "0"
PRAUTO$ncurses-5.9-r13.1$i586$4bcaad698464a6db2f1555aa662327ff = "0"
PRAUTO_ncurses-5.9-r13.1_i586 = "0"

Import Functionality

• When one imports a <file> using the bitbake-prserv-tool the bitbake -R conf/primport.conf -R <file> -p command runs and triggers primport_handler (defined in primport.bbclass)
• ParseStarted Event: start the prserv connection
• ParseCompleted Event: the oe.prservice.prserv_import_db(e.data) method is called at "ParseCompleted" which inserts data into the database table by parsing the PRAUTO$version$pkgarch$checksum variables