Network based PR service - Revision history

Lianhao.lu at 01:35, 12 January 2012

2012-01-12T01:35:50Z

← Older revision		Revision as of 01:35, 12 January 2012
Line 17:		Line 17:

	For many users, they may want to run the service daemon and the poky build system on the same machine most of the time. So they may not want to bother typing "bitbake-prserv --start". In this localhost case, it is desired to have poky launch the service daemon automatically at the start and stop it when the build finishes. Also it might need to run multiple daemons on the same machine at the same time.		For many users, they may want to run the service daemon and the poky build system on the same machine most of the time. So they may not want to bother typing "bitbake-prserv --start". In this localhost case, it is desired to have poky launch the service daemon automatically at the start and stop it when the build finishes. Also it might need to run multiple daemons on the same machine at the same time.

			Status: Done. To start launching local service automatically, set PRSERV_HOST to "localhost" and PRSERV_PORT to "0". See Bug #1126[http://bugzilla.pokylinux.org/show_bug.cgi?id=1126] for details.

	----'''2. Use different PR service for different recipes''' (Source: Frans Meulenbroeks)		----'''2. Use different PR service for different recipes''' (Source: Frans Meulenbroeks)
Line 25:		Line 27:

	''PRSERV_HOST_pn-myprivaterecipe = ""''		''PRSERV_HOST_pn-myprivaterecipe = ""''

			Status: Done. See Bug #1126[http://bugzilla.pokylinux.org/show_bug.cgi?id=1126] for details.

	----'''3. Support proxy''' (Source: Koen Kooi)		----'''3. Support proxy''' (Source: Koen Kooi)
Line 37:		Line 41:

	If the user uses local PR services, we need to way to dump the local databases to the central server, so other users may use the central server to generate the same package feeds as those using the local servers.		If the user uses local PR services, we need to way to dump the local databases to the central server, so other users may use the central server to generate the same package feeds as those using the local servers.

			Status: Done. See Bug #1556[http://bugzilla.pokylinux.org/show_bug.cgi?id=1556] for details.

Lianhao.lu: "Italic" the poky variables and corrected the grammar.

2011-05-30T08:18:36Z

"Italic" the poky variables and corrected the grammar.

Lianhao.lu: discussion and pending tasks of network based PR service

2011-05-30T08:03:07Z

discussion and pending tasks of network based PR service

New page

The network based PR service is a missing building block in current poky build system to enable the "basichash" BB_SIGNATURE_HANDLER for shared state. By having the network based PR service and the "basichash" signature handler, the poky users can modify the recipes without worrying about bumping the PR value manually in the recipes. The poky build system will automatically detect the change and rebuild only necessary pieces of work, and have the change reflected in the related field's value(i.e. Revision) of the final package feed. (See [https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html] for the details about shared state and BB_SIGNATURE_HANDLER).

The basic scenario of using the network based PR service is that the client asks the server for a value (i.e. PRAUTO) by including the task checksum as part of the search index. The server returns the stored value if the search index is found in its backend database. If the search index is NOT found, the server bumps the value, stores the new index into the database, and returns the incremented value to the client. The client then includes PRAUTO as part of the revision field value in the final created package feed. The service is network based so a central database can be used by providing multiple build client with the same value for the same index.

The base patches enabling the above scenario is in poky master branch now(disabled by default though). To use the PR service, the users should first start the service daemon by running the following commands:

''source ./oe-init-build-env''
''bitbake-prserv --start''

Then the users should make sure the PRHOST_ADDR and PRHOST_PORT are set correctly in conf/local.conf to match the configurations of the machine on which the service daemon is running, and then "bitbake" whatever they want to. (Please also make sure the BB_SIGNATURE_HANDLER is set to "basichash" instead of "basic").

By leaving the PRHOST_ADDR and PRHOST_PORT unset, the PR service mechanism is disabled and the poky runs exactly as same as it is now: the user needs to bump the PR manually in the recipe when changes are made to the recipe.

There are still pending tasks there, however, to make the PR service mechanism adapted to various usage model and scenarios:

----'''1. Automatically launching service daemon''' (Source: Richard Purdie)

For many users, they may want to run the service daemon and poky build system on the same machine for most of the time. So they may not want to bother typing "bitbake-prserv --start". In this localhost case, it is desired to have poky launch the service daemon automatically at start and stop it at finish. Also it might need to run multiple daemons on the same machine.

----'''2. Use different PR service for different recipes''' (Source: Frans Meulenbroeks)

Some users want to use different PRHOST or disable the PR service per recipe basis. We need to support something like:

''PRSERV_HOST_pn-myprivaterecipe = "someotherhost"''

''PRSERV_HOST_pn-myprivaterecipe = ""''

----'''3. Support proxy''' (Source: Koen Kooi)

Since it is a network based XMLRPC service, it would be nice to support the http proxy for the intranet build clients to contact internet PR service.

----'''4. Write permission control''' (Source: Koen Kooi)

Some use scenario might only allow certain authorized clients to have the write permission to bump the value in the PR service database, and other unauthorized client can only "read" the value back from the database. The question is what if an unauthorized client who doesn't have the write permission asks for value which is not stored in the database. Several ideas are brought up to handle this kind of situations. One is to generate a warning on the client side and an error in the server log. Another way is to use a specific string in the PRAUTO(i.e. custom) to indicate this situation.

----'''5. database sync-up''' (Source: Richard Purdie)

If the user uses local PR services, we need to way to dump the local databases to the central server, so other users may use the central server to generate the same package feeds as those using the local servers.

← Older revision		Revision as of 08:18, 30 May 2011
Line 1:		Line 1:
	The network based PR service is a missing building block in current poky build system to enable the "basichash" BB_SIGNATURE_HANDLER for shared state. By having the network based PR service and the "basichash" signature handler, the poky users can modify the recipes without worrying about bumping the PR value manually in the recipes. The ~~poky~~ build system will automatically detect the change ~~and~~ rebuild only necessary pieces of work, and have the ~~change~~ reflected in the ~~related~~ field's value~~(i.e. Revision)~~ of the final package feed. (See [https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html] for the details about shared state and BB_SIGNATURE_HANDLER).		The network based PR service is a missing building block in the current poky build system to enable the "basichash" ''BB_SIGNATURE_HANDLER'' for shared state. By having the network based PR service and the "basichash" signature handler, the poky users can modify the recipes without worrying about bumping the PR value manually in the recipes. The build system will automatically detect the change, rebuild only the necessary pieces of work, and have the changes reflected in the 'package revision' field value of the final package feed. (See [https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html] for the details about shared state and BB_SIGNATURE_HANDLER).

	The basic scenario of using the network based PR service is that the client asks the server for a value (i.e. PRAUTO) by including the task checksum as part of the search index. The server returns the stored value if the search index is found in its backend database. If the search index is ~~NOT found~~, the server bumps the value, stores the new index into the database, and returns the incremented value to the client. The client then includes PRAUTO as part of the revision field value in the ~~final~~ created package feed. The service is network based so a central database can be used ~~by providing~~ multiple build ~~client~~ with the same value for the same index.		The basic scenario of using the network based PR service is that the client asks the server for a value (i.e. ''PRAUTO'') by including the task checksum as part of the search index. The server returns the stored value if the search index is found in its backend database. If the search index is notfound, the server bumps the value, stores the new index into the database, and returns the incremented value to the client. The client then includes the PRAUTO as part of the 'package revision' field value in the created package feed. The service is network based so a central database can be used to provide multiple build clients with the same value for the same index.

	The base patches enabling the above scenario is in poky master branch now(disabled by default though). To use the PR service, the users should first start the service daemon by running the following commands:		The base patches enabling the above scenario is in poky master branch now(disabled by default though). To use the PR service, the users should first start the service daemon by running the following commands:
Line 8:		Line 8:
	''bitbake-prserv --start''		''bitbake-prserv --start''

	Then the users should make sure the PRHOST_ADDR and PRHOST_PORT are set correctly in conf/local.conf to match the configurations of the machine on which the service daemon is running, and then "bitbake" whatever they want to. (Please also make sure the BB_SIGNATURE_HANDLER is set to "basichash" instead of "basic").		Then the users should make sure the ''PRHOST_ADDR'' and ''PRHOST_PORT'' are set correctly in ''conf/local.conf'' to match the configurations of the machine on which the service daemon is running, and then "bitbake" whatever they want to. (Please also make sure the ''BB_SIGNATURE_HANDLER'' is set to "basichash" instead of "basic").

	By leaving the PRHOST_ADDR and PRHOST_PORT unset, the PR service mechanism is disabled and the poky runs exactly as same as it is now: the user needs to bump the PR manually in the recipe when changes are made ~~to the recipe~~.		By leaving the ''PRHOST_ADDR'' and ''PRHOST_PORT'' unset, the PR service mechanism is disabled and the poky runs exactly the same as it is now: the user needs to bump the PR manually in the recipe when changes are made .

	There are still pending tasks ~~there~~, however, to make the PR service mechanism adapted to various usage model and scenarios:		There are still some pending tasks, however, to make the PR service mechanism adapted to various usage model and scenarios:

	----'''1. Automatically launching service daemon''' (Source: Richard Purdie)		----'''1. Automatically launching service daemon''' (Source: Richard Purdie)

	For many users, they may want to run the service daemon and poky build system on the same machine ~~for~~ most of the time. So they may not want to bother typing "bitbake-prserv --start". In this localhost case, it is desired to have poky launch the service daemon automatically at start and stop it ~~at finish~~. Also it might need to run multiple daemons on the same machine.		For many users, they may want to run the service daemon and the poky build system on the same machine most of the time. So they may not want to bother typing "bitbake-prserv --start". In this localhost case, it is desired to have poky launch the service daemon automatically at the start and stop it when the build finishes. Also it might need to run multiple daemons on the same machine at the same time.

	----'''2. Use different PR service for different recipes''' (Source: Frans Meulenbroeks)		----'''2. Use different PR service for different recipes''' (Source: Frans Meulenbroeks)

	Some users want to use different ~~PRHOST~~ or disable the PR service per recipe basis. We need to support something like:		Some users want to use different PR service or disable the PR service per recipe basis. We need to support something like:

	''PRSERV_HOST_pn-myprivaterecipe = "someotherhost"''		''PRSERV_HOST_pn-myprivaterecipe = "someotherhost"''
Line 28:		Line 28:
	----'''3. Support proxy''' (Source: Koen Kooi)		----'''3. Support proxy''' (Source: Koen Kooi)

	Since it is a network based XMLRPC service, it would be nice to support the http proxy for the intranet build clients to contact ~~internet~~ PR service.		Since the PR service is a network based XMLRPC service, it would be nice to support the http proxy for the intranet build clients to contact the outside PR service.

	----'''4. Write permission control''' (Source: Koen Kooi)		----'''4. Write permission control''' (Source: Koen Kooi)

	Some ~~use~~ scenario might only allow certain authorized clients to have the write permission to bump the value in the PR service database, and other unauthorized ~~client~~ can only "read" the value back from the database. The question is what if an unauthorized client who doesn't have the write permission asks for value which is not stored in the database. Several ~~ideas~~ are brought up to handle this kind of ~~situations~~. One is to generate a warning on the client side and an error in the server log. Another way is to ~~use~~ a specific string in the PRAUTO(i.e. custom) to indicate this situation.		Some usage scenario might only allow certain authorized clients to have the write permission to bump the value in the PR service database, and other unauthorized clients can only "read" the value back from the database. The question is what if an unauthorized client who doesn't have the write permission asks for a value which is not stored in the database. Several ways are brought up to handle this kind of situation. One is to generate a warning on the client side and an error in the server log. Another way is to return a specific string as the ''PRAUTO''(i.e. "custom") to indicate this situation.

	----'''5. database sync-up''' (Source: Richard Purdie)		----'''5. database sync-up''' (Source: Richard Purdie)

	If the user uses local PR services, we need to way to dump the local databases to the central server, so other users may use the central server to generate the same package feeds as those using the local servers.		If the user uses local PR services, we need to way to dump the local databases to the central server, so other users may use the central server to generate the same package feeds as those using the local servers.