Obsolete - Yocto Buildbot Autobuilder Discussions

Place to discuss changes to the Yocto Autobuilder.

We will collaborate with OE people on this. Below is OE's proposal and requirements

New build-testing/QA-testing infrastructure

We need solution to:

1. Collect and report build env and results statistics per package;
2. Control build task execution on build slaves.

Requirements

Here we are trying to collect requirements to create such solution.

1. Client
   • report data to server per-build:
     • target MACHINE
     • target DISTRO
     • target IMAGE
     • build host ARCH and DISTRO
     • bitbake version
     • OE metadata version
     • builder name/id
     • build type (clean/incremental)
     • build status (successfull/failed)
   • report data to server per-package
     • package name
     • package status (successfull/failed + on which task)
     • log file(s) for failed/QA-failed package
2. Slave server
   • ask server for build task parameters
   • do a build
   • report results (as client above)
3. Master server
   • control slaves
   • collect info from clients and slaves
   • build reports on collected data:
     • "waterfall"
     • "matrix"
     • something like big table from OE's Testing page
   • provide ability to search collected data at least by:
     • builder name/id
     • target params (MACHINE/DISTRO/IMAGE)
     • package name
     • build status (failed builds)

Technical part

1. BuildBot http://trac.buildbot.net/ (used by Yocto)
2. Jenkins http://jenkins-ci.org/, remote access API: http://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API

Discussions

Discussion with RP on #yocto

[23:35] <Jay7> in short we (OE) are discussing now new QA- and build-testing infrastructure
[23:36] <Jay7> I've collected some requirements here: http://wiki.openembedded.net/index.php/BuildStats
[23:36] <Jay7> from merging perspective may be good idea to collaborate on this
[23:39] <RP__> Jay7: Have you seen buildstats.bbclass in poy?
[23:39] <Jay7> RP__: not yet
[23:39] <RP__> Jay7: That starts to address collecting the info you're after
[23:40] <Jay7> I have seen qemu runtime testing part and oestats-client in OE
[23:40] <RP__> Jay7: If we collect the data, you could then feed it to an oestats style service all in one
[23:40] <RP__> Jay7: So what I'm saying is that buildstats.bbclass is some effort on the Yocto side to start working on a piece of this problem
[23:40] <ka6sox> okay this would be after the entire build either works or fails?
[23:40] <Jay7> RP__: well, I'll look there then
[23:40] <RP__> ka6sox: Yes
[23:41] <ka6sox> okay does it include the logs?
[23:41] <RP__> ka6sox: At present, no
[23:41] <Jay7> RP__: have it server-side part?
[23:41] <Jay7> I mean something implemented
[23:41] <RP__> Jay7: At the moment its there purely to log the data onto disk as files
[23:42] <Jay7> ah, ok
[23:42] <RP__> Jay7: We have ideas about post processing it for various reasons, remote transmission is one of those things
[23:42] <RP__> and obviously what gets transmitted can be discussed
[23:42] <ka6sox> okay, well, since we are working on common goals comming up with a common solution would be good.
[23:42] <RP__> ka6sox: agreed, we should try and share what work we can
[23:43] <Jay7> so, you are using Buildbot as master->slave and buildstats.bbclass+some server to collect data from client builds
[23:43] <RP__> Jay7: currently we collect success/failure with buildbot
[23:43] <ka6sox> RP__, so what if we tackle the server side and you tackle the build side and we create a common dataset?
[23:43] <RP__> Jay7: buildstats is the start of our next generation plans
[23:44] <Jay7> ah, well..
[23:44] <RP__> ka6sox: We're open to any help and collaboration on the stats collection/transmission
[23:44] <Jay7> so have you plan to replace buildbot or integrate it with buildstats anyhow?
[23:45] <RP__> Jay7: I think we're moving towards collecting data separately from buildbot
[23:45] <RP__> Its UI to the data is suboptimal
[23:45] <RP__> its good at triggering builds at the right time and watching scms for changes
[23:45] <ka6sox> RP__, do you care about console on success or only on failure?
[23:46] <RP__> ka6sox: failure is certainly the more interesting
[23:46] <ka6sox> I would think so.
[23:46] <RP__> ka6sox: success has its uses as a reference but those could be turned off by default
[23:47] <Jay7> other's success is good to look when you have failure :)
[23:47] <RP__> What I really want to get to is being able to record what files were in a package, how big the package was, how long a task took to run etc
[23:47] <ka6sox> okay compression.
[23:47] <RP__> but not all data we record has to be transmitted for any given protocol
[23:47] <ka6sox> RP__ me too...right now its each individual task or the WHOLE thing.
[23:48] <ka6sox> by package would be best..as the way its reported often is minimal, 2008.1, binutils
[23:48] <ka6sox> by the package.
[23:48] <RP__> buildstats is simple right now but if you look at the mailing lists you'll see interesting graphs from even that data (http://tim.rpsys.net/bootchart.png)
[23:48] <RP__> I want to see how these things change over time
[23:49] <Jay7> cool graph
[23:49] <Jay7> I have some benchmarks but done with my own scripts :)
[23:50] <ka6sox> somewhere between "capture nothing" and "capture everything" there is capture what is needed.
[23:50] <Jay7> capture everything is better than nothing and HDD is cheap now.. and some FS have deduplication feature ;)
[23:51] <ka6sox> Jay7, okay if you want to keep it on the builders.
[23:51] <Jay7> I like doing graphs from data and look on it :)
[23:51] <ka6sox> but BW and HD space on servers that aggregate this is not.
[23:52] <zeddii> sgw, late pong. I had to go pickup a kid from school and missed the bug scrub.
[23:52] <ka6sox> I'd like to be able to get the data if needed...but to simply throw everything across the net is wasteful.
[23:52] <Jay7> I hope Intel provided good infrastucture for yocto ;)
[23:52] <Jay7> well, but I'm fine with client-only stats
[23:53] <Jay7> client-only detailed stats and server-based short stats
[23:53] <ka6sox> RP__, does buildbot allow clients that are behind corp FW's?
[23:53] <RP__> As I said, this is what I want to write onto disk during a build, not put over the wire which should be something different
[23:53] <RP__> ka6sox: I suspect not easily
[23:54] <ka6sox> nothing seems to.
[23:54] <ka6sox> the only thing I knew that worked fine with that was wanna-build.
[23:55] <Jay7> RP__: may be dedicate one TSC meeting to this?
[23:55] <Jay7> or may be non-TSC
[23:55] <ka6sox> at least its python...we can work with that.
[23:55] <RP__> Jay7: Its fine to have a meeting, we need people who have time to do the work...
[23:55] <Jay7> because OE's reaction in ML was empty
[23:56] <RP__> Jay7: We really struggled in this cycle to sort out buildstats for yocto. It is going to be assigned time for 1.1
[23:56] <ka6sox> Jay7, we have 3 folks so far...we can start with that.
[23:56] <RP__> pidge is the person who worked on buildstats for Yocto
[23:56] <RP__> She is also our release engineer so coming up to a release is a bad time to involve her in the discussions
[23:57] <ka6sox> RP__ we need to know what is important both to keep and to display.
[23:57] <Jay7> when release is planned?
[23:57] <RP__> Jay7: Start of April
[23:57] <ka6sox> the keeping isn't so bad its the display and collection.
[23:57] <RP__> We hit hard freeze on friday
[23:58] <RP__> ka6sox: I'm working on the principle that we can start to collect things whilst we work on the other parts
[23:58] <RP__> You can see my couple of hours efforts at display :)
[23:58] <ka6sox> Jay7, lets look @ what buildstats is today and see if OE can use that as a starting place.
[23:59] <ka6sox> RP__ yes, I can see there is good potential there.
[23:59] <Jay7> ok.. then let's wait for OE and Yocto releases then have continue this
[23:59] <ka6sox> Jay7, right, but lets familiarize ourselves with it so that we are ready to talk.
[23:59] <ka6sox> and ready to take action when its time.
[00:00] <RP__> ka6sox: The idea is we're planning to expand its scope over time
[00:00] <RP__> ka6sox: and most likely write something to convert the flat files into something xml like
[00:00] <RP__> (for storage/transfer purposes)
[00:00] <ka6sox> RP__, I like the idea of xml...
[00:00] <Jay7> and I dislike idea of xml :)
[00:00] <Jay7> but now it does not matters :)
[00:01] <ka6sox> most important to me are the definitions of what data is important and not.