Obsolete - Yocto Buildbot Autobuilder Discussions

Place to discuss changes to the Yocto Autobuilder.

We will collaborate with OE people on this. Below is OE's proposal and requirements

New build-testing/QA-testing infrastructure

We need solution to:

1. Collect and report build env and results statistics per package;
2. Control build task execution on build slaves.

Entities we should have:

1. Client. This is client-side software which should report some build/QA-stats to some master server. Clien-side build may be started by human or by build slave. Should be some bitbake 'plugin' like oe-stats.bbclass/buildstats.bbclass.
2. Slave. Slave should periodically ask master server for new builds, fetch build params and run appropriate builds. I.e. this is client-side software for 'build farm' nodes or for individuals, who will provide his computer to run community builds.
3. Master. Master have 3 different purposes and some database behind.
   1. Stats collector. It should accept and store reports sent by clients. Most loaded part.
   2. Slave supervisor. It should control slaves and provide work for them.
   3. Interface. It should provide UI for looking reports from collected stats.

NOTE: May be terminology master-slave should be replaced with something because of political correctness.

Requirements

Here we are trying to collect requirements to create such solution.

Client

• report build environment to server at build start:
  • build start timestamp in UTC
  • target MACHINE
  • target DISTRO
  • target IMAGE
  • build host ARCH and DISTRO
  • bitbake version
  • OE metadata version
  • builder name/id
  • build type (clean/incremental)
• report to server per-package
  • package name
  • package status (successfull/failed + on which task)
  • package build duration
  • log file(s) for failed/QA-failed package
• report overall build status after build end:
  • build duration
  • build status (successfull/failed, QA issues)

Good idea to store data in persistent storage and report periodically (when?) or after build finished. Then you may even do a build offline and flush all results to server when connected to internet.

Slave

• announce his capabilities (build environment) to master on connect
• ask server for build task parameters
• do a build
• should have possibility to link slave task with build results, because results are reported by client, not by slave itself.

Master

• control slaves
• collect info from clients and slaves
• build reports on collected data:
  • "waterfall"
  • "matrix"
  • something like big table from OE's Testing page
• provide ability to search collected data at least by:
  • builder name/id
  • target params (MACHINE/DISTRO/IMAGE)
  • package name
  • build status (failed builds)
• server task queue admins should have ability to change order of tasks for slave servers and attach tasks to slaves.

Technical part

1. BuildBot http://trac.buildbot.net/ (used by Yocto)
2. Jenkins http://jenkins-ci.org/, remote access API: http://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API

Discussions

Discussion with RP on #yocto

[23:35] <Jay7> in short we (OE) are discussing now new QA- and build-testing infrastructure
[23:36] <Jay7> I've collected some requirements here: http://wiki.openembedded.net/index.php/BuildStats
[23:36] <Jay7> from merging perspective may be good idea to collaborate on this
[23:39] <RP__> Jay7: Have you seen buildstats.bbclass in poy?
[23:39] <Jay7> RP__: not yet
[23:39] <RP__> Jay7: That starts to address collecting the info you're after
[23:40] <Jay7> I have seen qemu runtime testing part and oestats-client in OE
[23:40] <RP__> Jay7: If we collect the data, you could then feed it to an oestats style service all in one
[23:40] <RP__> Jay7: So what I'm saying is that buildstats.bbclass is some effort on the Yocto side to start working on a piece of this problem
[23:40] <ka6sox> okay this would be after the entire build either works or fails?
[23:40] <Jay7> RP__: well, I'll look there then
[23:40] <RP__> ka6sox: Yes
[23:41] <ka6sox> okay does it include the logs?
[23:41] <RP__> ka6sox: At present, no
[23:41] <Jay7> RP__: have it server-side part?
[23:41] <Jay7> I mean something implemented
[23:41] <RP__> Jay7: At the moment its there purely to log the data onto disk as files
[23:42] <Jay7> ah, ok
[23:42] <RP__> Jay7: We have ideas about post processing it for various reasons, remote transmission is one of those things
[23:42] <RP__> and obviously what gets transmitted can be discussed
[23:42] <ka6sox> okay, well, since we are working on common goals comming up with a common solution would be good.
[23:42] <RP__> ka6sox: agreed, we should try and share what work we can
[23:43] <Jay7> so, you are using Buildbot as master->slave and buildstats.bbclass+some server to collect data from client builds
[23:43] <RP__> Jay7: currently we collect success/failure with buildbot
[23:43] <ka6sox> RP__, so what if we tackle the server side and you tackle the build side and we create a common dataset?
[23:43] <RP__> Jay7: buildstats is the start of our next generation plans
[23:44] <Jay7> ah, well..
[23:44] <RP__> ka6sox: We're open to any help and collaboration on the stats collection/transmission
[23:44] <Jay7> so have you plan to replace buildbot or integrate it with buildstats anyhow?
[23:45] <RP__> Jay7: I think we're moving towards collecting data separately from buildbot
[23:45] <RP__> Its UI to the data is suboptimal
[23:45] <RP__> its good at triggering builds at the right time and watching scms for changes
[23:45] <ka6sox> RP__, do you care about console on success or only on failure?
[23:46] <RP__> ka6sox: failure is certainly the more interesting
[23:46] <ka6sox> I would think so.
[23:46] <RP__> ka6sox: success has its uses as a reference but those could be turned off by default
[23:47] <Jay7> other's success is good to look when you have failure :)
[23:47] <RP__> What I really want to get to is being able to record what files were in a package, how big the package was, how long a task took to run etc
[23:47] <ka6sox> okay compression.
[23:47] <RP__> but not all data we record has to be transmitted for any given protocol
[23:47] <ka6sox> RP__ me too...right now its each individual task or the WHOLE thing.
[23:48] <ka6sox> by package would be best..as the way its reported often is minimal, 2008.1, binutils
[23:48] <ka6sox> by the package.
[23:48] <RP__> buildstats is simple right now but if you look at the mailing lists you'll see interesting graphs from even that data (http://tim.rpsys.net/bootchart.png)
[23:48] <RP__> I want to see how these things change over time
[23:49] <Jay7> cool graph
[23:49] <Jay7> I have some benchmarks but done with my own scripts :)
[23:50] <ka6sox> somewhere between "capture nothing" and "capture everything" there is capture what is needed.
[23:50] <Jay7> capture everything is better than nothing and HDD is cheap now.. and some FS have deduplication feature ;)
[23:51] <ka6sox> Jay7, okay if you want to keep it on the builders.
[23:51] <Jay7> I like doing graphs from data and look on it :)
[23:51] <ka6sox> but BW and HD space on servers that aggregate this is not.
[23:52] <zeddii> sgw, late pong. I had to go pickup a kid from school and missed the bug scrub.
[23:52] <ka6sox> I'd like to be able to get the data if needed...but to simply throw everything across the net is wasteful.
[23:52] <Jay7> I hope Intel provided good infrastucture for yocto ;)
[23:52] <Jay7> well, but I'm fine with client-only stats
[23:53] <Jay7> client-only detailed stats and server-based short stats
[23:53] <ka6sox> RP__, does buildbot allow clients that are behind corp FW's?
[23:53] <RP__> As I said, this is what I want to write onto disk during a build, not put over the wire which should be something different
[23:53] <RP__> ka6sox: I suspect not easily
[23:54] <ka6sox> nothing seems to.
[23:54] <ka6sox> the only thing I knew that worked fine with that was wanna-build.
[23:55] <Jay7> RP__: may be dedicate one TSC meeting to this?
[23:55] <Jay7> or may be non-TSC
[23:55] <ka6sox> at least its python...we can work with that.
[23:55] <RP__> Jay7: Its fine to have a meeting, we need people who have time to do the work...
[23:55] <Jay7> because OE's reaction in ML was empty
[23:56] <RP__> Jay7: We really struggled in this cycle to sort out buildstats for yocto. It is going to be assigned time for 1.1
[23:56] <ka6sox> Jay7, we have 3 folks so far...we can start with that.
[23:56] <RP__> pidge is the person who worked on buildstats for Yocto
[23:56] <RP__> She is also our release engineer so coming up to a release is a bad time to involve her in the discussions
[23:57] <ka6sox> RP__ we need to know what is important both to keep and to display.
[23:57] <Jay7> when release is planned?
[23:57] <RP__> Jay7: Start of April
[23:57] <ka6sox> the keeping isn't so bad its the display and collection.
[23:57] <RP__> We hit hard freeze on friday
[23:58] <RP__> ka6sox: I'm working on the principle that we can start to collect things whilst we work on the other parts
[23:58] <RP__> You can see my couple of hours efforts at display :)
[23:58] <ka6sox> Jay7, lets look @ what buildstats is today and see if OE can use that as a starting place.
[23:59] <ka6sox> RP__ yes, I can see there is good potential there.
[23:59] <Jay7> ok.. then let's wait for OE and Yocto releases then have continue this
[23:59] <ka6sox> Jay7, right, but lets familiarize ourselves with it so that we are ready to talk.
[23:59] <ka6sox> and ready to take action when its time.
[00:00] <RP__> ka6sox: The idea is we're planning to expand its scope over time
[00:00] <RP__> ka6sox: and most likely write something to convert the flat files into something xml like
[00:00] <RP__> (for storage/transfer purposes)
[00:00] <ka6sox> RP__, I like the idea of xml...
[00:00] <Jay7> and I dislike idea of xml :)
[00:00] <Jay7> but now it does not matters :)
[00:01] <ka6sox> most important to me are the definitions of what data is important and not.

Discussion with asac on #linaro

[03:36] <Jay7> seems you are working on build stats tools as well :)
[03:37] * Jay7 have doing some prototype for Yocto/OE
[03:37] <Jay7> https://wiki.yoctoproject.org/wiki/Yocto_Buildbot_Autobuilder_Discussions
[03:37] <Jay7> here are my ideas
[03:38] <asac> Jay7: our stuff is a cloud setup ... and assume eversything is cross build ... so we dont need slaves querying for new builds
[03:39] <asac> we just spin up an instance, build, kill it
[03:39] <asac> oh its build-testing ;)
[03:39] <asac> hehe
[03:39] <asac> nevermind
[03:39] * asac cannot read
[03:39] <Jay7> there is a lot of things combined
[03:40] <Jay7> building/testing/collecting statistics/etc
[03:40] <asac> Jay7: whats the status of that project?
[03:40] <asac> Jay7: sure, building is often part of testing/validation
[03:40] <Jay7> asac: 3 different clients, 2 different dead servers and no master/slave :)
[03:41] <asac> lol
[03:41] <Jay7> seems we are starting from scratch
[03:41] <asac> Jay7: who is leading that effort?
[03:41] <asac> Jay7: why dont you join lava ;)
[03:41] <Jay7> lava?
[03:41] <asac> thats our testing/validation infrastructure
[03:41] <asac> feels quite similar
[03:41] <Jay7> I'm slowly trying to prototype this
[03:41] <Jay7> there are 2 people around as well
[03:42] <asac> we currently focus on image testing, benchmarking etc.
[03:42] <Jay7> asac: where I can read about lava? :)
[03:42] <asac> Jay7: we have a team of 6 ;)
[03:42] <asac> Jay7: https://wiki.linaro.org/Platform/Validation/LAVA
[03:43] <Jay7> asac: your team are paid to work on this I suspect ;)
[03:43] <asac> but i think building could be rather done with something like our build service and then injected into the test farm. focus is realyl about executing tests/benchmarks on various images on various boards etc.
[03:43] <asac> Jay7: yes, thats why i think its better to join here
[03:43] <Jay7> well
[03:43] <Jay7> pidge have talk on ELC about this
[03:44] <Jay7> pidge == Elisabeth Flanagan
[03:44] <Jay7> from Yocto side
[03:44] <asac> funny
[03:44] <asac> plars also talks about LAVA there ;)
[03:44] <Jay7> cool
[03:44] <Jay7> I should say pidge about this :)
[03:44] <asac> is she the projet owner?
[03:45] <asac> i think we should discuss, yes.
[03:45] <asac> we build it for android and ubuntu on ARM atm
[03:45] <asac> so adding another thing would be just "one" more
[03:45] <Jay7> asac: yes, it would be great imho