|
|
| (5 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| Place to discuss changes to the Yocto Autobuilder.
| | Obsolete - Can be deleted |
| | |
| We will collaborate with OE people on this. Below is OE's proposal and requirements
| |
| | |
| == New build-testing/QA-testing infrastructure ==
| |
| | |
| We need solution to:
| |
| # Collect and report build env and results statistics per package;
| |
| # Control build task execution on build slaves.
| |
| | |
| Entities we should have:
| |
| # '''Client'''. This is client-side software which should report some build/QA-stats to some master server. Clien-side build may be started by human or by build slave. Should be some bitbake 'plugin' like oe-stats.bbclass/buildstats.bbclass.
| |
| # '''Slave'''. Slave should periodically ask master server for new builds, fetch build params and run appropriate builds. I.e. this is client-side software for 'build farm' nodes or for individuals, who will provide his computer to run community builds.
| |
| # '''Master'''. Master have 3 different purposes and some database behind.
| |
| ## '''Stats collector'''. It should accept and store reports sent by clients. Most loaded part.
| |
| ## '''Slave supervisor'''. It should control slaves and provide work for them.
| |
| ## '''Interface'''. It should provide UI for looking reports from collected stats.
| |
| | |
| NOTE: May be terminology master-slave should be replaced with something because of political correctness.
| |
| | |
| == Requirements ==
| |
| Here we are trying to collect requirements to create such solution.
| |
| | |
| === Client ===
| |
| * report build environment to server at build start:
| |
| ** build start timestamp in UTC
| |
| ** target MACHINE
| |
| ** target DISTRO
| |
| ** target IMAGE
| |
| ** build host ARCH and DISTRO
| |
| ** bitbake version
| |
| ** OE metadata version
| |
| ** builder name/id
| |
| ** build type (clean/incremental)
| |
| * report to server per-package
| |
| ** package name
| |
| ** package status (successfull/failed + on which task)
| |
| ** package build duration
| |
| ** log file(s) for failed/QA-failed package
| |
| * report overall build status after build end:
| |
| ** build duration
| |
| ** build status (successfull/failed, QA issues)
| |
| | |
| Good idea to store data in persistent storage and report periodically (when?) or after build finished. Then you may even do a build offline and flush all results to server when connected to internet.
| |
| | |
| === Slave ===
| |
| * announce his capabilities (build environment) to master on connect
| |
| * ask server for build task parameters
| |
| * do a build
| |
| * should have possibility to link slave task with build results, because results are reported by client, not by slave itself.
| |
| | |
| === Master ===
| |
| * control slaves
| |
| * collect info from clients and slaves
| |
| * build reports on collected data:
| |
| ** "waterfall"
| |
| ** "matrix"
| |
| ** something like big table from OE's [http://wiki.openembedded.org/index.php/Testing Testing] page
| |
| * provide ability to search collected data at least by:
| |
| ** builder name/id
| |
| ** target params (MACHINE/DISTRO/IMAGE)
| |
| ** package name
| |
| ** build status (failed builds)
| |
| * server task queue admins should have ability to change order of tasks for slave servers and attach tasks to slaves.
| |
| | |
| == Details ==
| |
| | |
| | |
| == Technical part ==
| |
| | |
| # BuildBot http://trac.buildbot.net/ (used by Yocto)
| |
| # Jenkins http://jenkins-ci.org/, remote access API: http://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
| |
| # LAVA https://wiki.linaro.org/Platform/Validation/LAVA
| |
| | |
| == Discussions ==
| |
| | |
| === Discussion with RP on #yocto ===
| |
| [23:35] <Jay7> in short we (OE) are discussing now new QA- and build-testing infrastructure<br />
| |
| [23:36] <Jay7> I've collected some requirements here: http://wiki.openembedded.net/index.php/BuildStats<br />
| |
| [23:36] <Jay7> from merging perspective may be good idea to collaborate on this<br />
| |
| [23:39] <RP__> Jay7: Have you seen buildstats.bbclass in poy?<br />
| |
| [23:39] <Jay7> RP__: not yet<br />
| |
| [23:39] <RP__> Jay7: That starts to address collecting the info you're after<br />
| |
| [23:40] <Jay7> I have seen qemu runtime testing part and oestats-client in OE<br />
| |
| [23:40] <RP__> Jay7: If we collect the data, you could then feed it to an oestats style service all in one<br />
| |
| [23:40] <RP__> Jay7: So what I'm saying is that buildstats.bbclass is some effort on the Yocto side to start working on a piece of this problem<br />
| |
| [23:40] <ka6sox> okay this would be after the entire build either works or fails?<br />
| |
| [23:40] <Jay7> RP__: well, I'll look there then<br />
| |
| [23:40] <RP__> ka6sox: Yes<br />
| |
| [23:41] <ka6sox> okay does it include the logs?<br />
| |
| [23:41] <RP__> ka6sox: At present, no<br />
| |
| [23:41] <Jay7> RP__: have it server-side part?<br />
| |
| [23:41] <Jay7> I mean something implemented<br />
| |
| [23:41] <RP__> Jay7: At the moment its there purely to log the data onto disk as files<br />
| |
| [23:42] <Jay7> ah, ok <br />
| |
| [23:42] <RP__> Jay7: We have ideas about post processing it for various reasons, remote transmission is one of those things<br />
| |
| [23:42] <RP__> and obviously what gets transmitted can be discussed<br />
| |
| [23:42] <ka6sox> okay, well, since we are working on common goals comming up with a common solution would be good.<br />
| |
| [23:42] <RP__> ka6sox: agreed, we should try and share what work we can<br />
| |
| [23:43] <Jay7> so, you are using Buildbot as master->slave and buildstats.bbclass+some server to collect data from client builds<br />
| |
| [23:43] <RP__> Jay7: currently we collect success/failure with buildbot<br />
| |
| [23:43] <ka6sox> RP__, so what if we tackle the server side and you tackle the build side and we create a common dataset?<br />
| |
| [23:43] <RP__> Jay7: buildstats is the start of our next generation plans<br />
| |
| [23:44] <Jay7> ah, well..<br />
| |
| [23:44] <RP__> ka6sox: We're open to any help and collaboration on the stats collection/transmission<br />
| |
| [23:44] <Jay7> so have you plan to replace buildbot or integrate it with buildstats anyhow?<br />
| |
| [23:45] <RP__> Jay7: I think we're moving towards collecting data separately from buildbot<br />
| |
| [23:45] <RP__> Its UI to the data is suboptimal<br />
| |
| [23:45] <RP__> its good at triggering builds at the right time and watching scms for changes<br />
| |
| [23:45] <ka6sox> RP__, do you care about console on success or only on failure?<br />
| |
| [23:46] <RP__> ka6sox: failure is certainly the more interesting<br />
| |
| [23:46] <ka6sox> I would think so.<br />
| |
| [23:46] <RP__> ka6sox: success has its uses as a reference but those could be turned off by default<br />
| |
| [23:47] <Jay7> other's success is good to look when you have failure :)<br />
| |
| [23:47] <RP__> What I really want to get to is being able to record what files were in a package, how big the package was, how long a task took to run etc<br />
| |
| [23:47] <ka6sox> okay compression.<br />
| |
| [23:47] <RP__> but not all data we record has to be transmitted for any given protocol<br />
| |
| [23:47] <ka6sox> RP__ me too...right now its each individual task or the WHOLE thing.<br />
| |
| [23:48] <ka6sox> by package would be best..as the way its reported often is minimal, 2008.1, binutils<br />
| |
| [23:48] <ka6sox> by the package.<br />
| |
| [23:48] <RP__> buildstats is simple right now but if you look at the mailing lists you'll see interesting graphs from even that data (http://tim.rpsys.net/bootchart.png)<br />
| |
| [23:48] <RP__> I want to see how these things change over time<br />
| |
| [23:49] <Jay7> cool graph<br />
| |
| [23:49] <Jay7> I have some benchmarks but done with my own scripts :)<br />
| |
| [23:50] <ka6sox> somewhere between "capture nothing" and "capture everything" there is capture what is needed.<br />
| |
| [23:50] <Jay7> capture everything is better than nothing and HDD is cheap now.. and some FS have deduplication feature ;)<br />
| |
| [23:51] <ka6sox> Jay7, okay if you want to keep it on the builders.<br />
| |
| [23:51] <Jay7> I like doing graphs from data and look on it :)<br />
| |
| [23:51] <ka6sox> but BW and HD space on servers that aggregate this is not.<br />
| |
| [23:52] <zeddii> sgw, late pong. I had to go pickup a kid from school and missed the bug scrub.<br />
| |
| [23:52] <ka6sox> I'd like to be able to get the data if needed...but to simply throw everything across the net is wasteful.<br />
| |
| [23:52] <Jay7> I hope Intel provided good infrastucture for yocto ;)<br />
| |
| [23:52] <Jay7> well, but I'm fine with client-only stats<br />
| |
| [23:53] <Jay7> client-only detailed stats and server-based short stats<br />
| |
| [23:53] <ka6sox> RP__, does buildbot allow clients that are behind corp FW's?<br />
| |
| [23:53] <RP__> As I said, this is what I want to write onto disk during a build, not put over the wire which should be something different<br />
| |
| [23:53] <RP__> ka6sox: I suspect not easily<br />
| |
| [23:54] <ka6sox> nothing seems to.<br />
| |
| [23:54] <ka6sox> the only thing I knew that worked fine with that was wanna-build.<br />
| |
| [23:55] <Jay7> RP__: may be dedicate one TSC meeting to this?<br />
| |
| [23:55] <Jay7> or may be non-TSC<br />
| |
| [23:55] <ka6sox> at least its python...we can work with that.<br />
| |
| [23:55] <RP__> Jay7: Its fine to have a meeting, we need people who have time to do the work...<br />
| |
| [23:55] <Jay7> because OE's reaction in ML was empty<br />
| |
| [23:56] <RP__> Jay7: We really struggled in this cycle to sort out buildstats for yocto. It is going to be assigned time for 1.1<br />
| |
| [23:56] <ka6sox> Jay7, we have 3 folks so far...we can start with that.<br />
| |
| [23:56] <RP__> pidge is the person who worked on buildstats for Yocto<br />
| |
| [23:56] <RP__> She is also our release engineer so coming up to a release is a bad time to involve her in the discussions<br />
| |
| [23:57] <ka6sox> RP__ we need to know what is important both to keep and to display.<br />
| |
| [23:57] <Jay7> when release is planned?<br />
| |
| [23:57] <RP__> Jay7: Start of April<br />
| |
| [23:57] <ka6sox> the keeping isn't so bad its the display and collection.<br />
| |
| [23:57] <RP__> We hit hard freeze on friday<br />
| |
| [23:58] <RP__> ka6sox: I'm working on the principle that we can start to collect things whilst we work on the other parts<br />
| |
| [23:58] <RP__> You can see my couple of hours efforts at display :)<br />
| |
| [23:58] <ka6sox> Jay7, lets look @ what buildstats is today and see if OE can use that as a starting place.<br />
| |
| [23:59] <ka6sox> RP__ yes, I can see there is good potential there.<br />
| |
| [23:59] <Jay7> ok.. then let's wait for OE and Yocto releases then have continue this<br />
| |
| [23:59] <ka6sox> Jay7, right, but lets familiarize ourselves with it so that we are ready to talk.<br />
| |
| [23:59] <ka6sox> and ready to take action when its time.<br />
| |
| [00:00] <RP__> ka6sox: The idea is we're planning to expand its scope over time<br />
| |
| [00:00] <RP__> ka6sox: and most likely write something to convert the flat files into something xml like<br />
| |
| [00:00] <RP__> (for storage/transfer purposes)<br />
| |
| [00:00] <ka6sox> RP__, I like the idea of xml...<br />
| |
| [00:00] <Jay7> and I dislike idea of xml :)<br />
| |
| [00:00] <Jay7> but now it does not matters :)<br />
| |
| [00:01] <ka6sox> most important to me are the definitions of what data is important and not.<br />
| |
| | |
| | |
| === Discussion with asac on #linaro ===
| |
| [03:36] <Jay7> seems you are working on build stats tools as well :)<br />
| |
| [03:37] * Jay7 have doing some prototype for Yocto/OE<br />
| |
| [03:37] <Jay7> https://wiki.yoctoproject.org/wiki/Yocto_Buildbot_Autobuilder_Discussions<br />
| |
| [03:37] <Jay7> here are my ideas<br />
| |
| [03:38] <asac> Jay7: our stuff is a cloud setup ... and assume eversything is cross build ... so we dont need slaves querying for new builds<br />
| |
| [03:39] <asac> we just spin up an instance, build, kill it<br />
| |
| [03:39] <asac> oh its build-testing ;)<br />
| |
| [03:39] <asac> hehe<br />
| |
| [03:39] <asac> nevermind<br />
| |
| [03:39] * asac cannot read<br />
| |
| [03:39] <Jay7> there is a lot of things combined<br />
| |
| [03:40] <Jay7> building/testing/collecting statistics/etc<br />
| |
| [03:40] <asac> Jay7: whats the status of that project?<br />
| |
| [03:40] <asac> Jay7: sure, building is often part of testing/validation<br />
| |
| [03:40] <Jay7> asac: 3 different clients, 2 different dead servers and no master/slave :)<br />
| |
| [03:41] <asac> lol<br />
| |
| [03:41] <Jay7> seems we are starting from scratch<br />
| |
| [03:41] <asac> Jay7: who is leading that effort?<br />
| |
| [03:41] <asac> Jay7: why dont you join lava ;)<br />
| |
| [03:41] <Jay7> lava?<br />
| |
| [03:41] <asac> thats our testing/validation infrastructure<br />
| |
| [03:41] <asac> feels quite similar<br />
| |
| [03:41] <Jay7> I'm slowly trying to prototype this<br />
| |
| [03:41] <Jay7> there are 2 people around as well<br />
| |
| [03:42] <asac> we currently focus on image testing, benchmarking etc.<br />
| |
| [03:42] <Jay7> asac: where I can read about lava? :)<br />
| |
| [03:42] <asac> Jay7: we have a team of 6 ;)<br />
| |
| [03:42] <asac> Jay7: https://wiki.linaro.org/Platform/Validation/LAVA<br />
| |
| [03:43] <Jay7> asac: your team are paid to work on this I suspect ;)<br />
| |
| [03:43] <asac> but i think building could be rather done with something like our build service and then injected into the test farm. focus is realyl about executing tests/benchmarks on various images on various boards etc.<br />
| |
| [03:43] <asac> Jay7: yes, thats why i think its better to join here<br />
| |
| [03:43] <Jay7> well<br />
| |
| [03:43] <Jay7> pidge have talk on ELC about this<br />
| |
| [03:44] <Jay7> pidge == Elisabeth Flanagan<br />
| |
| [03:44] <Jay7> from Yocto side<br />
| |
| [03:44] <asac> funny<br />
| |
| [03:44] <asac> plars also talks about LAVA there ;)<br />
| |
| [03:44] <Jay7> cool<br />
| |
| [03:44] <Jay7> I should say pidge about this :)<br />
| |
| [03:44] <asac> is she the projet owner?<br />
| |
| [03:45] <asac> i think we should discuss, yes.<br />
| |
| [03:45] <asac> we build it for android and ubuntu on ARM atm<br />
| |
| [03:45] <asac> so adding another thing would be just "one" more<br />
| |
| [03:45] <Jay7> asac: yes, it would be great imho<br />
| |
