|
|
| (24 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| Place to discuss changes to the Yocto Autobuilder.
| | Obsolete - Can be deleted |
| | |
| We will collaborate with OE people on this. Below is OE's proposal and requirements
| |
| | |
| == New build-testing/QA-testing infrastructure ==
| |
| | |
| We need solution to:
| |
| # Collect and report build env and results statistics per package;
| |
| # Control build task execution on build slaves.
| |
| | |
| === Requirements ===
| |
| Here we are trying to collect requirements to create such solution.
| |
| | |
| # '''Client'''
| |
| #* report data to server per-build:
| |
| #** target MACHINE
| |
| #** target DISTRO
| |
| #** target IMAGE
| |
| #** build host ARCH and DISTRO
| |
| #** bitbake version
| |
| #** OE metadata version
| |
| #** builder name/id
| |
| #** build type (clean/incremental)
| |
| #** build status (successfull/failed)
| |
| #* report data to server per-package
| |
| #** package name
| |
| #** package status (successfull/failed + on which task)
| |
| #** log file(s) for failed/QA-failed package
| |
| # '''Slave server'''
| |
| #* ask server for build task parameters
| |
| #* do a build
| |
| #* report results (as client above)
| |
| # '''Master server'''
| |
| #* control slaves
| |
| #* collect info from clients and slaves
| |
| #* build reports on collected data:
| |
| #** "waterfall"
| |
| #** "matrix"
| |
| #** something like big table from OE's [http://wiki.openembedded.org/index.php/Testing Testing] page
| |
| #* provide ability to search collected data at least by:
| |
| #** builder name/id
| |
| #** target params (MACHINE/DISTRO/IMAGE)
| |
| #** package name
| |
| #** build status (failed builds)
| |
| | |
| === Technical part ===
| |
| | |
| # BuildBot http://trac.buildbot.net/ (used by Yocto)
| |
| # Jenkins http://jenkins-ci.org/, remote access API: http://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
| |
| | |
| === Discussions ===
| |
| | |
| ==== Discussion with RP on #yocto ====
| |
| [23:35] <Jay7> in short we (OE) are discussing now new QA- and build-testing infrastructure<br />
| |
| [23:36] <Jay7> I've collected some requirements here: http://wiki.openembedded.net/index.php/BuildStats<br />
| |
| [23:36] <Jay7> from merging perspective may be good idea to collaborate on this<br />
| |
| [23:39] <RP__> Jay7: Have you seen buildstats.bbclass in poy?<br />
| |
| [23:39] <Jay7> RP__: not yet<br />
| |
| [23:39] <RP__> Jay7: That starts to address collecting the info you're after<br />
| |
| [23:40] <Jay7> I have seen qemu runtime testing part and oestats-client in OE<br />
| |
| [23:40] <RP__> Jay7: If we collect the data, you could then feed it to an oestats style service all in one<br />
| |
| [23:40] <RP__> Jay7: So what I'm saying is that buildstats.bbclass is some effort on the Yocto side to start working on a piece of this problem<br />
| |
| [23:40] <ka6sox> okay this would be after the entire build either works or fails?<br />
| |
| [23:40] <Jay7> RP__: well, I'll look there then<br />
| |
| [23:40] <RP__> ka6sox: Yes<br />
| |
| [23:41] <ka6sox> okay does it include the logs?<br />
| |
| [23:41] <RP__> ka6sox: At present, no<br />
| |
| [23:41] <Jay7> RP__: have it server-side part?<br />
| |
| [23:41] <Jay7> I mean something implemented<br />
| |
| [23:41] <RP__> Jay7: At the moment its there purely to log the data onto disk as files<br />
| |
| [23:42] <Jay7> ah, ok <br />
| |
| [23:42] <RP__> Jay7: We have ideas about post processing it for various reasons, remote transmission is one of those things<br />
| |
| [23:42] <RP__> and obviously what gets transmitted can be discussed<br />
| |
| [23:42] <ka6sox> okay, well, since we are working on common goals comming up with a common solution would be good.<br />
| |
| [23:42] <RP__> ka6sox: agreed, we should try and share what work we can<br />
| |
| [23:43] <Jay7> so, you are using Buildbot as master->slave and buildstats.bbclass+some server to collect data from client builds<br />
| |
| [23:43] <RP__> Jay7: currently we collect success/failure with buildbot<br />
| |
| [23:43] <ka6sox> RP__, so what if we tackle the server side and you tackle the build side and we create a common dataset?<br />
| |
| [23:43] <RP__> Jay7: buildstats is the start of our next generation plans<br />
| |
| [23:44] <Jay7> ah, well..<br />
| |
| [23:44] <RP__> ka6sox: We're open to any help and collaboration on the stats collection/transmission<br />
| |
| [23:44] <Jay7> so have you plan to replace buildbot or integrate it with buildstats anyhow?<br />
| |
| [23:45] <RP__> Jay7: I think we're moving towards collecting data separately from buildbot<br />
| |
| [23:45] <RP__> Its UI to the data is suboptimal<br />
| |
| [23:45] <RP__> its good at triggering builds at the right time and watching scms for changes<br />
| |
| [23:45] <ka6sox> RP__, do you care about console on success or only on failure?<br />
| |
| [23:46] <RP__> ka6sox: failure is certainly the more interesting<br />
| |
| [23:46] <ka6sox> I would think so.<br />
| |
| [23:46] <RP__> ka6sox: success has its uses as a reference but those could be turned off by default<br />
| |
| [23:47] <Jay7> other's success is good to look when you have failure :)<br />
| |
| [23:47] <RP__> What I really want to get to is being able to record what files were in a package, how big the package was, how long a task took to run etc<br />
| |
| [23:47] <ka6sox> okay compression.<br />
| |
| [23:47] <RP__> but not all data we record has to be transmitted for any given protocol<br />
| |
| [23:47] <ka6sox> RP__ me too...right now its each individual task or the WHOLE thing.<br />
| |
| [23:48] <ka6sox> by package would be best..as the way its reported often is minimal, 2008.1, binutils<br />
| |
| [23:48] <ka6sox> by the package.<br />
| |
| [23:48] <RP__> buildstats is simple right now but if you look at the mailing lists you'll see interesting graphs from even that data (http://tim.rpsys.net/bootchart.png)<br />
| |
| [23:48] <RP__> I want to see how these things change over time<br />
| |
| [23:49] <Jay7> cool graph<br />
| |
| [23:49] <Jay7> I have some benchmarks but done with my own scripts :)<br />
| |
| [23:50] <ka6sox> somewhere between "capture nothing" and "capture everything" there is capture what is needed.<br />
| |
| [23:50] <Jay7> capture everything is better than nothing and HDD is cheap now.. and some FS have deduplication feature ;)<br />
| |
| [23:51] <ka6sox> Jay7, okay if you want to keep it on the builders.<br />
| |
| [23:51] <Jay7> I like doing graphs from data and look on it :)<br />
| |
| [23:51] <ka6sox> but BW and HD space on servers that aggregate this is not.<br />
| |
| [23:52] <zeddii> sgw, late pong. I had to go pickup a kid from school and missed the bug scrub.<br />
| |
| [23:52] <ka6sox> I'd like to be able to get the data if needed...but to simply throw everything across the net is wasteful.<br />
| |
| [23:52] <Jay7> I hope Intel provided good infrastucture for yocto ;)<br />
| |
| [23:52] <Jay7> well, but I'm fine with client-only stats<br />
| |
| [23:53] <Jay7> client-only detailed stats and server-based short stats<br />
| |
| [23:53] <ka6sox> RP__, does buildbot allow clients that are behind corp FW's?<br />
| |
| [23:53] <RP__> As I said, this is what I want to write onto disk during a build, not put over the wire which should be something different<br />
| |
| [23:53] <RP__> ka6sox: I suspect not easily<br />
| |
| [23:54] <ka6sox> nothing seems to.<br />
| |
| [23:54] <ka6sox> the only thing I knew that worked fine with that was wanna-build.<br />
| |
| [23:55] <Jay7> RP__: may be dedicate one TSC meeting to this?<br />
| |
| [23:55] <Jay7> or may be non-TSC<br />
| |
| [23:55] <ka6sox> at least its python...we can work with that.<br />
| |
| [23:55] <RP__> Jay7: Its fine to have a meeting, we need people who have time to do the work...<br />
| |
| [23:55] <Jay7> because OE's reaction in ML was empty<br />
| |
| [23:56] <RP__> Jay7: We really struggled in this cycle to sort out buildstats for yocto. It is going to be assigned time for 1.1<br />
| |
| [23:56] <ka6sox> Jay7, we have 3 folks so far...we can start with that.<br />
| |
| [23:56] <RP__> pidge is the person who worked on buildstats for Yocto<br />
| |
| [23:56] <RP__> She is also our release engineer so coming up to a release is a bad time to involve her in the discussions<br />
| |
| [23:57] <ka6sox> RP__ we need to know what is important both to keep and to display.<br />
| |
| [23:57] <Jay7> when release is planned?<br />
| |
| [23:57] <RP__> Jay7: Start of April<br />
| |
| [23:57] <ka6sox> the keeping isn't so bad its the display and collection.<br />
| |
| [23:57] <RP__> We hit hard freeze on friday<br />
| |
| [23:58] <RP__> ka6sox: I'm working on the principle that we can start to collect things whilst we work on the other parts<br />
| |
| [23:58] <RP__> You can see my couple of hours efforts at display :)<br />
| |
| [23:58] <ka6sox> Jay7, lets look @ what buildstats is today and see if OE can use that as a starting place.<br />
| |
| [23:59] <ka6sox> RP__ yes, I can see there is good potential there.<br />
| |
| [23:59] <Jay7> ok.. then let's wait for OE and Yocto releases then have continue this<br />
| |
| [23:59] <ka6sox> Jay7, right, but lets familiarize ourselves with it so that we are ready to talk.<br />
| |
| [23:59] <ka6sox> and ready to take action when its time.<br />
| |
| [00:00] <RP__> ka6sox: The idea is we're planning to expand its scope over time<br />
| |
| [00:00] <RP__> ka6sox: and most likely write something to convert the flat files into something xml like<br />
| |
| [00:00] <RP__> (for storage/transfer purposes)<br />
| |
| [00:00] <ka6sox> RP__, I like the idea of xml...<br />
| |
| [00:00] <Jay7> and I dislike idea of xml :)<br />
| |
| [00:00] <Jay7> but now it does not matters :)<br />
| |
| [00:01] <ka6sox> most important to me are the definitions of what data is important and not.<br />
| |
