Obsolete - Yocto Buildbot Autobuilder Discussions: Difference between revisions

From Yocto Project
Jump to navigationJump to search
m (Michael Opdenacker moved page Yocto Buildbot Autobuilder Discussions to Obsolete - Yocto Buildbot Autobuilder Discussions: Obsolete contents, can be deleted)
 
(24 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Place to discuss changes to the Yocto Autobuilder.
Obsolete - Can be deleted
 
We will collaborate with OE people on this. Below is OE's proposal and requirements
 
== New build-testing/QA-testing infrastructure ==
 
We need solution to:
# Collect and report build env and results statistics per package;
# Control build task execution on build slaves.
 
=== Requirements ===
Here we are trying to collect requirements to create such solution.
 
# '''Client'''
#* report data to server per-build:
#** target MACHINE
#** target DISTRO
#** target IMAGE
#** build host ARCH and DISTRO
#** bitbake version
#** OE metadata version
#** builder name/id
#** build type (clean/incremental)
#** build status (successfull/failed)
#* report data to server per-package
#** package name
#** package status (successfull/failed + on which task)
#** log file(s) for failed/QA-failed package
# '''Slave server'''
#* ask server for build task parameters
#* do a build
#* report results (as client above)
# '''Master server'''
#* control slaves
#* collect info from clients and slaves
#* build reports on collected data:
#** "waterfall"
#** "matrix"
#** something like big table from OE's [http://wiki.openembedded.org/index.php/Testing Testing] page
#* provide ability to search collected data at least by:
#** builder name/id
#** target params (MACHINE/DISTRO/IMAGE)
#** package name
#** build status (failed builds)
 
=== Technical part ===
 
# BuildBot http://trac.buildbot.net/ (used by Yocto)
# Jenkins http://jenkins-ci.org/, remote access API: http://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
 
=== Discussions ===
 
==== Discussion with RP on #yocto ====
[23:35] <Jay7> in short we (OE) are discussing now new QA- and build-testing infrastructure<br />
[23:36] <Jay7> I've collected some requirements here: http://wiki.openembedded.net/index.php/BuildStats<br />
[23:36] <Jay7> from merging perspective may be good idea to collaborate on this<br />
[23:39] <RP__> Jay7: Have you seen buildstats.bbclass in poy?<br />
[23:39] <Jay7> RP__: not yet<br />
[23:39] <RP__> Jay7: That starts to address collecting the info you're after<br />
[23:40] <Jay7> I have seen qemu runtime testing part and oestats-client in OE<br />
[23:40] <RP__> Jay7: If we collect the data, you could then feed it to an oestats style service all in one<br />
[23:40] <RP__> Jay7: So what I'm saying is that buildstats.bbclass is some effort on the Yocto side to start working on a piece of this problem<br />
[23:40] <ka6sox> okay this would be after the entire build either works or fails?<br />
[23:40] <Jay7> RP__: well, I'll look there then<br />
[23:40] <RP__> ka6sox: Yes<br />
[23:41] <ka6sox> okay does it include the logs?<br />
[23:41] <RP__> ka6sox: At present, no<br />
[23:41] <Jay7> RP__: have it server-side part?<br />
[23:41] <Jay7> I mean something implemented<br />
[23:41] <RP__> Jay7: At the moment its there purely to log the data onto disk as files<br />
[23:42] <Jay7> ah, ok <br />
[23:42] <RP__> Jay7: We have ideas about post processing it for various reasons, remote transmission is one of those things<br />
[23:42] <RP__> and obviously what gets transmitted can be discussed<br />
[23:42] <ka6sox> okay, well, since we are working on common goals comming up with a common solution would be good.<br />
[23:42] <RP__> ka6sox: agreed, we should try and share what work we can<br />
[23:43] <Jay7> so, you are using Buildbot as master->slave and buildstats.bbclass+some server to collect data from client builds<br />
[23:43] <RP__> Jay7: currently we collect success/failure with buildbot<br />
[23:43] <ka6sox> RP__, so what if we tackle the server side and you tackle the build side and we create a common dataset?<br />
[23:43] <RP__> Jay7: buildstats is the start of our next generation plans<br />
[23:44] <Jay7> ah, well..<br />
[23:44] <RP__> ka6sox: We're open to any help and collaboration on the stats collection/transmission<br />
[23:44] <Jay7> so have you plan to replace buildbot or integrate it with buildstats anyhow?<br />
[23:45] <RP__> Jay7: I think we're moving towards collecting data separately from buildbot<br />
[23:45] <RP__> Its UI to the data is suboptimal<br />
[23:45] <RP__> its good at triggering builds at the right time and watching scms for changes<br />
[23:45] <ka6sox> RP__, do you care about console on success or only on failure?<br />
[23:46] <RP__> ka6sox: failure is certainly the more interesting<br />
[23:46] <ka6sox> I would think so.<br />
[23:46] <RP__> ka6sox: success has its uses as a reference but those could be turned off by default<br />
[23:47] <Jay7> other's success is good to look when you have failure :)<br />
[23:47] <RP__> What I really want to get to is being able to record what files were in a package, how big the package was, how long a task took to run etc<br />
[23:47] <ka6sox> okay compression.<br />
[23:47] <RP__> but not all data we record has to be transmitted for any given protocol<br />
[23:47] <ka6sox> RP__ me too...right now its each individual task or the WHOLE thing.<br />
[23:48] <ka6sox> by package would be best..as the way its reported often is minimal, 2008.1, binutils<br />
[23:48] <ka6sox> by the package.<br />
[23:48] <RP__> buildstats is simple right now but if you look at the mailing lists you'll see interesting graphs from even that data (http://tim.rpsys.net/bootchart.png)<br />
[23:48] <RP__> I want to see how these things change over time<br />
[23:49] <Jay7> cool graph<br />
[23:49] <Jay7> I have some benchmarks but done with my own scripts :)<br />
[23:50] <ka6sox> somewhere between "capture nothing" and "capture everything" there is capture what is needed.<br />
[23:50] <Jay7> capture everything is better than nothing and HDD is cheap now.. and some FS have deduplication feature ;)<br />
[23:51] <ka6sox> Jay7, okay if you want to keep it on the builders.<br />
[23:51] <Jay7> I like doing graphs from data and look on it :)<br />
[23:51] <ka6sox> but BW and HD space on servers that aggregate this is not.<br />
[23:52] <zeddii> sgw, late pong. I had to go pickup a kid from school and missed the bug scrub.<br />
[23:52] <ka6sox> I'd like to be able to get the data if needed...but to simply throw everything across the net is wasteful.<br />
[23:52] <Jay7> I hope Intel provided good infrastucture for yocto ;)<br />
[23:52] <Jay7> well, but I'm fine with client-only stats<br />
[23:53] <Jay7> client-only detailed stats and server-based short stats<br />
[23:53] <ka6sox> RP__, does buildbot allow clients that are behind corp FW's?<br />
[23:53] <RP__> As I said, this is what I want to write onto disk during a build, not put over the wire which should be something different<br />
[23:53] <RP__> ka6sox: I suspect not easily<br />
[23:54] <ka6sox> nothing seems to.<br />
[23:54] <ka6sox> the only thing I knew that worked fine with that was wanna-build.<br />
[23:55] <Jay7> RP__: may be dedicate one TSC meeting to this?<br />
[23:55] <Jay7> or may be non-TSC<br />
[23:55] <ka6sox> at least its python...we can work with that.<br />
[23:55] <RP__> Jay7: Its fine to have a meeting, we need people who have time to do the work...<br />
[23:55] <Jay7> because OE's reaction in ML was empty<br />
[23:56] <RP__> Jay7: We really struggled in this cycle to sort out buildstats for yocto. It is going to be assigned time for 1.1<br />
[23:56] <ka6sox> Jay7, we have 3 folks so far...we can start with that.<br />
[23:56] <RP__> pidge is the person who worked on buildstats for Yocto<br />
[23:56] <RP__> She is also our release engineer so coming up to a release is a bad time to involve her in the discussions<br />
[23:57] <ka6sox> RP__ we need to know what is important both to keep and to display.<br />
[23:57] <Jay7> when release is planned?<br />
[23:57] <RP__> Jay7: Start of April<br />
[23:57] <ka6sox> the keeping isn't so bad its the display and collection.<br />
[23:57] <RP__> We hit hard freeze on friday<br />
[23:58] <RP__> ka6sox: I'm working on the principle that we can start to collect things whilst we work on the other parts<br />
[23:58] <RP__> You can see my couple of hours efforts at display :)<br />
[23:58] <ka6sox> Jay7, lets look @ what buildstats is today and see if OE can use that as a starting place.<br />
[23:59] <ka6sox> RP__ yes, I can see there is good potential there.<br />
[23:59] <Jay7> ok.. then let's wait for OE and Yocto releases then have continue this<br />
[23:59] <ka6sox> Jay7, right, but lets familiarize ourselves with it so that we are ready to talk.<br />
[23:59] <ka6sox> and ready to take action when its time.<br />
[00:00] <RP__> ka6sox: The idea is we're planning to expand its scope over time<br />
[00:00] <RP__> ka6sox: and most likely write something to convert the flat files into something xml like<br />
[00:00] <RP__> (for storage/transfer purposes)<br />
[00:00] <ka6sox> RP__, I like the idea of xml...<br />
[00:00] <Jay7> and I dislike idea of xml :)<br />
[00:00] <Jay7> but now it does not matters :)<br />
[00:01] <ka6sox> most important to me are the definitions of what data is important and not.<br />

Latest revision as of 18:25, 14 January 2022

Obsolete - Can be deleted