TipsAndTricks/InvestigatingBuildTime: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
mNo edit summary
Line 1: Line 1:
== Investigating Build Time ==
== Draft: Investigating Build Time ==
 
(Placeholder. Demonstrate use of buildstats and pybootchartgui)
 
'''Current status''': The intention is to use the scripts under scripts/contrib/bb-perf to parse
buildstat data and produce useful plots.


(Placeholder.  Demonstrate use of buildstats and pybootchartgui)


The <tt>buildstats</tt> data produce by the class with the same name if included in <tt>USER_CLASSES</tt> variable  
The <tt>buildstats</tt> data produce by the class with the same name if included in <tt>USER_CLASSES</tt> variable  
Line 14: Line 18:
   $ . oe-init-build-env
   $ . oe-init-build-env
   $ bitbake world
   $ bitbake world
=== Build Machine ===
Build time depends on the build machine HW and for this particular article we use a Linux box with the following characteristics:


=== Buildstats ===
=== Buildstats ===


There are two set of process' stat data: time and IO stats. For the moment, the time stats to be used are the following:
buildstats.bbclass stores mainly two set of process' stats: time and IO stats. For the moment, we are currently focusing on the following
 
time data:


* /proc/[pid]/stat: utime, stime
* /proc/[pid]/stat: utime, stime
Line 30: Line 29:
* python resource module: Child rusage ru_utime, Child rusage ru_stime
* python resource module: Child rusage ru_utime, Child rusage ru_stime


and the following IO stats:
and the IO stats:


* /proc/[pid]/io: IO rchar, IO wchar
* /proc/[pid]/io: IO rchar, IO wchar
* /proc/[pid]/io: IO read_bytes, IO write_bytes
* /proc/[pid]/io: IO read_bytes, IO write_bytes


With this separation of buildstats, proceed running the following script (once the <code>bitbake world</code> command has finished)
With this data, the following script (once the <code>bitbake world</code> command has finished) is used
to plot the top N stat consumers
to either plot the top N recipes-task stat values


  $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -N $N | gnuplot -p
  $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -N $N | gnuplot -p


In case we want a more general picture, getting data per task type instead of recipe's task, the script needs <code>-S</code> (for Sum)
or sum all stats for a particular task for all (world) recipes
argument


  $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -S | gnuplot -p
  $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -S | gnuplot -p


The former produces a more granular plot and second a more general idea where the type is spent per task set.


=== Results ===
=== Setup ===


  $ IMAGE='set terminal png fontscale 1 size 1280,720; set output "IO-write_bytes:IO-read_bytes.png"'
  GPLIMAGE='set terminal png fontscale 1 size 1280,720'
function bsplots() {
  local STAT=$1
  OUT="set output '$STAT.task-recipe.png'"
  ../scripts/contrib/bb-perf/buildstats-plot.sh -s "$STAT" -n 20 > input.gpl
  gnuplot -e "$GPLIMAGE;$OUT" > input.gpl
  OUT="set output '$STAT.task.png'"
  ../scripts/contrib/bb-perf/buildstats-plot.sh -s "$STAT" -S > input.gpl
  gnuplot -e "$GPLIMAGE;$OUT" > input.gpl
}


==== utime and stime ====


Time spent by a certain process can be obtain looking at the <tt>stime<</tt> and <tt>time</tt> corresponding to the amount of time the process
=== Results ===
has been scheduled in kernel and user mode.


$ IMAGE='set terminal png fontscale 1 size 1280,720; set output "IO-write_bytes:IO-read_bytes.png"'
  bsplots "stime:utime"
$ ../scripts/contrib/bb-perf/buildstats-plot.sh -s 'stime:utime' -N 20 > input.gpl; gnuplot -e "\'$IMAGE\'" input.gpl
  bsplots "IO rchar:IO wchar"
$ ../scripts/contrib/bb-perf/buildstats-plot.sh -s 'stime:utime' -S | gnuplot -p

Revision as of 00:07, 15 November 2016

Draft: Investigating Build Time

(Placeholder. Demonstrate use of buildstats and pybootchartgui)

Current status: The intention is to use the scripts under scripts/contrib/bb-perf to parse buildstat data and produce useful plots.


The buildstats data produce by the class with the same name if included in USER_CLASSES variable provides process statistics for every task. This data can offer insight of the time spent by the builder (bitbake) when constructing a particular target at process level. This article focus more on profiling than optimization, the former being the base for the latter.

Setup

 $ cd poky
 $ git checkout -b morty origin/morty
 $ . oe-init-build-env
 $ bitbake world

Buildstats

buildstats.bbclass stores mainly two set of process' stats: time and IO stats. For the moment, we are currently focusing on the following time data:

  • /proc/[pid]/stat: utime, stime
  • /proc/[pid]/stat: cutime, cstime
  • python resource module: rusage ru_utime, rusage ru_stime
  • python resource module: Child rusage ru_utime, Child rusage ru_stime

and the IO stats:

  • /proc/[pid]/io: IO rchar, IO wchar
  • /proc/[pid]/io: IO read_bytes, IO write_bytes

With this data, the following script (once the bitbake world command has finished) is used to either plot the top N recipes-task stat values 

$ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -N $N | gnuplot -p

or sum all stats for a particular task for all (world) recipes 

$ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -S | gnuplot -p

The former produces a more granular plot and second a more general idea where the type is spent per task set.

Setup

GPLIMAGE='set terminal png fontscale 1 size 1280,720'
function bsplots() {
  local STAT=$1
  OUT="set output '$STAT.task-recipe.png'"
  ../scripts/contrib/bb-perf/buildstats-plot.sh -s "$STAT" -n 20 > input.gpl
  gnuplot -e "$GPLIMAGE;$OUT" > input.gpl
  OUT="set output '$STAT.task.png'"
  ../scripts/contrib/bb-perf/buildstats-plot.sh -s "$STAT" -S > input.gpl
  gnuplot -e "$GPLIMAGE;$OUT" > input.gpl
}


Results

 bsplots "stime:utime"
 bsplots "IO rchar:IO wchar"
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=TipsAndTricks/InvestigatingBuildTime&oldid=21210"