TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with "= Leveraging Pkg Feeds or How I got Tired of Burning Images = I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to acc...")
 
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Leveraging Pkg Feeds or How I got Tired of Burning Images =
= Leveraging Pkg Feeds or How I got Tired of Burning Images =
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with.  This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes.  I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
<strong> Note: This works as well with kernel modules as it does with libraries/binaries</strong>
<br>
<br>
In this example, we will be building znc, and irc bouncer.  This is just a good complex example.  In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way]
In this example, we will be building znc 1.6.5, an irc bouncer.  This is just a good complex example.  In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]
== Steps ==
* It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
  EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management  dev-pkgs tools-sdk tools-debug tools-profile "
* Now I'll grab the src code we will try to build <strong>Note: this is an ON TARGET build</strong> the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure
This worked, but we decide we want to have cyrus support as well.
* First we try
./configure --enable-cyrus
but we get
configure: error: could not find libsasl2. Try --disable-cyrus.
 
* So we need the cyrus-sals-dev pkg.  If we aren't sure where that is, we can use [https://layers.openembedded.org/layerindex/branch/master/layers/ openembedded layer index] and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
  bitbake cyrus-sasl
and remake the package index
bitbake package-index
* target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)
If this fails, make sure you followed the instructions in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]. You should have set up the on target repo in  /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.
* now we rerun configure and get:
./configure --enable-cyrus
 
  ZNC 1.6.5 configured
prefix:      /usr/local
debug:        no
ipv6:        yes
openssl:      yes
dns:          threads
perl:        no
python:      no
swig:        not needed
cyrus:        yes
tcl:          no
charset:      yes
zlib:        yes
run from src: no
 
Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.
* Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.
Whoops, we need git submodule...  This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule
$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py:            fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc:    ${libexecdir}/git-core/git-submodule \
If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:
PACKAGES =+ "${PN}-perltools".
So we can pull in the git-perltools as well on the target:
# dnf install git-perltools
and rerun the git submodule init
# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'
 
It worked :), and now we can successfully type make.  Having to reflash the image for each of these steps would have been much slower!
# make
Building module autocycle...
Linking  module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...
 
ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.
 
* I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index
Target ->
# dnf install zile
 
== Ways to Serve up the rpm repo ==
 
* very simple - use python twistd.
** let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo. 
** <code >
    cd /home/me/mrrepo;
    twistd -n web --path . -p 5678
  </code>
** similarly, if you are just serving up the build directories rpm's you can just do
*** <code >
    cd /home/me/mrrepo;
    twistd -n web --path tmp/deploy/rpm -p 5678
  </code>
***  Note , only 1 process can own a given port.
* less simple, but better response and better stability- use lighthttpd. 
** I advocate using this from a container as it is simpler to maintain.
** <code>
  docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd
</code>
** What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
** Where did they come from? Copied from inside the container
*** <code> docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd </code>
*** <code> # cp -a /etc/lighttpd/* /frog </code>
*** Now your /home/me/lightppd has the necessary config files.
** The files in /home/me/lighthttpd:
*** lighttpd.conf  lighttpd.conf~  mime-types.conf  mod_cgi.conf  mod_fastcgi.conf  mod_fastcgi_fpm.conf
**** From this you can see I run emacs.  The only change I did was to turn on directory listing .
**** diff  lighttpd.conf  lighttpd.conf~
*****  <code>
114c114
< dir-listing.activate      = "enable"
---
> #  dir-listing.activate      = "enable"
</code>
 
** the  -p 5678:80  exposes the default port the container runs it's service (aka 80)  on to the host port 5678.

Latest revision as of 18:24, 20 September 2017

Leveraging Pkg Feeds or How I got Tired of Burning Images

I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process. Note: This works as well with kernel modules as it does with libraries/binaries
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in PkgFeeds the EasiestWay

Steps

  • It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
 EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management  dev-pkgs tools-sdk tools-debug tools-profile "
  • Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure

This worked, but we decide we want to have cyrus support as well.

  • First we try
./configure --enable-cyrus

but we get

configure: error: could not find libsasl2. Try --disable-cyrus.
  • So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use openembedded layer index and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
 bitbake cyrus-sasl

and remake the package index

bitbake package-index
  • target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)

If this fails, make sure you followed the instructions in PkgFeeds the EasiestWay. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.

  • now we rerun configure and get:
./configure --enable-cyrus
 ZNC 1.6.5 configured
prefix:       /usr/local
debug:        no
ipv6:         yes
openssl:      yes
dns:          threads
perl:         no
python:       no
swig:         not needed
cyrus:        yes
tcl:          no
charset:      yes
zlib:         yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.

  • Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.

Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule

$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py:            fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc:    ${libexecdir}/git-core/git-submodule \

If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:

PACKAGES =+ "${PN}-perltools".

So we can pull in the git-perltools as well on the target:

# dnf install git-perltools

and rerun the git submodule init

# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!

# make
Building module autocycle...
Linking  module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...
ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.
  • I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index

Target ->

# dnf install zile

Ways to Serve up the rpm repo

  • very simple - use python twistd.
    • let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.
    cd /home/me/mrrepo;
    twistd -n web --path . -p 5678
 
    • similarly, if you are just serving up the build directories rpm's you can just do
    cd /home/me/mrrepo;
    twistd -n web --path tmp/deploy/rpm -p 5678
 
      • Note , only 1 process can own a given port.
  • less simple, but better response and better stability- use lighthttpd.
    • I advocate using this from a container as it is simpler to maintain.
 docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd 

    • What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
    • Where did they come from? Copied from inside the container
      • docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd
      • # cp -a /etc/lighttpd/* /frog
      • Now your /home/me/lightppd has the necessary config files.
    • The files in /home/me/lighthttpd:
      • lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
        • From this you can see I run emacs. The only change I did was to turn on directory listing .
        • diff lighttpd.conf lighttpd.conf~
114c114
< dir-listing.activate      = "enable"
---
> #   dir-listing.activate      = "enable"

    • the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.