TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

Leveraging Pkg Feeds or How I got Tired of Burning Images

I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process. Note: This works as well with kernel modules as it does with libraries/binaries
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in PkgFeeds the EasiestWay

Steps

• It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:

 EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management  dev-pkgs tools-sdk tools-debug tools-profile "

• Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...

git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure

This worked, but we decide we want to have cyrus support as well.

• First we try

./configure --enable-cyrus

but we get

configure: error: could not find libsasl2. Try --disable-cyrus.

• So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use openembedded layer index and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.

 bitbake cyrus-sasl

and remake the package index

bitbake package-index

• target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.

$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)

If this fails, make sure you followed the instructions in PkgFeeds the EasiestWay. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.

• now we rerun configure and get:

./configure --enable-cyrus

 ZNC 1.6.5 configured
prefix:       /usr/local
debug:        no
ipv6:         yes
openssl:      yes
dns:          threads
perl:         no
python:       no
swig:         not needed
cyrus:        yes
tcl:          no
charset:      yes
zlib:         yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.

• Let's build it

~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.

Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule

$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py:            fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc:    ${libexecdir}/git-core/git-submodule \

If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:

PACKAGES =+ "${PN}-perltools".

So we can pull in the git-perltools as well on the target:

# dnf install git-perltools

and rerun the git submodule init

# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!

# make
Building module autocycle...
Linking  module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...

ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.

• I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:

$ bitbake zile
$ bitbake package-index

Target ->

# dnf install zile

Ways to Serve up the rpm repo

• very simple - use python twistd.
  • let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.

    cd /home/me/mrrepo;
    twistd -n web --path . -p 5678
 

• • similarly, if you are just serving up the build directories rpm's you can just do

    cd /home/me/mrrepo;
    twistd -n web --path tmp/deploy/rpm -p 5678
 

• • • Note , only 1 process can own a given port.
• less simple, but better response and better stability- use lighthttpd.
  • I advocate using this from a container as it is simpler to maintain.

 docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd 

• • What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
  • Where did they come from? Copied from inside the container
    • docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd
    • # cp -a /etc/lighttpd/* /frog
    • Now your /home/me/lightppd has the necessary config files.
  • The files in /home/me/lighthttpd:
    • lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
      • From this you can see I run emacs. The only change I did was to turn on directory listing .
      • diff lighttpd.conf lighttpd.conf~

114c114
< dir-listing.activate      = "enable"
---
> #   dir-listing.activate      = "enable"

• • the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.