Yocto Project - User contributions [en]

TipsAndTricks/KernelDevelopmentWithEsdk

2017-08-09T21:41:01Z

Todor Minchev: /* Create image with new kernel */

== Overview ==
This article describes an efficient workflow for Linux kernel development using Yocto Project. The goal is to get the build, deploy and test cycle to under a minute. We will be working with the generix86-64 BSP and the Intel Minnowboard Turbot in this article but the process is the same for any platform and CPU architecture.

The extensible SDK (eSDK) is a key part of this workflow for a number of reasons
# It contains the target toolchain which is not easily accessible in the bitbake environment
# It contains '''devtool''' which gives you an easy way of getting the kernel source your target is using (if using linux-yocto or linux-intel)
# devtool will also automatically create patches for your kernel updates and add them to the the kernel recipe

There are two separate steps to this process
# Build or download an eSDK for your platform
# Install and use that eSDK to build your kernel

== Build Extensible SDK ==
Open a new terminal window, we'll refer to this terminal as your '''bitbake terminal'''. First we'll checkout poky and configure bitbake environment.
<pre>
$ git clone -b pyro git://git.yoctoproject.org/poky
$ source poky/oe-init-build-env
</pre>
Now we need to do some configuration for the Minnowboard image.
* Set MACHINE (as default is qemux86)
* Enable kernel modules (disabled by default for minimal images)
Add the following to conf/local.conf
MACHINE = "genericx86-64"
MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "kernel-modules"
We also need to create a layer to take the kernel patches we'll be creating. Be sure to create example recipe and append files when you run you run the "yocto-layer" command.
<pre>
$ yocto-layer create my-kernel -o ../meta-my-kernel
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n] y
Please enter the name you'd like to use for your example recipe: [default: example]
Would you like to have an example bbappend file created? (y/n) [default: n] y
Please enter the name you'd like to use for your bbappend file: [default: example]
Please enter the version number you'd like to use for your bbappend file (this should match the recipe you're appending to): [default: 0.1]

New layer created in ../meta-my-kernel.

Don't forget to add it to your BBLAYERS (for details see ../meta-my-kernel/README).
</pre>
As directed, we need to add our layer to BBLAYERS as follows:
<pre>
$ bitbake-layers add-layer ../meta-my-kernel
</pre>
Now build an extensible SDK installer for the Minnowboard in the bitbake terminal
<pre>
$ bitbake core-image-minimal -c populate_sdk_ext
</pre>
The eSDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh

Alternatively, if you are not using a yocto-linux kernel, you can use a prebuilt eSDK installer that includes the toolchain you need. For example, the pyro core-image-minimal esdk release for core2-64 is located [[http://downloads.yoctoproject.org/releases/yocto/yocto-2.3.1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh here]]. If you downloaded the installer, make it executable.

== Install Extensible SDK ==
Run the installer as follows. If you do not want to use the default ~/poky_sdk directory for installation of the toolchain, use the -d option to specify a destination.
<pre>
$ ./poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh
Poky (Yocto Project Reference Distro) Extensible SDK installer version 2.3.1
============================================================================
Enter target directory for SDK (default: ~/poky_sdk):
You are about to install the SDK to "/home/scottrif/poky_sdk". Proceed[Y/n]? Y
Extracting SDK......................................done
Setting it up...
Extracting buildtools...
Preparing build system...
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:52
Initialising tasks: 100% |######################################################################################################| Time: 0:00:04
Checking sstate mirror object availability: 100% |##############################################################################| Time: 0:00:00
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:33
Initialising tasks: 100% |######################################################################################################| Time: 0:00:00
done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /home/scottrif/poky_sdk/environment-setup-core2-64-poky-linux
</pre>
Take note of the final line, which shows you the environment setup script you must run each time you want to use the eSDK.

== Setup your ESDK build environment (ESDK terminal) ==
'''YOU MUST OPEN A NEW TERMINAL WHEN USING THE ESDK. DO NOT RE-USE YOUR BITBAKE TERMINAL.'''
Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'. Run the setup script given by the installer.
$ source /path/to/esdk/environment-setup-core2-64-poky-linux
"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.

If you see this
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a
new shell session instead."
You didn't open a new terminal!

== Build initial image with ESDK ==
In the ESDK terminal, build the image
<pre>
$ devtool build-image
Parsing recipes: 100% |##########################################| Time: 0:00:05
Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 47 skipped, 0 masked, 0 errors.
WARNING: No packages to add, building image core-image-minimal unmodified
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 1299 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |#######################################| Time: 0:00:07
Checking sstate mirror object availability: 100% |###############| Time: 0:00:00
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2866 tasks of which 2604 didn't need to be rerun and all succeeded.
NOTE: Successfully built core-image-minimal. You can find output files in /path/to/esdk/tmp/deploy/images/genericx86-64
</pre>
Now flash to image to USB stick, assuming USB stick is on /dev/sdd
$ sudo dd if=tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot the Minnowboard with this image and check to be sure that everything is OK.

== Working with Yocto kernel ==
=== Checkout kernel source ===
First we must use devtool to checkout the kernel source code in its workspace. Do the following in the ESDK terminal. Note that we use the virtual kernel provider so we don't need to know the kernel recipe name.

<pre>
$ devtool modify virtual/kernel
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Mapping virtual/kernel to linux-yocto
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_checkout...
NOTE: Tasks Summary: Attempted 3 tasks of which 2 didn't need to be rerun and all succeeded.
NOTE: Patching...
NOTE: Executing RunQueue Tasks
NOTE: Executing do_validate_branches...
NOTE: Executing do_kernel_metadata...
NOTE: Executing do_patch...
NOTE: Tasks Summary: Attempted 6 tasks of which 3 didn't need to be rerun and all succeeded.
NOTE: Generating kernel config
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_configme...
NOTE: Executing do_prepare_recipe_sysroot...
NOTE: Executing do_configure...
NOTE: Tasks Summary: Attempted 9 tasks of which 6 didn't need to be rerun and all succeeded.
NOTE: Copying kernel config to srctree
NOTE: Source tree extracted to /home/scottrif/poky_sdk/workspace/sources/linux-yocto
NOTE: Recipe linux-yocto now set up to build from /home/scottrif/poky_sdk/workspace/sources/linux-yocto
</pre>

There is a bug that may cause error messages such as the following to be displayed. These can be ignored, the source code will be correct checked out.
ERROR: Taskhash mismatch 2c793438c2d9f8c3681fd5f7bc819efa versus be3a89ce7c47178880ba7bf6293d7404 for /path/to/esdk/layers/poky/meta/recipes-kernel/linux/linux-yocto_4.10.bb.do_unpack

=== Edit and build driver ===
Kernel source will be in the location shown at end of the <tt>devtool modify virtual/kernel</tt> output. Let's make a trivial change to the serial driver in drivers/tty/serial/8250/8250_core.c. After the line
pr_info("Serial: 8250/16550 driver, %d ports, IRQ sharing %sabled\n",
nr_uarts, share_irqs ? "en" : "dis");
Add the line
pr_info("Serial: 8250/16550 driver modified with eSDK");
To build the updated kernel source, in the SDK terminal run make in your kernel source folder. Adjust the -j option to match number of cores on your workstation.
$ make -C workspace/sources/linux-yocto -j4
=== Create image with new kernel ===
Normally you'd make a new image with <tt>devtool build-image</tt> but this can take a minute or so and messes with the kernel source folder. A faster option is to use kpartx to splice the new kernel into the image you have already built. If your build system does not have kpartx installed, be sure you install it.

First make a copy of your wic file in say /tmp
$ cp tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic /tmp
Then create loopback devices for each partition in wic file.
NOTE: The first unused loopback device (loopX) will be automatically allocated e.g loop3.
$ sudo kpartx -v -a /tmp/core-image-minimal-genericx86-64.wic
add map loopXp1 (253:6): 0 47446 linear /dev/loopX 2048
add map loopXp2 (253:7): 0 119356 linear /dev/loopX 51200
add map loopXp3 (253:8): 0 90112 linear /dev/loopX 170556
Kernel is in the first device, so let's mount /dev/mapper/loopXp1 to take a look
NOTE: Replace loopXp1 with the automatically allocated loopback device e.g loop3p1.
$ sudo mkdir /mnt/wic-p1
$ sudo mount /dev/mapper/loopXp1 /mnt/wic-p1
Now copy over new kernel
$ sudo cp workspace/sources/linux-yocto/arch/x86/boot/bzImage /mnt/wic-p1
Then unmount and unkpartx...
NOTE: Replace loopX with the automatically allocated loopback device e.g loop3.
$ sudo umount /mnt/wic-p1
$ sudo kpartx -d /dev/loopX
Now flash image
$ dd if=/tmp/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot Minnowboard, log on and look in log for driver message
# root@genericx86-64:~# dmesg | grep Serial:
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Serial: 8250/16550 driver modified with eSDK
It worked!

=== Export patches and create a bbappend file ===
To export your commits as patches and a bbappend file use the following command in the ESDK terminal. We'll target the "my-kernel" layer we created in the bitbake terminal
$ devtool finish linux-yocto /path/to/meta-my-kernel
The patches and the bbappend can be found in /path/to/meta-my-kernel/recipes-kernel/linux.

== Working with upstream kernel ==
Slightly different workflow where we use a kernel from kernel.org.

Content TBD

== Build image with your modified kernel ==
You can now build an image which will include your kernel patches.
Execute the following command from your build directory in the bitbake terminal
bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-08-09T21:39:07Z

Todor Minchev: /* Create image with new kernel */

== Overview ==
This article describes an efficient workflow for Linux kernel development using Yocto Project. The goal is to get the build, deploy and test cycle to under a minute. We will be working with the generix86-64 BSP and the Intel Minnowboard Turbot in this article but the process is the same for any platform and CPU architecture.

The extensible SDK (eSDK) is a key part of this workflow for a number of reasons
# It contains the target toolchain which is not easily accessible in the bitbake environment
# It contains '''devtool''' which gives you an easy way of getting the kernel source your target is using (if using linux-yocto or linux-intel)
# devtool will also automatically create patches for your kernel updates and add them to the the kernel recipe

There are two separate steps to this process
# Build or download an eSDK for your platform
# Install and use that eSDK to build your kernel

== Build Extensible SDK ==
Open a new terminal window, we'll refer to this terminal as your '''bitbake terminal'''. First we'll checkout poky and configure bitbake environment.
<pre>
$ git clone -b pyro git://git.yoctoproject.org/poky
$ source poky/oe-init-build-env
</pre>
Now we need to do some configuration for the Minnowboard image.
* Set MACHINE (as default is qemux86)
* Enable kernel modules (disabled by default for minimal images)
Add the following to conf/local.conf
MACHINE = "genericx86-64"
MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "kernel-modules"
We also need to create a layer to take the kernel patches we'll be creating. Be sure to create example recipe and append files when you run you run the "yocto-layer" command.
<pre>
$ yocto-layer create my-kernel -o ../meta-my-kernel
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n] y
Please enter the name you'd like to use for your example recipe: [default: example]
Would you like to have an example bbappend file created? (y/n) [default: n] y
Please enter the name you'd like to use for your bbappend file: [default: example]
Please enter the version number you'd like to use for your bbappend file (this should match the recipe you're appending to): [default: 0.1]

New layer created in ../meta-my-kernel.

Don't forget to add it to your BBLAYERS (for details see ../meta-my-kernel/README).
</pre>
As directed, we need to add our layer to BBLAYERS as follows:
<pre>
$ bitbake-layers add-layer ../meta-my-kernel
</pre>
Now build an extensible SDK installer for the Minnowboard in the bitbake terminal
<pre>
$ bitbake core-image-minimal -c populate_sdk_ext
</pre>
The eSDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh

Alternatively, if you are not using a yocto-linux kernel, you can use a prebuilt eSDK installer that includes the toolchain you need. For example, the pyro core-image-minimal esdk release for core2-64 is located [[http://downloads.yoctoproject.org/releases/yocto/yocto-2.3.1/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh here]]. If you downloaded the installer, make it executable.

== Install Extensible SDK ==
Run the installer as follows. If you do not want to use the default ~/poky_sdk directory for installation of the toolchain, use the -d option to specify a destination.
<pre>
$ ./poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh
Poky (Yocto Project Reference Distro) Extensible SDK installer version 2.3.1
============================================================================
Enter target directory for SDK (default: ~/poky_sdk):
You are about to install the SDK to "/home/scottrif/poky_sdk". Proceed[Y/n]? Y
Extracting SDK......................................done
Setting it up...
Extracting buildtools...
Preparing build system...
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:52
Initialising tasks: 100% |######################################################################################################| Time: 0:00:04
Checking sstate mirror object availability: 100% |##############################################################################| Time: 0:00:00
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:33
Initialising tasks: 100% |######################################################################################################| Time: 0:00:00
done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /home/scottrif/poky_sdk/environment-setup-core2-64-poky-linux
</pre>
Take note of the final line, which shows you the environment setup script you must run each time you want to use the eSDK.

== Setup your ESDK build environment (ESDK terminal) ==
'''YOU MUST OPEN A NEW TERMINAL WHEN USING THE ESDK. DO NOT RE-USE YOUR BITBAKE TERMINAL.'''
Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'. Run the setup script given by the installer.
$ source /path/to/esdk/environment-setup-core2-64-poky-linux
"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.

If you see this
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a
new shell session instead."
You didn't open a new terminal!

== Build initial image with ESDK ==
In the ESDK terminal, build the image
<pre>
$ devtool build-image
Parsing recipes: 100% |##########################################| Time: 0:00:05
Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 47 skipped, 0 masked, 0 errors.
WARNING: No packages to add, building image core-image-minimal unmodified
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 1299 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |#######################################| Time: 0:00:07
Checking sstate mirror object availability: 100% |###############| Time: 0:00:00
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2866 tasks of which 2604 didn't need to be rerun and all succeeded.
NOTE: Successfully built core-image-minimal. You can find output files in /path/to/esdk/tmp/deploy/images/genericx86-64
</pre>
Now flash to image to USB stick, assuming USB stick is on /dev/sdd
$ sudo dd if=tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot the Minnowboard with this image and check to be sure that everything is OK.

== Working with Yocto kernel ==
=== Checkout kernel source ===
First we must use devtool to checkout the kernel source code in its workspace. Do the following in the ESDK terminal. Note that we use the virtual kernel provider so we don't need to know the kernel recipe name.

<pre>
$ devtool modify virtual/kernel
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Mapping virtual/kernel to linux-yocto
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_checkout...
NOTE: Tasks Summary: Attempted 3 tasks of which 2 didn't need to be rerun and all succeeded.
NOTE: Patching...
NOTE: Executing RunQueue Tasks
NOTE: Executing do_validate_branches...
NOTE: Executing do_kernel_metadata...
NOTE: Executing do_patch...
NOTE: Tasks Summary: Attempted 6 tasks of which 3 didn't need to be rerun and all succeeded.
NOTE: Generating kernel config
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_configme...
NOTE: Executing do_prepare_recipe_sysroot...
NOTE: Executing do_configure...
NOTE: Tasks Summary: Attempted 9 tasks of which 6 didn't need to be rerun and all succeeded.
NOTE: Copying kernel config to srctree
NOTE: Source tree extracted to /home/scottrif/poky_sdk/workspace/sources/linux-yocto
NOTE: Recipe linux-yocto now set up to build from /home/scottrif/poky_sdk/workspace/sources/linux-yocto
</pre>

There is a bug that may cause error messages such as the following to be displayed. These can be ignored, the source code will be correct checked out.
ERROR: Taskhash mismatch 2c793438c2d9f8c3681fd5f7bc819efa versus be3a89ce7c47178880ba7bf6293d7404 for /path/to/esdk/layers/poky/meta/recipes-kernel/linux/linux-yocto_4.10.bb.do_unpack

=== Edit and build driver ===
Kernel source will be in the location shown at end of the <tt>devtool modify virtual/kernel</tt> output. Let's make a trivial change to the serial driver in drivers/tty/serial/8250/8250_core.c. After the line
pr_info("Serial: 8250/16550 driver, %d ports, IRQ sharing %sabled\n",
nr_uarts, share_irqs ? "en" : "dis");
Add the line
pr_info("Serial: 8250/16550 driver modified with eSDK");
To build the updated kernel source, in the SDK terminal run make in your kernel source folder. Adjust the -j option to match number of cores on your workstation.
$ make -C workspace/sources/linux-yocto -j4
=== Create image with new kernel ===
Normally you'd make a new image with <tt>devtool build-image</tt> but this can take a minute or so and messes with the kernel source folder. A faster option is to use kpartx to splice the new kernel into the image you have already built. First make a copy of your wic file in say /tmp
$ cp tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic /tmp
Then create loopback devices for each partition in wic file.
NOTE: The first unused loopback device (loopX) will be automatically allocated e.g loop3.
$ sudo kpartx -v -a /tmp/core-image-minimal-genericx86-64.wic
add map loopXp1 (253:6): 0 47446 linear /dev/loopX 2048
add map loopXp2 (253:7): 0 119356 linear /dev/loopX 51200
add map loopXp3 (253:8): 0 90112 linear /dev/loopX 170556
Kernel is in the first device, so let's mount /dev/mapper/loopXp1 to take a look
$ sudo mkdir /mnt/wic-p1
$ sudo mount /dev/mapper/loopXp1 /mnt/wic-p1
Now copy over new kernel
$ sudo cp workspace/sources/linux-yocto/arch/x86/boot/bzImage /mnt/wic-p1
Then unmount and unkpartx...
$ sudo umount /mnt/wic-p1
$ sudo kpartx -d /dev/loopX
Now flash image
$ dd if=/tmp/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot Minnowboard, log on and look in log for driver message
# root@genericx86-64:~# dmesg | grep Serial:
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Serial: 8250/16550 driver modified with eSDK
It worked!

=== Export patches and create a bbappend file ===
To export your commits as patches and a bbappend file use the following command in the ESDK terminal. We'll target the "my-kernel" layer we created in the bitbake terminal
$ devtool finish linux-yocto /path/to/meta-my-kernel
The patches and the bbappend can be found in /path/to/meta-my-kernel/recipes-kernel/linux.

== Working with upstream kernel ==
Slightly different workflow where we use a kernel from kernel.org.

Content TBD

== Build image with your modified kernel ==
You can now build an image which will include your kernel patches.
Execute the following command from your build directory in the bitbake terminal
bitbake core-image-minimal

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-26T21:12:51Z

Todor Minchev: /* Configuring It */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--prefix=/usr' '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-26T21:11:34Z

Todor Minchev: /* Building It */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/local/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/local/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--prefix=/usr' '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-26T21:08:53Z

Todor Minchev: /* Building It */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/local/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/local/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-25T23:38:58Z

Todor Minchev: /* Configuring It */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/local/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/local/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-25T23:38:30Z

Todor Minchev: /* Configuring It */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/local/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-26T21:48:08Z

Todor Minchev: /* Overview */

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your bitbake build environment ==

Open a new terminal window and navigate to your poky directory. We'll refer to this terminal as your 'bitbake terminal'.

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Build Extensible SDK ==

bitbake -c populate_sdk_ext core-image-minimal
Alternatively, if you are based on a poky derivative for a particular release, you can download the minimal esdk and install it. For example, the pyro core-image-minimal esdk release for x86-64 is located [[http://downloads.yoctoproject.org/releases/yocto/yocto-2.3/toolchain/x86_64/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.sh here]].

== Install Extensible SDK ==

The ESDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/

./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-corei7-64-toolchain-ext-2.3.sh

Follow the installer instructions and provide a target directory for your ESDK installation e.g /path_to_esdk/

== Checkout the kernel source tree and construct .config ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

If you are just using a kernel from somewhere else, for example kernel.org, you can just clone it instead of using devtool modify in this step.

== Setup your ESDK build environment (ESDK terminal) ==

Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'

. /path_to_esdk/environment-setup-corei7-64-poky-linux

NOTE: If you see the message below you will really need to open a new terminal :).

"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead."

== Build your kernel (ESDK terminal) ==

Navigate to your workspace directory e.g /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto
and build the kernel

make

== Modify your kernel and commit your changes (ESDK terminal)==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel. Commit your kernel changes.

== Build an image with your kernel changes (optional) (bitbake terminal) ==

devtool build-image core-image-minimal

For faster iteration, in 2.4 some wic commands are being included to allow you to insert the kernel in an image without going through the full recreation of the image. For older releases you can do:
sudo kpartx -a <foo>.wic
sudo mount /dev/mapper/loopNpM /mnt/wic-partionM
In this case, it will be the next available loopback device. If you are not using any , it will be loop0. so in the std case it would look like
sudo mount /dev/mapper/loop0p1 /mnt/wic-partion1
Then you could dd the kernel into the kernel partion. Then unmount and unkaprtx...
sudo umount mnt/wic-partionM
sudo kaprtx -d /dev/loopN

Note that this requires root/sudo rights. The assumption here is most kernel devs would have this or could do it in a vm/container.

== Export patches and create a bbappend file (bitbake terminal) ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image with your modified kernel (bitbake terminal) ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-22T23:41:33Z

Todor Minchev: /* Setup your ESDK build environment (ESDK terminal) */

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your bitbake build environment ==

Open a new terminal window and navigat to your poky directory. We'll refer to this terminal as your 'bitbake terminal'.

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Build Extensible SDK (bitbake terminal) ==

bitbake -c populate_sdk_ext core-image-minimal

== Install Extensible SDK (bitbake terminal) ==

The ESDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/

./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-corei7-64-toolchain-ext-2.3.sh

Follow the installer instructions and provide a target directory for your ESDK installation e.g /path_to_esdk/

== Checkout the kernel source tree and construct .config (bitbake terminal) ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

== Setup your ESDK build environment (ESDK terminal) ==

Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'

. /path_to_esdk/environment-setup-corei7-64-poky-linux

NOTE: If you see the message below you will really need to open a new terminal :).

"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead."

== Build your kernel (ESDK terminal) ==

Navigate to your workspace directory e.g /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto
and build the kernel

make

== Modify your kernel and commit your changes (ESDK terminal)==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel. Commit your kernel changes.

== Build an image with your kernel changes (optional) (bitbake terminal) ==

devtool build-image core-image-minimal

== Export patches and create a bbappend file (bitbake terminal) ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image with your modified kernel (bitbake terminal) ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-22T23:41:16Z

Todor Minchev: /* Setup your ESDK build environment (ESDK terminal) */

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your bitbake build environment ==

Open a new terminal window and navigat to your poky directory. We'll refer to this terminal as your 'bitbake terminal'.

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Build Extensible SDK (bitbake terminal) ==

bitbake -c populate_sdk_ext core-image-minimal

== Install Extensible SDK (bitbake terminal) ==

The ESDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/

./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-corei7-64-toolchain-ext-2.3.sh

Follow the installer instructions and provide a target directory for your ESDK installation e.g /path_to_esdk/

== Checkout the kernel source tree and construct .config (bitbake terminal) ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

== Setup your ESDK build environment (ESDK terminal) ==

Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'

. /path_to_esdk/environment-setup-corei7-64-poky-linux

NOTE: If you see this message you will really need to open a new terminal :).

"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead."

== Build your kernel (ESDK terminal) ==

Navigate to your workspace directory e.g /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto
and build the kernel

make

== Modify your kernel and commit your changes (ESDK terminal)==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel. Commit your kernel changes.

== Build an image with your kernel changes (optional) (bitbake terminal) ==

devtool build-image core-image-minimal

== Export patches and create a bbappend file (bitbake terminal) ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image with your modified kernel (bitbake terminal) ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-22T23:40:51Z

Todor Minchev: /* Setup your ESDK build environment (ESDK terminal) */

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your bitbake build environment ==

Open a new terminal window and navigat to your poky directory. We'll refer to this terminal as your 'bitbake terminal'.

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Build Extensible SDK (bitbake terminal) ==

bitbake -c populate_sdk_ext core-image-minimal

== Install Extensible SDK (bitbake terminal) ==

The ESDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/

./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-corei7-64-toolchain-ext-2.3.sh

Follow the installer instructions and provide a target directory for your ESDK installation e.g /path_to_esdk/

== Checkout the kernel source tree and construct .config (bitbake terminal) ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

== Setup your ESDK build environment (ESDK terminal) ==

Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'

. /path_to_esdk/environment-setup-corei7-64-poky-linux

NOTE: If you see this message you will really need to open a new terminal :).

SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.

== Build your kernel (ESDK terminal) ==

Navigate to your workspace directory e.g /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto
and build the kernel

make

== Modify your kernel and commit your changes (ESDK terminal)==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel. Commit your kernel changes.

== Build an image with your kernel changes (optional) (bitbake terminal) ==

devtool build-image core-image-minimal

== Export patches and create a bbappend file (bitbake terminal) ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image with your modified kernel (bitbake terminal) ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-22T23:39:49Z

Todor Minchev: /* Overview */

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your bitbake build environment ==

Open a new terminal window and navigat to your poky directory. We'll refer to this terminal as your 'bitbake terminal'.

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Build Extensible SDK (bitbake terminal) ==

bitbake -c populate_sdk_ext core-image-minimal

== Install Extensible SDK (bitbake terminal) ==

The ESDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/

./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-corei7-64-toolchain-ext-2.3.sh

Follow the installer instructions and provide a target directory for your ESDK installation e.g /path_to_esdk/

== Checkout the kernel source tree and construct .config (bitbake terminal) ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

== Setup your ESDK build environment (ESDK terminal) ==

Open a new terminal and setup your build environment. Let's refer to this terminal as your 'ESDK terminal'

. /path_to_esdk/environment-setup-corei7-64-poky-linux

NOTE: If you see this message you will really need to open a new terminal :).

SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.

== Build your kernel (ESDK terminal) ==

Navigate to your workspace directory e.g /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto
and build the kernel

make

== Modify your kernel and commit your changes (ESDK terminal)==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel. Commit your kernel changes.

== Build an image with your kernel changes (optional) (bitbake terminal) ==

devtool build-image core-image-minimal

== Export patches and create a bbappend file (bitbake terminal) ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image with your modified kernel (bitbake terminal) ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-21T23:59:15Z

Todor Minchev:

= Overview =

This manual describes the Linux kernel development process with Yocto and devtool.

== Setup your environment ==

The following command will configure the Yocto build environment

source poky/oe-init-build-env kernel-dev

== Checkout the kernel source tree and construct .config ==

Check out the Linux kernel source and construct a .config from the Yocto kernel configuration segments. The sources can be found in /path_to_build_directory/kernel-dev/workspace/sources/linux-yocto.

devtool modify linux-yocto

== Modify your kernel and commit your changes ==

You can navigate to the Linux source directory (/path_to_build_directory/kernel-dev/workspace/sources/linux-yocto) and modify the kernel.

== Build your kernel ==

Build the kernel in your workspace directory

devtool build linux-yocto

== Build an image with your kernel changes ==

devtool build-image core-image-minimal

== Export patches and create a bbappend file ==
To export your commits to patches and a bbappend file use the following command.

devtool finish linux-yocto /path_to_poky_directory/meta-yocto-bsp/

The patches and the bbappend can be found in /path_to_poky_directory/meta-yocto-bsp/recipes-kernel/linux.

== Build image ==

You can now build an image which will include your kernel patches.

Execute the following command from your build directory ( e.g /path_to_build_directory/kernel-dev/)

bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-06-21T23:58:53Z

Todor Minchev: Created page with "= Overview = This manual describes the Linux kernel development process with Yocto and devtool. == Setup your environment == The following command will configure the Yocto ..."

TipsAndTricks

2017-06-21T23:27:31Z

Todor Minchev:

== Background ==
This wiki page captures ideas for "Tips and tricks" articles that are aimed Yocto Project users that have a basic understanding of the core tools and want to extend their knowledge. Articles will written and published using the following process
* Articles must refer to the current (or earlier) release. They must not cover features in development in the master branch.
* Anyone can add entries to the '''Ideas for Articles''' section based on challenges they have encountered when using the Yocto Project
* More experienced developers can start to flesh out articles in the ''Ideas'' section move then to '''Articles in Development''' or contribute directly to this section.
* Approximately once a month an article ready for publishing will be chosen for publication by [mailto:jeffrey.osier-mixon@intel.com Jefro] and moved to the '''Finished Articles ''' section.

Please contact [mailto:henry.bruce@intel.com?Subject=Yocto%20Project%20Tips%20and%20Tricks Henry Bruce] with any questions.

== Ideas for Articles ==
* [[TipsAndTricks/GeneratingASDK]] (Ross, Paul help please...)
* [[TipsAndTricks/AddingALicense]] (Ross)
* [[TipsAndTricks/UsingBuildstatsDiff]] (Ross)
* [[TipsAndTricks/DevPyShell]] (Richard Purdie)
* [[TipsAndTricks/CreatingABSP]]. And what not to put in it.
* [[TipsAndTricks/Patchwork]]
* [[TipsAndTricks/Patchtest]] (Leo)
* [[TipsAndTricks/CreateNewLayer]] (Brendan)
* [[TipsAndTricks/DockerOnImage]] (bavery)

== Articles in development ==
* [[TipsAndTricks/KernelDevelopment]] (Todor Minchev)
* [[TipsAndTricks/DebugNativeRecipeWithGdb]] (Joshua Lock)
* [[TipsAndTricks/Netconsole]] (Ross Burton)
* [[TipsAndTricks/ParsingProfiling]] (Richard Purdie)
* [[TipsAndTricks/DemystifyingTheLinuxYoctoKernel]] (Tom Zanussi)
* [[TipsAndTricks/Patching the source for a recipe]] (Paul Eggleton)
* [[TipsAndTricks/Incorporating closed source components]] (Paul Eggleton)
* [[TipsAndTricks/DebuggingAvoidingRebuilds]] (Richard Purdie)
* [[TipsAndTricks/GitBisectABitbake]] (Ross)
* [[TipsAndTricks/RunningEclipseAgainstBuiltImage]] (bavery)
* [[TipsAndTricks/PrelinkSomePointersAndWorkarounds]] (bavery)
* [[TipsAndTricks/RunningQemuOnMacOSX]] (Stephano)
* [[TipsAndTricks/Understanding what changed (diffsigs etc)]] (Joshua)
* [[TipsAndTricks/CropsCLIContainers]] (bavery)
* [[TipsAndTricks/QuickAndDirtyKernelConfig]] (bavery)
* [[TipsAndTricks/TestingToasterWithContainers]] (bavery)
* [[TipsAndTricks/InvestigatingBuildTime]] (Leo Sandoval)
* [[TipsAndTricks/ResolvingLocaleIssues]] (bavery)
* [[TipsAndTricks/DebuggingBitbakeInPudb]] (bavery)
* [[TipsAndTricks/DebuggingBitbakeInWingIDE]] (bavery)
* [[TipsAndTricks/CliBuildsInToasterBuilddir]] (bavery)
* [[TipsAndTricks/BuildingZephyrImages]] (Henry)
* [[TipsAndTricks/Cmake,Eclipse, and SDKS]] (bavery)
* [[TipsAndTricks/LinuxKernelAndSDKs]] (bavery)
* [[TipsAndTricks/Running YP binaries on Ubuntu and Vice Versa]] (bavery)
* [[TipsAndTricks/LockSharedState]] (Stephano)
* [[TipsAndTricks/Building and booting for Joule or MinnowBoard]] (Cal)
* [[TipsAndTricks/EnablingAPackageFeed]] (Henry)
* [[TipsAndTricks/Creating Recipes for ROS modules]] (Henry)

== Finished Articles ==
* [[TipsAndTricks/Packaging Prebuilt Libraries | Packaging Prebuilt Libraries]] (Ross and Henry)
* [[TipsAndTricks/NPM | Packaging Node.js Projects]] (Brendan and Henry)