Yocto Project - User contributions [en]

TipsAndTricks/Building core-image-minimal on AWS

2017-11-03T01:03:04Z

Bavery: /* Results */

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will be.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key to making a quick build! Here's a sample site.conf, (this was done for the 2.4 release):
<code>
SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
** Note, time to image is only 2 1/2 minutes . Pretty nice!
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...
* make sure to terminate the instance when you are done with it. this will delete the files, so make sure you have what you want saved.

= What's the Big Deal? =
Nothing, really. The point of this very short article is that it is trivial to get an EC2 instance up that can build a YP image and do so in a very short time, assuming you leverage our sstate mirrors.
 
Note: The free tier instance type <italic>t2.micro</italic> can NOT build core-image-minimal as it runs out of ram in it's default configuration.
= Results =
* Time to build core-image-minimal leveraging a release sstate-mirror - 2.5 minutes
* Cost on AWS (on demand cost which is higher than spot) - $0.29
** this includes me pulling in the necessary build dependencies, cloning poky, and building.

TipsAndTricks/Building core-image-minimal on AWS

2017-11-03T01:00:32Z

Bavery:

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:18:21Z

Bavery: /* After you have successfully ssh'ed into your EC2 Instance, what now? */

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:17:20Z

Bavery: /* After you have successfully ssh'ed into your EC2 Instance, what now? */

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will be.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>
SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...
* make sure to terminate the instance when you are done with it. this will delete the files, so make sure you have what you want saved.

= What's the Big Deal? =
Nothing, really. The point of this very short article is that it is trivial to get an EC2 instance up that can build a YP image and do so in a very short time, assuming you leverage our sstate mirrors.
 
Note: The free tier instance type <italic>t2.micro</italic> can NOT build core-image-minimal as it runs out of ram in it's default configuration.

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:16:23Z

Bavery: /* What's the Big Deal? */

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>
SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...
* make sure to terminate the instance when you are done with it. this will delete the files, so make sure you have what you want saved.

= What's the Big Deal? =
Nothing, really. The point of this very short article is that it is trivial to get an EC2 instance up that can build a YP image and do so in a very short time, assuming you leverage our sstate mirrors.
 
Note: The free tier instance type <italic>t2.micro</italic> can NOT build core-image-minimal as it runs out of ram in it's default configuration.

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:16:07Z

Bavery:

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>
SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...
* make sure to terminate the instance when you are done with it. this will delete the files, so make sure you have what you want saved.

= What's the Big Deal? =
Nothing, really. The point of this very short article is that it is trivial to get an EC2 instance up that can build a YP image and do so in a very short time, assuming you leverage our sstate mirrors.
Note: The free tier instance type <italic>t2.micro</italic> can NOT build core-image-minimal as it runs out of ram in it's default configuration.

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:11:52Z

Bavery: /* After you have successfully ssh'ed into your EC2 Instance, what now? */

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>
SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:11:31Z

Bavery:

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>

SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...

TipsAndTricks/Building core-image-minimal on AWS

2017-11-02T01:11:17Z

Bavery: Created page with "= Sometimes, a Cloud Server is just easier = I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long..."

= Sometimes, a Cloud Server is just easier =

I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html]

= My Setup =
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
<code>
du -sh ~/work
1.7G /home/ubuntu/work
</code>
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html quick start guide], you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: <code> $ git clone git://git.yoctoproject.org/poky </code> and checkout the release: <code> $ git checkout yocto-2.4 </code>
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will.
* set up the environment <code> $ . ./poky/oe-init-build-env </code>
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key Sample site.conf, (this was done for the 2.4 release):
<code>

SSTATE_MIRRORS = "\
file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
"
</code>
* build it <code> $ time bitbake core-image-minimal </code>
<code>
... many setscene tasks ...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.

real 2m26.947s
user 0m5.832s
sys 0m0.924s
</code>
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...

TipsAndTricks

2017-11-02T00:40:00Z

Bavery:

== Background ==
This wiki page captures ideas for "Tips and tricks" articles that are aimed Yocto Project users that have a basic understanding of the core tools and want to extend their knowledge. Articles will written and published using the following process
* Articles must refer to the current (or earlier) release. They must not cover features in development in the master branch.
* Anyone can add entries to the '''Ideas for Articles''' section based on challenges they have encountered when using the Yocto Project
* More experienced developers can start to flesh out articles in the ''Ideas'' section move then to '''Articles in Development''' or contribute directly to this section.
* Approximately once a month an article ready for publishing will be chosen for publication by [mailto:jeffrey.osier-mixon@intel.com Jefro] and moved to the '''Finished Articles ''' section.

Please contact [mailto:henry.bruce@intel.com?Subject=Yocto%20Project%20Tips%20and%20Tricks Henry Bruce] with any questions.

== Ideas for Articles ==
* [[TipsAndTricks/GeneratingASDK]] (Ross, Paul help please...)
* [[TipsAndTricks/AddingALicense]] (Ross)
* [[TipsAndTricks/UsingBuildstatsDiff]] (Ross)
* [[TipsAndTricks/DevPyShell]] (Richard Purdie)
* [[TipsAndTricks/CreatingABSP]]. And what not to put in it.
* [[TipsAndTricks/Patchwork]]
* [[TipsAndTricks/Patchtest]] (Leo)
* [[TipsAndTricks/CreateNewLayer]] (Brendan)
* [[TipsAndTricks/JenkinsPipelinesForContinuousIntegration]] (Tim Orling)

== Articles in development ==
* [[TipsAndTricks/Building core-image-minimal on AWS]] (bavery)
* [[TipsAndTricks/DockerOnImage]] (bavery)
* [[TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds]] (bavery)
* [[TipsAndTricks/BuildingAndRunningClearContainersonTarget]] (bavery)
* [[TipsAndTricks/KernelDevelopmentWithEsdk]] (Todor Minchev, Henry Bruce)
* [[TipsAndTricks/DebugNativeRecipeWithGdb]] (Joshua Lock)
* [[TipsAndTricks/Netconsole]] (Ross Burton)
* [[TipsAndTricks/ParsingProfiling]] (Richard Purdie)
* [[TipsAndTricks/DemystifyingTheLinuxYoctoKernel]] (Tom Zanussi)
* [[TipsAndTricks/Patching the source for a recipe]] (Paul Eggleton)
* [[TipsAndTricks/Incorporating closed source components]] (Paul Eggleton)
* [[TipsAndTricks/DebuggingAvoidingRebuilds]] (Richard Purdie)
* [[TipsAndTricks/GitBisectABitbake]] (Ross)
* [[TipsAndTricks/RunningEclipseAgainstBuiltImage]] (bavery)
* [[TipsAndTricks/PrelinkSomePointersAndWorkarounds]] (bavery)
* [[TipsAndTricks/RunningQemuOnMacOSX]] (Stephano)
* [[TipsAndTricks/Understanding what changed (diffsigs etc)]] (Joshua)
* [[TipsAndTricks/CropsCLIContainers]] (bavery)
* [[TipsAndTricks/QuickAndDirtyKernelConfig]] (bavery)
* [[TipsAndTricks/TestingToasterWithContainers]] (bavery)
* [[TipsAndTricks/InvestigatingBuildTime]] (Leo Sandoval)
* [[TipsAndTricks/ResolvingLocaleIssues]] (bavery)
* [[TipsAndTricks/DebuggingBitbakeInPudb]] (bavery)
* [[TipsAndTricks/DebuggingBitbakeInWingIDE]] (bavery)
* [[TipsAndTricks/CliBuildsInToasterBuilddir]] (bavery)
* [[TipsAndTricks/BuildingZephyrImages]] (Henry)
* [[TipsAndTricks/Cmake,Eclipse, and SDKS]] (bavery)
* [[TipsAndTricks/LinuxKernelAndSDKs]] (bavery)
* [[TipsAndTricks/Running YP binaries on Ubuntu and Vice Versa]] (bavery)
* [[TipsAndTricks/LockSharedState]] (Stephano)
* [[TipsAndTricks/Building and booting for Joule or MinnowBoard]] (Cal)
* [[TipsAndTricks/EnablingAPackageFeed]] (Henry)
* [[TipsAndTricks/Creating Recipes for ROS modules]] (Henry)
* [[TipsAndTricks/TeamWorkflows]] (Joshua)

== Finished Articles ==
* [[TipsAndTricks/Packaging Prebuilt Libraries | Packaging Prebuilt Libraries]] (Ross and Henry)
* [[TipsAndTricks/NPM | Packaging Node.js Projects]] (Brendan and Henry)

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-10-19T22:25:15Z

Bavery: /* DEPRECATED */

= DEPRECATED =
 There is now a http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-clear-containers/ layer that can be built as part of a standard YP build. 
 
 
 
 

 

= Historical value only =
== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--prefix=/usr' '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-10-19T22:24:43Z

Bavery: /* DEPRECATED */

= DEPRECATED =
 There is now a http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-clear-containers/ layer that can be built as part of a standard YP build. 

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--prefix=/usr' '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-10-19T22:24:33Z

Bavery:

= DEPRECATED =
 There is now a http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-clear-containers/ layer that can be built as part of a standard YP build. 

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--prefix=/usr' '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--extra-cflags=-Wno-format-truncation' '--target-list=x86_64-softmmu' "$@"


Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/KernelDevelopmentWithEsdk

2017-10-04T19:50:43Z

Bavery:

== Overview ==
This article describes an efficient workflow for Linux kernel development using Yocto Project. The goal is to get the build, deploy and test cycle to under a minute. We will be working with the generix86-64 BSP and the Intel Minnowboard Turbot in this article but the process is the same for any platform and CPU architecture.

The extensible SDK (eSDK) is a key part of this workflow for a number of reasons
# It contains the target toolchain which is not easily accessible in the bitbake environment
# It contains '''devtool''' which gives you an easy way of getting the kernel source your target is using (if using linux-yocto or linux-intel)
# devtool will also automatically create patches for your kernel updates and add them to the the kernel recipe

There are two separate steps to this process
# Build or download an eSDK for your platform
# Install and use that eSDK to build your kernel

== Build Extensible SDK ==
Open a new terminal window, we'll refer to this terminal as your '''bitbake terminal'''. First we'll checkout poky and configure bitbake environment.
<pre>
$ cd ~
$ git clone -b pyro git://git.yoctoproject.org/poky
$ source poky/oe-init-build-env
</pre>
Now we need to do some configuration for the Minnowboard image.
* Set MACHINE (as default is qemux86)
* Enable kernel modules (disabled by default for minimal images)
Add the following to conf/local.conf
MACHINE = "genericx86-64"
MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "kernel-modules"

Now build an extensible SDK installer for the Minnowboard in the bitbake terminal
<pre>
$ bitbake core-image-minimal -c populate_sdk_ext
</pre>
The eSDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh

== Install Extensible SDK ==
Run the installer as follows. If you do not want to use the default ~/poky_sdk directory for installation of the toolchain, use the -d option to specify a destination.
<pre>
$ ./poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh
Poky (Yocto Project Reference Distro) Extensible SDK installer version 2.3.1
============================================================================
Enter target directory for SDK (default: ~/poky_sdk):
You are about to install the SDK to "/home/scottrif/poky_sdk". Proceed[Y/n]? Y
Extracting SDK......................................done
Setting it up...
Extracting buildtools...
Preparing build system...
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:52
Initialising tasks: 100% |######################################################################################################| Time: 0:00:04
Checking sstate mirror object availability: 100% |##############################################################################| Time: 0:00:00
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:33
Initialising tasks: 100% |######################################################################################################| Time: 0:00:00
done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /home/scottrif/poky_sdk/environment-setup-core2-64-poky-linux
</pre>
Take note of the final line, which shows you the environment setup script you must run each time you want to use the eSDK.

== Create a Layer ==
We also need to create a layer to take the kernel patches we'll be creating.
<pre>
$ cd ~/poky
$ yocto-layer create my-kernel -o ../meta-my-kernel
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n]
Would you like to have an example bbappend file created? (y/n) [default: n]

New layer created in ../meta-my-kernel.

Don't forget to add it to your BBLAYERS (for details see ../meta-my-kernel/README).
</pre>
As directed, we need to add our layer to BBLAYERS as follows:
<pre>
$ cd ~/poky/build
$ bitbake-layers add-layer ../../meta-my-kernel
</pre>

== Setup your ESDK build environment (ESDK terminal) ==
'''YOU MUST OPEN A NEW TERMINAL WHEN USING THE ESDK. DO NOT RE-USE YOUR BITBAKE TERMINAL.'''
Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'. Run the setup script given by the installer.
$ source /path/to/esdk/environment-setup-core2-64-poky-linux
"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.

If you see this
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a
new shell session instead."
You didn't open a new terminal!

== Build initial image with ESDK ==
In the ESDK terminal, build the image
<pre>
$ devtool build-image
Parsing recipes: 100% |##########################################| Time: 0:00:05
Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 47 skipped, 0 masked, 0 errors.
WARNING: No packages to add, building image core-image-minimal unmodified
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 1299 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |#######################################| Time: 0:00:07
Checking sstate mirror object availability: 100% |###############| Time: 0:00:00
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2866 tasks of which 2604 didn't need to be rerun and all succeeded.
NOTE: Successfully built core-image-minimal. You can find output files in /path/to/esdk/tmp/deploy/images/genericx86-64
</pre>
Now flash to image to USB stick, assuming USB stick is on /dev/sdd
$ sudo dd if=tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot the Minnowboard with this image and check to be sure that everything is OK.

== Working with Yocto kernel ==
=== Checkout kernel source ===
First we must use devtool to checkout the kernel source code in its workspace. Do the following in the ESDK terminal. Note that we use the virtual kernel provider so we don't need to know the kernel recipe name.

<pre>
$ devtool modify virtual/kernel
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Mapping virtual/kernel to linux-yocto
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_checkout...
NOTE: Tasks Summary: Attempted 3 tasks of which 2 didn't need to be rerun and all succeeded.
NOTE: Patching...
NOTE: Executing RunQueue Tasks
NOTE: Executing do_validate_branches...
NOTE: Executing do_kernel_metadata...
NOTE: Executing do_patch...
NOTE: Tasks Summary: Attempted 6 tasks of which 3 didn't need to be rerun and all succeeded.
NOTE: Generating kernel config
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_configme...
NOTE: Executing do_prepare_recipe_sysroot...
NOTE: Executing do_configure...
NOTE: Tasks Summary: Attempted 9 tasks of which 6 didn't need to be rerun and all succeeded.
NOTE: Copying kernel config to srctree
NOTE: Source tree extracted to /home/scottrif/poky_sdk/workspace/sources/linux-yocto
NOTE: Recipe linux-yocto now set up to build from /home/scottrif/poky_sdk/workspace/sources/linux-yocto
</pre>

There is a bug that may cause error messages such as the following to be displayed. These can be ignored, the source code will be correct checked out.
ERROR: Taskhash mismatch 2c793438c2d9f8c3681fd5f7bc819efa versus be3a89ce7c47178880ba7bf6293d7404 for /path/to/esdk/layers/poky/meta/recipes-kernel/linux/linux-yocto_4.10.bb.do_unpack

=== Edit and build driver ===
Kernel source will be in the location shown at end of the <tt>devtool modify virtual/kernel</tt> output. Let's make a trivial change to the serial driver in drivers/tty/serial/8250/8250_core.c. After the line
pr_info("Serial: 8250/16550 driver, %d ports, IRQ sharing %sabled\n",
nr_uarts, share_irqs ? "en" : "dis");
Add the line
pr_info("Serial: 8250/16550 driver modified with eSDK");
To build the updated kernel source, in the SDK terminal run make in your kernel source folder. Adjust the -j option to match number of cores on your workstation.
$ make -C workspace/sources/linux-yocto -j4
Alternatively, you can run
$ devtool build linux-yocto

=== Create image with new kernel 2.4 Way ===
Note: There are several bugs outstanding to make this process a little easier as of 9/2017. They mostly deal with unwieldy paths and are easy to work around. I will detail the workarounds here and if it is clear that you don't need them due to the bugs getting fixed and merged, so much the better.
 
Below I detail using kpartx to pull apart a wic image so you can just copy your kernel (and other artifacts if needed like a dtd) into the image to shorten turnaround time. In this section, we will exploit the new features in wic that allow us to do the same thing but easier. This does require you to install the host side mtools package, currently.

==== Image manipulation with wic to facilitate kernel development ====
* See what's in your image:
$ wic ls /tmp/core-image-base-qemux86-64.wic
Num Start End Size Fstype
1 1048576 25534463 24485888 fat16
2 26214400 13938011135 13911796736 ext4
* See what's in your partition:
$ wic ls /tmp/core-image-base-qemux86-64.wic:1

If you see the following error:
ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
output: Total number of sectors (47824) not a multiple of sectors per track (32)!
Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
Just do what it says and put
mtools_skip_check=1
into your ~/.mtoolsrc file and you will get:
$wic ls /tmp/core-image-base-qemux86-64.wic:1
Volume in drive : is boot
Volume Serial Number is 6BF1-7DEC
Directory for ::/

libcom32 c32 186500 2017-10-03 21:53
libutil c32 24148 2017-10-03 21:53
syslinux cfg 220 2017-10-03 21:53
vesamenu c32 27104 2017-10-03 21:53
vmlinuz 7397936 2017-10-03 21:53
5 files 7 635 908 bytes
16 582 656 bytes free

* remove the old kernel from your image:
$ wic rm /tmp/core-image-base-qemux86-64.wic:1/vmlinuz
* add in your freshly constructed kernel.
** if you built the kernel with devtool, the resulting bzImage is in the tmp/work directory as below.
$ wic cp ~/poky_sdk/tmp/work/qemux86_64-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage /tmp/core-image-base-qemux86-64.wic:1/vmlinuz
** if you built it straight with make in the workspace/linux-yocto directory, it is in a more standard place:
$ cp workspace/sources/linux-yocto/arch/x86/boot/bzImage /tmp/core-image-base-qemux86-64.wic:1/vmlinuz

* then you can dd your wic image onto a sd card/usb key and you can test your target.
** note: bmap-copy using the bmap tools is generally ~ 10x-20x faster than a straight dd. Docs -> [http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#flashing-images-using-bmaptool Official bmaptool copy docs]

=== Create image with new kernel Pre 2.4 Way ===
Normally you'd make a new image with <tt>devtool build-image</tt> but this can take several minutes or so and messes with the kernel source folder. A faster option is to use kpartx to splice the new kernel into the image you have already built. NOTE: If your build system does not have kpartx installed, be sure you install it.

First make a copy of your wic file in say /tmp
$ cp tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic /tmp
Then create loopback devices for each partition in wic file.
NOTE: The first unused loopback device is automatically allocated. The example command output here uses a variable "X" to indicate the loopback device. The actual output you get when you run the command will list the actual loopback devices (e.g. "loop0p1", "loop0p2", and "loop0p3").
$ sudo kpartx -v -a /tmp/core-image-minimal-genericx86-64.wic
add map loopXp1 (253:6): 0 47446 linear /dev/loopX 2048
add map loopXp2 (253:7): 0 119356 linear /dev/loopX 51200
add map loopXp3 (253:8): 0 90112 linear /dev/loopX 170556
Kernel is in the first device, so let's mount /dev/mapper/loopXp1 to take a look
NOTE: Replace loopXp1 with the automatically allocated loopback device (e.g loop0p1).
$ sudo mkdir /mnt/wic-p1
$ sudo mount /dev/mapper/loopXp1 /mnt/wic-p1
Now copy over new kernel
$ sudo cp workspace/sources/linux-yocto/arch/x86/boot/bzImage /mnt/wic-p1
Then unmount and unkpartx...
NOTE: Replace "loopX" with the automatically allocated loopback device from earlier (e.g "loop0").
$ sudo umount /mnt/wic-p1
$ sudo kpartx -d /dev/loopX
Now flash image
$ dd if=/tmp/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot Minnowboard, log on and look in log for driver message
# root@genericx86-64:~# dmesg | grep Serial:
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Serial: 8250/16550 driver modified with eSDK
It worked!

=== Export patches and create a bbappend file ===
To export your commits as patches and a bbappend file use the following command in the ESDK terminal. We'll target the "my-kernel" layer we created in the bitbake terminal
$ devtool finish linux-yocto /path/to/meta-my-kernel
The patches and the bbappend can be found in /path/to/meta-my-kernel/recipes-kernel/linux.

== Working with upstream kernel ==
Slightly different workflow where we use a kernel from kernel.org.

Content TBD

== Build image with your modified kernel ==
You can now build an image which will include your kernel patches.
Execute the following command from your build directory in the bitbake terminal
bitbake core-image-minimal

TipsAndTricks/KernelDevelopmentWithEsdk

2017-10-04T18:26:00Z

Bavery:

== Overview ==
This article describes an efficient workflow for Linux kernel development using Yocto Project. The goal is to get the build, deploy and test cycle to under a minute. We will be working with the generix86-64 BSP and the Intel Minnowboard Turbot in this article but the process is the same for any platform and CPU architecture.

The extensible SDK (eSDK) is a key part of this workflow for a number of reasons
# It contains the target toolchain which is not easily accessible in the bitbake environment
# It contains '''devtool''' which gives you an easy way of getting the kernel source your target is using (if using linux-yocto or linux-intel)
# devtool will also automatically create patches for your kernel updates and add them to the the kernel recipe

There are two separate steps to this process
# Build or download an eSDK for your platform
# Install and use that eSDK to build your kernel

== Build Extensible SDK ==
Open a new terminal window, we'll refer to this terminal as your '''bitbake terminal'''. First we'll checkout poky and configure bitbake environment.
<pre>
$ cd ~
$ git clone -b pyro git://git.yoctoproject.org/poky
$ source poky/oe-init-build-env
</pre>
Now we need to do some configuration for the Minnowboard image.
* Set MACHINE (as default is qemux86)
* Enable kernel modules (disabled by default for minimal images)
Add the following to conf/local.conf
MACHINE = "genericx86-64"
MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "kernel-modules"

Now build an extensible SDK installer for the Minnowboard in the bitbake terminal
<pre>
$ bitbake core-image-minimal -c populate_sdk_ext
</pre>
The eSDK installer can be found in /path_to_build_directory/kernel-dev/tmp/deploy/sdk/
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh

== Install Extensible SDK ==
Run the installer as follows. If you do not want to use the default ~/poky_sdk directory for installation of the toolchain, use the -d option to specify a destination.
<pre>
$ ./poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-ext-2.3.1.sh
Poky (Yocto Project Reference Distro) Extensible SDK installer version 2.3.1
============================================================================
Enter target directory for SDK (default: ~/poky_sdk):
You are about to install the SDK to "/home/scottrif/poky_sdk". Proceed[Y/n]? Y
Extracting SDK......................................done
Setting it up...
Extracting buildtools...
Preparing build system...
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:52
Initialising tasks: 100% |######################################################################################################| Time: 0:00:04
Checking sstate mirror object availability: 100% |##############################################################################| Time: 0:00:00
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:33
Initialising tasks: 100% |######################################################################################################| Time: 0:00:00
done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /home/scottrif/poky_sdk/environment-setup-core2-64-poky-linux
</pre>
Take note of the final line, which shows you the environment setup script you must run each time you want to use the eSDK.

== Create a Layer ==
We also need to create a layer to take the kernel patches we'll be creating.
<pre>
$ cd ~/poky
$ yocto-layer create my-kernel -o ../meta-my-kernel
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n]
Would you like to have an example bbappend file created? (y/n) [default: n]

New layer created in ../meta-my-kernel.

Don't forget to add it to your BBLAYERS (for details see ../meta-my-kernel/README).
</pre>
As directed, we need to add our layer to BBLAYERS as follows:
<pre>
$ cd ~/poky/build
$ bitbake-layers add-layer ../../meta-my-kernel
</pre>

== Setup your ESDK build environment (ESDK terminal) ==
'''YOU MUST OPEN A NEW TERMINAL WHEN USING THE ESDK. DO NOT RE-USE YOUR BITBAKE TERMINAL.'''
Open a new terminal and setup your build environment. We'll refer to this terminal as your 'ESDK terminal'. Run the setup script given by the installer.
$ source /path/to/esdk/environment-setup-core2-64-poky-linux
"SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.

If you see this
WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a
new shell session instead."
You didn't open a new terminal!

== Build initial image with ESDK ==
In the ESDK terminal, build the image
<pre>
$ devtool build-image
Parsing recipes: 100% |##########################################| Time: 0:00:05
Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 47 skipped, 0 masked, 0 errors.
WARNING: No packages to add, building image core-image-minimal unmodified
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 1299 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |#######################################| Time: 0:00:07
Checking sstate mirror object availability: 100% |###############| Time: 0:00:00
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2866 tasks of which 2604 didn't need to be rerun and all succeeded.
NOTE: Successfully built core-image-minimal. You can find output files in /path/to/esdk/tmp/deploy/images/genericx86-64
</pre>
Now flash to image to USB stick, assuming USB stick is on /dev/sdd
$ sudo dd if=tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot the Minnowboard with this image and check to be sure that everything is OK.

== Working with Yocto kernel ==
=== Checkout kernel source ===
First we must use devtool to checkout the kernel source code in its workspace. Do the following in the ESDK terminal. Note that we use the virtual kernel provider so we don't need to know the kernel recipe name.

<pre>
$ devtool modify virtual/kernel
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Mapping virtual/kernel to linux-yocto
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1300 entries from dependency cache.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_checkout...
NOTE: Tasks Summary: Attempted 3 tasks of which 2 didn't need to be rerun and all succeeded.
NOTE: Patching...
NOTE: Executing RunQueue Tasks
NOTE: Executing do_validate_branches...
NOTE: Executing do_kernel_metadata...
NOTE: Executing do_patch...
NOTE: Tasks Summary: Attempted 6 tasks of which 3 didn't need to be rerun and all succeeded.
NOTE: Generating kernel config
NOTE: Executing RunQueue Tasks
NOTE: Executing do_kernel_configme...
NOTE: Executing do_prepare_recipe_sysroot...
NOTE: Executing do_configure...
NOTE: Tasks Summary: Attempted 9 tasks of which 6 didn't need to be rerun and all succeeded.
NOTE: Copying kernel config to srctree
NOTE: Source tree extracted to /home/scottrif/poky_sdk/workspace/sources/linux-yocto
NOTE: Recipe linux-yocto now set up to build from /home/scottrif/poky_sdk/workspace/sources/linux-yocto
</pre>

There is a bug that may cause error messages such as the following to be displayed. These can be ignored, the source code will be correct checked out.
ERROR: Taskhash mismatch 2c793438c2d9f8c3681fd5f7bc819efa versus be3a89ce7c47178880ba7bf6293d7404 for /path/to/esdk/layers/poky/meta/recipes-kernel/linux/linux-yocto_4.10.bb.do_unpack

=== Edit and build driver ===
Kernel source will be in the location shown at end of the <tt>devtool modify virtual/kernel</tt> output. Let's make a trivial change to the serial driver in drivers/tty/serial/8250/8250_core.c. After the line
pr_info("Serial: 8250/16550 driver, %d ports, IRQ sharing %sabled\n",
nr_uarts, share_irqs ? "en" : "dis");
Add the line
pr_info("Serial: 8250/16550 driver modified with eSDK");
To build the updated kernel source, in the SDK terminal run make in your kernel source folder. Adjust the -j option to match number of cores on your workstation.
$ make -C workspace/sources/linux-yocto -j4
Alternatively, you can run
$ devtool build linux-yocto

=== Create image with new kernel 2.4 Way ===
Note: There are several bugs outstanding to make this process a little easier as of 9/2017. They mostly deal with unwieldy paths and are easy to work around. I will detail the workarounds here and if it is clear that you don't need them due to the bugs getting fixed and merged, so much the better.
 
Below I detail using kpartx to pull apart a wic image so you can just copy your kernel (and other artifacts if needed like a dtd) into the image to shorten turnaround time. In this section, we will exploit the new features in wic that allow us to do the same thing but easier. This does require you to install the host side mtools package, currently.

==== Image manipulation with wic to facilitate kernel development ====
* See what's in your image:
$ wic ls /tmp/core-image-base-qemux86-64.wic
Num Start End Size Fstype
1 1048576 25534463 24485888 fat16
2 26214400 13938011135 13911796736 ext4
* See what's in your partition:
$ wic ls /tmp/core-image-base-qemux86-64.wic:1

If you see the following error:
ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
output: Total number of sectors (47824) not a multiple of sectors per track (32)!
Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
Just do what it says and put
mtools_skip_check=1
into your ~/.mtoolsrc file and you will get:
$wic ls /tmp/core-image-base-qemux86-64.wic:1
Volume in drive : is boot
Volume Serial Number is 6BF1-7DEC
Directory for ::/

libcom32 c32 186500 2017-10-03 21:53
libutil c32 24148 2017-10-03 21:53
syslinux cfg 220 2017-10-03 21:53
vesamenu c32 27104 2017-10-03 21:53
vmlinuz 7397936 2017-10-03 21:53
5 files 7 635 908 bytes
16 582 656 bytes free

=== Create image with new kernel Pre 2.4 Way ===
Normally you'd make a new image with <tt>devtool build-image</tt> but this can take several minutes or so and messes with the kernel source folder. A faster option is to use kpartx to splice the new kernel into the image you have already built. NOTE: If your build system does not have kpartx installed, be sure you install it.

First make a copy of your wic file in say /tmp
$ cp tmp/deploy/images/genericx86-64/core-image-minimal-genericx86-64.wic /tmp
Then create loopback devices for each partition in wic file.
NOTE: The first unused loopback device is automatically allocated. The example command output here uses a variable "X" to indicate the loopback device. The actual output you get when you run the command will list the actual loopback devices (e.g. "loop0p1", "loop0p2", and "loop0p3").
$ sudo kpartx -v -a /tmp/core-image-minimal-genericx86-64.wic
add map loopXp1 (253:6): 0 47446 linear /dev/loopX 2048
add map loopXp2 (253:7): 0 119356 linear /dev/loopX 51200
add map loopXp3 (253:8): 0 90112 linear /dev/loopX 170556
Kernel is in the first device, so let's mount /dev/mapper/loopXp1 to take a look
NOTE: Replace loopXp1 with the automatically allocated loopback device (e.g loop0p1).
$ sudo mkdir /mnt/wic-p1
$ sudo mount /dev/mapper/loopXp1 /mnt/wic-p1
Now copy over new kernel
$ sudo cp workspace/sources/linux-yocto/arch/x86/boot/bzImage /mnt/wic-p1
Then unmount and unkpartx...
NOTE: Replace "loopX" with the automatically allocated loopback device from earlier (e.g "loop0").
$ sudo umount /mnt/wic-p1
$ sudo kpartx -d /dev/loopX
Now flash image
$ dd if=/tmp/core-image-minimal-genericx86-64.wic of=/dev/sdd bs=1MB
$ sync
Boot Minnowboard, log on and look in log for driver message
# root@genericx86-64:~# dmesg | grep Serial:
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Serial: 8250/16550 driver modified with eSDK
It worked!

=== Export patches and create a bbappend file ===
To export your commits as patches and a bbappend file use the following command in the ESDK terminal. We'll target the "my-kernel" layer we created in the bitbake terminal
$ devtool finish linux-yocto /path/to/meta-my-kernel
The patches and the bbappend can be found in /path/to/meta-my-kernel/recipes-kernel/linux.

== Working with upstream kernel ==
Slightly different workflow where we use a kernel from kernel.org.

Content TBD

== Build image with your modified kernel ==
You can now build an image which will include your kernel patches.
Execute the following command from your build directory in the bitbake terminal
bitbake core-image-minimal

TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

2017-09-20T18:24:45Z

Bavery: /* Ways to Serve up the rpm repo */

= Leveraging Pkg Feeds or How I got Tired of Burning Images =
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
 Note: This works as well with kernel modules as it does with libraries/binaries
 
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]
== Steps ==
* It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management dev-pkgs tools-sdk tools-debug tools-profile "
* Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure
This worked, but we decide we want to have cyrus support as well.
* First we try
./configure --enable-cyrus
but we get
configure: error: could not find libsasl2. Try --disable-cyrus.

* So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use [https://layers.openembedded.org/layerindex/branch/master/layers/ openembedded layer index] and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
bitbake cyrus-sasl
and remake the package index
bitbake package-index
* target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)
If this fails, make sure you followed the instructions in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.
* now we rerun configure and get:
./configure --enable-cyrus

ZNC 1.6.5 configured
prefix: /usr/local
debug: no
ipv6: yes
openssl: yes
dns: threads
perl: no
python: no
swig: not needed
cyrus: yes
tcl: no
charset: yes
zlib: yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.
* Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.
Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule
$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py: fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc: ${libexecdir}/git-core/git-submodule \
If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:
PACKAGES =+ "${PN}-perltools".
So we can pull in the git-perltools as well on the target:
# dnf install git-perltools
and rerun the git submodule init
# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!
# make
Building module autocycle...
Linking module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...

ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.

* I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index
Target ->
# dnf install zile

== Ways to Serve up the rpm repo ==

* very simple - use python twistd.
** let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.
** <code >
cd /home/me/mrrepo;
twistd -n web --path . -p 5678
</code>
** similarly, if you are just serving up the build directories rpm's you can just do
*** <code >
cd /home/me/mrrepo;
twistd -n web --path tmp/deploy/rpm -p 5678
</code>
*** Note , only 1 process can own a given port.
* less simple, but better response and better stability- use lighthttpd.
** I advocate using this from a container as it is simpler to maintain.
** <code>
docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd
</code>
** What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
** Where did they come from? Copied from inside the container
*** <code> docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd </code>
*** <code> # cp -a /etc/lighttpd/* /frog </code>
*** Now your /home/me/lightppd has the necessary config files.
** The files in /home/me/lighthttpd:
*** lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
**** From this you can see I run emacs. The only change I did was to turn on directory listing .
**** diff lighttpd.conf lighttpd.conf~
***** <code>
114c114
< dir-listing.activate = "enable"
---
> # dir-listing.activate = "enable"
</code>

** the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.

TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

2017-09-20T18:24:05Z

Bavery: /* Ways to Serve up the rpm repo */

= Leveraging Pkg Feeds or How I got Tired of Burning Images =
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
 Note: This works as well with kernel modules as it does with libraries/binaries
 
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]
== Steps ==
* It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management dev-pkgs tools-sdk tools-debug tools-profile "
* Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure
This worked, but we decide we want to have cyrus support as well.
* First we try
./configure --enable-cyrus
but we get
configure: error: could not find libsasl2. Try --disable-cyrus.

* So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use [https://layers.openembedded.org/layerindex/branch/master/layers/ openembedded layer index] and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
bitbake cyrus-sasl
and remake the package index
bitbake package-index
* target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)
If this fails, make sure you followed the instructions in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.
* now we rerun configure and get:
./configure --enable-cyrus

ZNC 1.6.5 configured
prefix: /usr/local
debug: no
ipv6: yes
openssl: yes
dns: threads
perl: no
python: no
swig: not needed
cyrus: yes
tcl: no
charset: yes
zlib: yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.
* Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.
Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule
$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py: fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc: ${libexecdir}/git-core/git-submodule \
If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:
PACKAGES =+ "${PN}-perltools".
So we can pull in the git-perltools as well on the target:
# dnf install git-perltools
and rerun the git submodule init
# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!
# make
Building module autocycle...
Linking module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...

ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.

* I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index
Target ->
# dnf install zile

== Ways to Serve up the rpm repo ==

* very simple - use python twistd.
** let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.
** <code >
cd /home/me/mrrepo;
twistd -n web --path . -p 5678
</code>
** similarly, if you are just serving up the build directories rpm's you can just do
*** <code >
cd /home/me/mrrepo;
twistd -n web --path tmp/deploy/rpm -p 5678
</code>
*** Note , only 1 process can own a given port.
* less simple, but better response and better stability- use lighthttpd.
** I advocate using this from a container as it is simpler to maintain.
** <code>
docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd
</code>
** What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
** Where did they come from? Copied from inside the container
*** <code> docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd </code>
*** <code> # cp -a /etc/lighttpd/* /frog </code>
*** Now your /home/me/lightppd has the necessary config files.
** The files in /home/me/lighthttpd:
*** lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
**** From this you can see I run emacs. The only change I did was to turn on directory listing .
**** diff lighttpd.conf lighttpd.conf~
***** <code> 114c114
< dir-listing.activate = "enable"
---
> # dir-listing.activate = "enable"
</code>

** the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.

TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

2017-09-20T18:23:53Z

Bavery: /* Ways to Serve up the rpm repo */

= Leveraging Pkg Feeds or How I got Tired of Burning Images =
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
 Note: This works as well with kernel modules as it does with libraries/binaries
 
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]
== Steps ==
* It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management dev-pkgs tools-sdk tools-debug tools-profile "
* Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure
This worked, but we decide we want to have cyrus support as well.
* First we try
./configure --enable-cyrus
but we get
configure: error: could not find libsasl2. Try --disable-cyrus.

* So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use [https://layers.openembedded.org/layerindex/branch/master/layers/ openembedded layer index] and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
bitbake cyrus-sasl
and remake the package index
bitbake package-index
* target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)
If this fails, make sure you followed the instructions in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.
* now we rerun configure and get:
./configure --enable-cyrus

ZNC 1.6.5 configured
prefix: /usr/local
debug: no
ipv6: yes
openssl: yes
dns: threads
perl: no
python: no
swig: not needed
cyrus: yes
tcl: no
charset: yes
zlib: yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.
* Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.
Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule
$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py: fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc: ${libexecdir}/git-core/git-submodule \
If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:
PACKAGES =+ "${PN}-perltools".
So we can pull in the git-perltools as well on the target:
# dnf install git-perltools
and rerun the git submodule init
# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!
# make
Building module autocycle...
Linking module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...

ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.

* I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index
Target ->
# dnf install zile

== Ways to Serve up the rpm repo ==

* very simple - use python twistd.
** let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.
** <code > cd /home/me/mrrepo;
twistd -n web --path . -p 5678
</code>
** similarly, if you are just serving up the build directories rpm's you can just do
*** <code >
cd /home/me/mrrepo;
twistd -n web --path tmp/deploy/rpm -p 5678
</code>
*** Note , only 1 process can own a given port.
* less simple, but better response and better stability- use lighthttpd.
** I advocate using this from a container as it is simpler to maintain.
** <code>
docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd
</code>
** What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
** Where did they come from? Copied from inside the container
*** <code> docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd </code>
*** <code> # cp -a /etc/lighttpd/* /frog </code>
*** Now your /home/me/lightppd has the necessary config files.
** The files in /home/me/lighthttpd:
*** lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
**** From this you can see I run emacs. The only change I did was to turn on directory listing .
**** diff lighttpd.conf lighttpd.conf~
***** <code> 114c114
< dir-listing.activate = "enable"
---
> # dir-listing.activate = "enable"
</code>

** the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.

TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

2017-09-20T18:23:32Z

Bavery:

= Leveraging Pkg Feeds or How I got Tired of Burning Images =
I was trying to build a sw repo on target and found that I lacked many of the tools and libraries in order to accomplish this. I tend to do this first, if I want to make a recipe from scratch for software that I am not too familiar with. This way, I can follow the instructions for building w/o reinterpretation, verify that I have or can build the necessary dependencies, and check that the sw works properly; all before diving into recipes. I have found reflashing and rebooting the image on the target to be quite slow. Luckily YP supports a much better workflow for this process.
 Note: This works as well with kernel modules as it does with libraries/binaries
 
In this example, we will be building znc 1.6.5, an irc bouncer. This is just a good complex example. In order to use this guide, you should set up your bitbake environment and your target as outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]
== Steps ==
* It's typically easies to make an image that has *most* of what you think you'll need. In this example, I added the following to my local.conf:
EXTRA_IMAGE_FEATURES += " ssh-server-openssh package-management dev-pkgs tools-sdk tools-debug tools-profile "
* Now I'll grab the src code we will try to build Note: this is an ON TARGET build the build host side is just for the dependencies...
git clone https://github.com/znc/znc.git
cd znc
git checkout -b 1.6.5 znc-1.6.5
* follow the build instructions
./autogen.sh
./configure
This worked, but we decide we want to have cyrus support as well.
* First we try
./configure --enable-cyrus
but we get
configure: error: could not find libsasl2. Try --disable-cyrus.

* So we need the cyrus-sals-dev pkg. If we aren't sure where that is, we can use [https://layers.openembedded.org/layerindex/branch/master/layers/ openembedded layer index] and search for the recipe cyrus (this finds cyrus-sasl2 in meta-networking). So, we add meta-networking to bblayers (and meta-oe and meta-python, as its dependencies) then build cyrus-sasl.
bitbake cyrus-sasl
and remake the package index
bitbake package-index
* target side, pull in the new package. Since I am building stuff, I need the -dev pkg for configure to be happy.
$dnf install cyrus-sasl-dev (note this will also pull in the cyrus-sasl rpm as a dependency.)
If this fails, make sure you followed the instructions in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/EnablingAPackageFeed#Easiest_Way PkgFeeds the EasiestWay]. You should have set up the on target repo in /etc/yum.repos.d/myrepo.repo and have a web server of some type pointed to tmp/deploy/rpm (like twistd -n web --path tmp/deploy/rpm -p 5678) on your build host.
* now we rerun configure and get:
./configure --enable-cyrus

ZNC 1.6.5 configured
prefix: /usr/local
debug: no
ipv6: yes
openssl: yes
dns: threads
perl: no
python: no
swig: not needed
cyrus: yes
tcl: no
charset: yes
zlib: yes
run from src: no

Pretty Easy, you can see how this cycle may be repeated a number of times for more complex projects.
* Let's build it
~/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [Makefile:168: third_party/Csocket/Csocket.h] Error 1
~/znc# git submodule update --init --recursive
git: 'submodule' is not a git command. See 'git --help'.
Whoops, we need git submodule... This one is harder to find. After verifying that we already have git-dev installed, I ended up searching the poky tree for git-submodule
$ git grep git-submodule
bitbake/lib/bb/tests/fetch.py: fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165
meta/recipes-devtools/git/git.inc: ${libexecdir}/git-core/git-submodule \
If we look in meta/recipes-devtools/git/git.inc we can see that the git-submodule is included in git-perltools:
PACKAGES =+ "${PN}-perltools".
So we can pull in the git-perltools as well on the target:
# dnf install git-perltools
and rerun the git submodule init
# git submodule update --init --recursive
Submodule 'Csocket' (https://github.com/jimloco/Csocket.git) registered for path 'third_party/Csocket'
Cloning into '/home/root/znc/third_party/Csocket'...
Submodule path 'third_party/Csocket': checked out '448e18a29ed383451db3cb648a72da4fcbb3f8e5'

It worked :), and now we can successfully type make. Having to reflash the image for each of these steps would have been much slower!
# make
Building module autocycle...
Linking module autocycle...
...
Building core object SSLVerifyHost...
Linking znc...

ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.

* I prefer emacs, so I tend to to this a lot as zile is a small emacs analogue:
$ bitbake zile
$ bitbake package-index
Target ->
# dnf install zile

== Ways to Serve up the rpm repo ==

* very simple - use python twistd.
** let's suppose you have an rpm repo on the host that is in dir /home/me/mrrepo.
** <code > cd /home/me/mrrepo;
twistd -n web --path . -p 5678
</code>
** similarly, if you are just serving up the build directories rpm's you can just do
*** <code > cd /home/me/mrrepo;
twistd -n web --path tmp/deploy/rpm -p 5678
</code>
*** Note , only 1 process can own a given port.
* less simple, but better response and better stability- use lighthttpd.
** I advocate using this from a container as it is simpler to maintain.
** <code>
docker run -it --rm --name user-rpm-server -v /home/me/mrrepo:/var/www/localhost/htdocs -v /home/me/lighttpd:/etc/lighttpd -p 5678:80 sebp/lighttpd
</code>
** What goes in /home/me/lighttpd? the config for the lighthttpd container you are running.
** Where did they come from? Copied from inside the container
*** <code> docker run -it --rm --user=root --entrypoint=/bin/sh -v /home/me/lighttpd:/frog sebp/lighttpd </code>
*** <code> # cp -a /etc/lighttpd/* /frog </code>
*** Now your /home/me/lightppd has the necessary config files.
** The files in /home/me/lighthttpd:
*** lighttpd.conf lighttpd.conf~ mime-types.conf mod_cgi.conf mod_fastcgi.conf mod_fastcgi_fpm.conf
**** From this you can see I run emacs. The only change I did was to turn on directory listing .
**** diff lighttpd.conf lighttpd.conf~
***** <code> 114c114
< dir-listing.activate = "enable"
---
> # dir-listing.activate = "enable"
</code>

** the -p 5678:80 exposes the default port the container runs it's service (aka 80) on to the host port 5678.

TipsAndTricks/RunningEclipseAgainstBuiltImage

2017-09-14T20:59:38Z

Bavery:

= Cookbook guide to Making an Eclipse Debug Capable Image =
Suppose you are building images and would like to be able to use Eclipse and the Yocto Eclipse plugin to develop/debug a C/C++ application on either a remote hardware target or on qemu. This cookbook will explain the small number of steps needed to accomplish this.
 
So that the commands are specific and can be cut and pasted to try, we will assume the following:
* Target Image-> core-image-sato-sdk
** see section[[#Why SDK Image]] for why you have to build the sdk image and how to work around it if you really don't want to.
* Target Machine -> qemux86
 
=== Making a Suitable Qemux86 Image===
* We need to build a core-image-sato-sdk that has the pieces needed by Eclipse, so we add/change the following in our conf/local.conf:
** We add the following to EXTRA_IMAGE_FEATURES in conf/local.conf
*** <code>EXTRA_IMAGE_FEATURES += " eclipse-debug "</code>
**** This adds gdbserver,tcf-agent (for Target Communication Framework), and openssh-sftp-server)
*** <code>EXTRA_IMAGE_FEATURES += " tools-sdk "</code>
**** This adds the build requirements on the target rootfs. This is needed since the Yocto Eclipse plugin is assuming that the qemu rootfs and the sysroot are synonymous.
*** <code>EXTRA_IMAGE_FEATURES += " ssh-server-openssh "</code>
**** This defaults to the openssl ssh server rather than dropbear. You can use either so this line can be omitted since the sftp server works with either.
** Execute: <code> $bitbake core-image-sato</code>
*** This makes a rootfs in tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2
* We need to build the toolchain eclipse will use
** Execute <code> $bitbake core-image-sato -c populate_sdk </code>
*** This makes sdk stuff such as the toolchains eclipse will use to build,
* We need to build a version of qemu that can run natively on our workstation as well as a userspace nfs daemon.
** Execute <code> $bitbake meta-ide-support </code>
* We need an extracted rootfs that can be used by the userspace nfs daemon to boot qemu.
** Execute <code> $mkdir MY_QEMU_ROOTFS </code>
*** Execute <code> $runqemu-extract-sdk tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2 MY_QEMU_ROOTFS</code>
**** This will result in a fully extracted rootfs in MY_QEMU_ROOTFS and a set of permissions maintained by pseudo (a yocto tool similar to but more functional than fakeroot) for the rootfs in MY_QEMU_ROOTFS.pseudo_state

=== Running Eclipse Against the Built Qemux86 Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Built Image ====
* Goto Windows->Preferences->Yocto Project SDK
** Select Build System Derived Toolchain
** enter Toolchain Root Location: <build dir>
*** this is where you have been running bitbake core-image-sato, bitbake etc.
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: <path>/tmp/deploy/images/qemux86/bzImage-qemux86.bin
*** this is the kernel we built
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Alternatively, Downloading what we need for Qemux86===
Sometimes, you may not want to build everything yourself but just want to download what you need. These instructions will target the Yocto 2.1 release, but you should be able to translate it into other releases pretty easily.
* Download the image http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/core-image-sato-sdk-qemux86.tar.bz2 Note, the image MUST be an SDK image. Images without sdk in the name will NOT work.
* Download the toolchain: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh Note, the toolchain must NOT be one with -ext in the name. The ext toolchains are for the extensible SDK which is useful for CLI and recipe development but which is not currently compatible with Eclipse.
* unpack the toolchain to a dir by running the shell script file you downloaded:
<code>$chmod a+x poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
<code>$sh poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
 
This will put the toolchain in /opt/poky/2.1 by default.
* source the toolchain environment file
<code> $source environment-setup-i586-poky-linux </code>
* extract the image you downloaded so that Qemu can use it
<code> $runqemu-extract-sdk core-image-sato-sdk-qemux86.tar.bz2 MY_QEMU_ROOTFS </code>
* Downloads a kernel Qemu can use to boot: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/bzImage-qemux86.bin

=== Running Eclipse Against the Downloaded Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Downloaded Image====
* Goto Windows->Preferences->Yocto Project SDK
** Select Standalone pre built toolchain
** enter Toolchain Root Location: /opt/poky/2.1
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: bzImage-qemux86.bin
*** this is the kernel you downloaded
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Run Qemu ===
* goto <code> Run->External Tools-> External Tools Configurations... </code> and select qemu_i586-poky-linux (in our example; in general it will be architecture-distro-os)
** click Run - this will bring up a Qemu instance.
==== Put the binary on the Qemu instance and debug it ====
* goto <code>Run->Debug Configurations...</code> and select your project (looks like hello_gdb_i586-poky-linux in our case if you named the project hello)
* make a new connection - I prefer SSH so I'll show that here. TCF should also work.
** Click New... on Connection line
** put 192.168.7.2 (probably, look at the end of the qemu xterm that launched and find ip=192.168.7.XX::192.168.7.XX-1 ) in the Host name
** name it
** click finish
===== Eclipse Oxygen and TCF =====
* In Mars, TCF would appear as a connection type alongside ssh.
* In Oxygen, this is no longer true. If you want to use tcf as the connection type you need to do the following:
** At the bottom of the DEbug Configuration page, by default, you will be in "Using GDB (DSF) Automatic Remote Debugging Launcher" - Select other...
** Click on "Select other..." and you can choose GDB (DSF) Automatic REmove Debugging via TCF/TE Launcher. This is the oxygen equivalent of a TCF connection.
** If they cannot be selected, you may need to select the "Use configuration specific settings" checkbox at the top of the dialog box.
*** at this point, your connection will be of type tcf
** I found the neon implementation of tcf to be buggy...
===== Back to Using the Connection whether it be ssh or tcf =====
* Browse to where the prog will go. I tend to use /home/root.
** Remote Absolute File Path for C/C++ Application -> click Browse...
** Goto Root->/->home->root remember the user is root and the password is empty by default.
** double click root
** click ok
** If the browse to Remote Absolute File Path for C/C++ Application hangs or errors out Check
*** qemu is running
*** you can ssh root@<address> from a terminal window
*** make sure that the proxies eclipse knows about in Window->Preferences->Network Connections is not "eating" the ssh connection traffic. On Oxygen, for instance, I need to switch the network connection to manual and add 192.168.7.2 as a Proxy bypass host.
* click Apply
* click Debug and open the Debug Perspective.

== Other Useful Info ==
=== Why SDK Image ===
The current poky-eclipse plugin mashes together 2 concepts that should have been disjoint: namely, the target sysroot and the rootfs image qemu will boot. Under <code> Window->Preferences->Yocto Project SDK </code>, you choose a sysroot location. Up above I told you to point this to the MY_QEMU_ROOTFS you extracted with runqemu-extract-sdk. This will work, but only if you have built the sdk image. If you built the core-image-sato instead, then the MY_QEMU_ROOTFS will be missing a lot of important development files like stdio.h...
==== How to Work around this ====
Let's say you build core-image-sato (this is a standin for your particular image that can't have the sdk parts tossed in for some reason). You can still use eclipse and qemu. You need to do the same things as above with the following changes:
* <code>Sysroot in Window->Preferences->Yocto Project SDK </code> points to <builddir>/tmp/sysroots/qemux86-64. This will give the build system access to the development files it needs.
* In the dialog for <code> Run->External Tools-> External Tools Configurations... </code> you will need to read the long Argument string in the big box. It will have something like <code> -e "source env-setup-<sdkarch>; runqemu <kernel path> <rootfs path> </code> Because the current plugin munges the sysroot and qemu image concepts, the <rootfs path> will incorrectly be pointing to your sysroot. Just delete the <rootfs path> and put in the ABSOLUTE path to the MY_QEMU_ROOTFS you extracted with the runqemu-extract-sdk tool. This way, you can build & debug the image even if it's not an sdk image.

=== Adding static tap devices ===
It is often easier to preconfigure several tap devices so that qemu can run and use them without needing to be root or use sudo. To do this, run <code> runqemu-gen-tapdevs `id -u <username>` `id -g <username>` 4 `find ./tmp/sysroots -iname tunctl`</code> as root. This will make the inet devices tap0,tap1,tap2,tap3. You can see them with <code> ifconfig -a | grep tap </code>. Typically, tap0 is 192.168.7.1 and so when the first qemu is run it's address will be 192.168.7.2 and 192.168.7.1 will be that qemu instance's gateway. The runqemu-gen-tapdevs is located in bitbake/scripts.

scripts/runqemu-gen-tapdevs

=== Switching autotools C/C++ project from one toolchain/sysroot to another and build failing===
Sometimes you will have an existing project and want to switch to a different toolchain or sysroot or machine (x86->x86-64 for instance). If you find that you are getting errors like <code>install-sh not found</code> it may mean that your autotools files have gotten out of sync. Here's how you can force them back. All of these are found by Right clicking on the project and going to Invoke Autotools->:
<code> Invoke Libtoolize option = "--force"</code>
<code> Invoke Aclocal</code>
<code> Invoke Autoheader</code>
<code> Invoke Automake option = "--force-missing --add-missing"</code>
<code> Invoke Autoconf</code>
Then Right click on the project and select <code> Reconfigure Project </code>. You should now be able to build the project again.

=== Dealing with java.lang.NullPointerException ===
Eclipse upstream has some issues with *some* of the GTK3 libraries shipped with some of the distributions(https://bugs.eclipse.org/bugs/show_bug.cgi?id=430736). If you are seeing a NullPointer Exception try adding the following to your eclipse.ini file:
<code>
--launcher.GTK_version
2
</code>

Here's an in place snippet of a happy ini file:
<code>
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
--launcher.GTK_version
2
--launcher.appendVmargs
-vmargs
</code>
Alternatively, you can set
<code>
$export SWT_GTK3=0
</code>
But this might affect other programs as well.

=== XWayland workaround with <code>gtk3</code> (Fedora-25) ===

If you see bizarre screen layout of controls in Eclipse Neon, it is probably because the Neon (4.6.x) release of Eclipse is not fully compatible with straight Wayland.

Fedora-25 now has Wayland by default. To enable the XWayland backend:

<code>export GDK_BACKEND=x11</code>

This will still use gtk3, but will use the XWayland bridge rather than straight Wayland. This is expected to be fixed in the Oxygen (4.7) release of Eclipse to support straight Wayland.(https://bugs.eclipse.org/bugs/show_bug.cgi?id=496923)

Thanks to Roland Grunber (Red Hat Linux Tools team) for pointing me in the right direction on #fedora-linux (freenode IRC).

[Source] https://coffeeorientedprogramming.com/2016/10/06/make-applications-eclipse-use-x11-backend-on-wayland-fedora-25/#more-502

==== How to know what version of <code>gtk</code> is being used in your session? ====
<code>Help -> About -> Installation Details -> Configuration Tab</code>

Look for something like:

<code>org.eclipse.swt.internal.gtk.version=3.22.6</code>

[Source] https://coffeeorientedprogramming.com/2014/10/27/how-to-tell-if-you-are-running-eclipse-on-gtk2-or-on-gtk3/

==== How do you know you are running Wayland? ====
<code>loginctl</code> which should show you a list of sessions, one of which will be <code>gdm</code>, remember that session number.

<code>loginctl show-session <session number from above> -p Type</code>

Result should be:

<code>Type=wayland</code>

[Source] http://unix.stackexchange.com/questions/202891/how-to-know-whether-wayland-or-x11-is-being-used

TipsAndTricks/RunningEclipseAgainstBuiltImage

2017-09-14T20:21:42Z

Bavery:

= Cookbook guide to Making an Eclipse Debug Capable Image =
Suppose you are building images and would like to be able to use Eclipse and the Yocto Eclipse plugin to develop/debug a C/C++ application on either a remote hardware target or on qemu. This cookbook will explain the small number of steps needed to accomplish this.
 
So that the commands are specific and can be cut and pasted to try, we will assume the following:
* Target Image-> core-image-sato-sdk
** see section[[#Why SDK Image]] for why you have to build the sdk image and how to work around it if you really don't want to.
* Target Machine -> qemux86
 
=== Making a Suitable Qemux86 Image===
* We need to build a core-image-sato-sdk that has the pieces needed by Eclipse, so we add/change the following in our conf/local.conf:
** We add the following to EXTRA_IMAGE_FEATURES in conf/local.conf
*** <code>EXTRA_IMAGE_FEATURES += " eclipse-debug "</code>
**** This adds gdbserver,tcf-agent (for Target Communication Framework), and openssh-sftp-server)
*** <code>EXTRA_IMAGE_FEATURES += " tools-sdk "</code>
**** This adds the build requirements on the target rootfs. This is needed since the Yocto Eclipse plugin is assuming that the qemu rootfs and the sysroot are synonymous.
*** <code>EXTRA_IMAGE_FEATURES += " ssh-server-openssh "</code>
**** This defaults to the openssl ssh server rather than dropbear. You can use either so this line can be omitted since the sftp server works with either.
** Execute: <code> $bitbake core-image-sato</code>
*** This makes a rootfs in tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2
* We need to build the toolchain eclipse will use
** Execute <code> $bitbake core-image-sato -c populate_sdk </code>
*** This makes sdk stuff such as the toolchains eclipse will use to build,
* We need to build a version of qemu that can run natively on our workstation as well as a userspace nfs daemon.
** Execute <code> $bitbake meta-ide-support </code>
* We need an extracted rootfs that can be used by the userspace nfs daemon to boot qemu.
** Execute <code> $mkdir MY_QEMU_ROOTFS </code>
*** Execute <code> $runqemu-extract-sdk tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2 MY_QEMU_ROOTFS</code>
**** This will result in a fully extracted rootfs in MY_QEMU_ROOTFS and a set of permissions maintained by pseudo (a yocto tool similar to but more functional than fakeroot) for the rootfs in MY_QEMU_ROOTFS.pseudo_state

=== Running Eclipse Against the Built Qemux86 Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Built Image ====
* Goto Windows->Preferences->Yocto Project SDK
** Select Build System Derived Toolchain
** enter Toolchain Root Location: <build dir>
*** this is where you have been running bitbake core-image-sato, bitbake etc.
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: <path>/tmp/deploy/images/qemux86/bzImage-qemux86.bin
*** this is the kernel we built
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Alternatively, Downloading what we need for Qemux86===
Sometimes, you may not want to build everything yourself but just want to download what you need. These instructions will target the Yocto 2.1 release, but you should be able to translate it into other releases pretty easily.
* Download the image http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/core-image-sato-sdk-qemux86.tar.bz2 Note, the image MUST be an SDK image. Images without sdk in the name will NOT work.
* Download the toolchain: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh Note, the toolchain must NOT be one with -ext in the name. The ext toolchains are for the extensible SDK which is useful for CLI and recipe development but which is not currently compatible with Eclipse.
* unpack the toolchain to a dir by running the shell script file you downloaded:
<code>$chmod a+x poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
<code>$sh poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
 
This will put the toolchain in /opt/poky/2.1 by default.
* source the toolchain environment file
<code> $source environment-setup-i586-poky-linux </code>
* extract the image you downloaded so that Qemu can use it
<code> $runqemu-extract-sdk core-image-sato-sdk-qemux86.tar.bz2 MY_QEMU_ROOTFS </code>
* Downloads a kernel Qemu can use to boot: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/bzImage-qemux86.bin

=== Running Eclipse Against the Downloaded Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Downloaded Image====
* Goto Windows->Preferences->Yocto Project SDK
** Select Standalone pre built toolchain
** enter Toolchain Root Location: /opt/poky/2.1
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: bzImage-qemux86.bin
*** this is the kernel you downloaded
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Run Qemu ===
* goto <code> Run->External Tools-> External Tools Configurations... </code> and select qemu_i586-poky-linux (in our example; in general it will be architecture-distro-os)
** click Run - this will bring up a Qemu instance.
==== Put the binary on the Qemu instance and debug it ====
* goto <code>Run->Debug Configurations...</code> and select your project (looks like hello_gdb_i586-poky-linux in our case if you named the project hello)
* make a new connection - I prefer SSH so I'll show that here. TCF should also work.
** Click New... on Connection line
** put 192.168.7.2 (probably, look at the end of the qemu xterm that launched and find ip=192.168.7.XX::192.168.7.XX-1 ) in the Host name
** name it
** click finish
===== Eclipse Oxygen and TCF =====
* In Mars, TCF would appear as a connection type alongside ssh.
* In Oxygen, this is no longer true. If you want to use tcf as the connection type you need to do the following:
** At the bottom of the DEbug Configuration page, by default, you will be in "Using GDB (DSF) Automatic Remote Debugging Launcher" - Select other...
** Click on "Select other..." and you can choose GDB (DSF) Automatic REmove Debugging via TCF/TE Launcher. This is the oxygen equivalent of a TCF connection.
*** at this point, your connection will be of type tcf
===== Back to Using the Connection whether it be ssh or tcf =====
* Browse to where the prog will go. I tend to use /home/root.
** Remote Absolute File Path for C/C++ Application -> click Browse...
** Goto Root->/->home->root remember the user is root and the password is empty by default.
** double click root
** click ok
** If the browse to Remote Absolute File Path for C/C++ Application hangs or errors out Check
*** qemu is running
*** you can ssh root@<address> from a terminal window
*** make sure that the proxies eclipse knows about in Window->Preferences->Network Connections is not "eating" the ssh connection traffic. On Oxygen, for instance, I need to switch the network connection to manual and add 192.168.7.2 as a Proxy bypass host.
* click Apply
* click Debug and open the Debug Perspective.

== Other Useful Info ==
=== Why SDK Image ===
The current poky-eclipse plugin mashes together 2 concepts that should have been disjoint: namely, the target sysroot and the rootfs image qemu will boot. Under <code> Window->Preferences->Yocto Project SDK </code>, you choose a sysroot location. Up above I told you to point this to the MY_QEMU_ROOTFS you extracted with runqemu-extract-sdk. This will work, but only if you have built the sdk image. If you built the core-image-sato instead, then the MY_QEMU_ROOTFS will be missing a lot of important development files like stdio.h...
==== How to Work around this ====
Let's say you build core-image-sato (this is a standin for your particular image that can't have the sdk parts tossed in for some reason). You can still use eclipse and qemu. You need to do the same things as above with the following changes:
* <code>Sysroot in Window->Preferences->Yocto Project SDK </code> points to <builddir>/tmp/sysroots/qemux86-64. This will give the build system access to the development files it needs.
* In the dialog for <code> Run->External Tools-> External Tools Configurations... </code> you will need to read the long Argument string in the big box. It will have something like <code> -e "source env-setup-<sdkarch>; runqemu <kernel path> <rootfs path> </code> Because the current plugin munges the sysroot and qemu image concepts, the <rootfs path> will incorrectly be pointing to your sysroot. Just delete the <rootfs path> and put in the ABSOLUTE path to the MY_QEMU_ROOTFS you extracted with the runqemu-extract-sdk tool. This way, you can build & debug the image even if it's not an sdk image.

=== Adding static tap devices ===
It is often easier to preconfigure several tap devices so that qemu can run and use them without needing to be root or use sudo. To do this, run <code> runqemu-gen-tapdevs `id -u <username>` `id -g <username>` 4 `find ./tmp/sysroots -iname tunctl`</code> as root. This will make the inet devices tap0,tap1,tap2,tap3. You can see them with <code> ifconfig -a | grep tap </code>. Typically, tap0 is 192.168.7.1 and so when the first qemu is run it's address will be 192.168.7.2 and 192.168.7.1 will be that qemu instance's gateway. The runqemu-gen-tapdevs is located in bitbake/scripts.

scripts/runqemu-gen-tapdevs

=== Switching autotools C/C++ project from one toolchain/sysroot to another and build failing===
Sometimes you will have an existing project and want to switch to a different toolchain or sysroot or machine (x86->x86-64 for instance). If you find that you are getting errors like <code>install-sh not found</code> it may mean that your autotools files have gotten out of sync. Here's how you can force them back. All of these are found by Right clicking on the project and going to Invoke Autotools->:
<code> Invoke Libtoolize option = "--force"</code>
<code> Invoke Aclocal</code>
<code> Invoke Autoheader</code>
<code> Invoke Automake option = "--force-missing --add-missing"</code>
<code> Invoke Autoconf</code>
Then Right click on the project and select <code> Reconfigure Project </code>. You should now be able to build the project again.

=== Dealing with java.lang.NullPointerException ===
Eclipse upstream has some issues with *some* of the GTK3 libraries shipped with some of the distributions(https://bugs.eclipse.org/bugs/show_bug.cgi?id=430736). If you are seeing a NullPointer Exception try adding the following to your eclipse.ini file:
<code>
--launcher.GTK_version
2
</code>

Here's an in place snippet of a happy ini file:
<code>
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
--launcher.GTK_version
2
--launcher.appendVmargs
-vmargs
</code>
Alternatively, you can set
<code>
$export SWT_GTK3=0
</code>
But this might affect other programs as well.

=== XWayland workaround with <code>gtk3</code> (Fedora-25) ===

If you see bizarre screen layout of controls in Eclipse Neon, it is probably because the Neon (4.6.x) release of Eclipse is not fully compatible with straight Wayland.

Fedora-25 now has Wayland by default. To enable the XWayland backend:

<code>export GDK_BACKEND=x11</code>

This will still use gtk3, but will use the XWayland bridge rather than straight Wayland. This is expected to be fixed in the Oxygen (4.7) release of Eclipse to support straight Wayland.(https://bugs.eclipse.org/bugs/show_bug.cgi?id=496923)

Thanks to Roland Grunber (Red Hat Linux Tools team) for pointing me in the right direction on #fedora-linux (freenode IRC).

[Source] https://coffeeorientedprogramming.com/2016/10/06/make-applications-eclipse-use-x11-backend-on-wayland-fedora-25/#more-502

==== How to know what version of <code>gtk</code> is being used in your session? ====
<code>Help -> About -> Installation Details -> Configuration Tab</code>

Look for something like:

<code>org.eclipse.swt.internal.gtk.version=3.22.6</code>

[Source] https://coffeeorientedprogramming.com/2014/10/27/how-to-tell-if-you-are-running-eclipse-on-gtk2-or-on-gtk3/

==== How do you know you are running Wayland? ====
<code>loginctl</code> which should show you a list of sessions, one of which will be <code>gdm</code>, remember that session number.

<code>loginctl show-session <session number from above> -p Type</code>

Result should be:

<code>Type=wayland</code>

[Source] http://unix.stackexchange.com/questions/202891/how-to-know-whether-wayland-or-x11-is-being-used

TipsAndTricks/RunningEclipseAgainstBuiltImage

2017-09-14T20:21:04Z

Bavery: /* Put the binary on the Qemu instance and debug it */

= Cookbook guide to Making an Eclipse Debug Capable Image =
Suppose you are building images and would like to be able to use Eclipse and the Yocto Eclipse plugin to develop/debug a C/C++ application on either a remote hardware target or on qemu. This cookbook will explain the small number of steps needed to accomplish this.
 
So that the commands are specific and can be cut and pasted to try, we will assume the following:
* Target Image-> core-image-sato-sdk
** see section[[#Why SDK Image]] for why you have to build the sdk image and how to work around it if you really don't want to.
* Target Machine -> qemux86
 
=== Making a Suitable Qemux86 Image===
* We need to build a core-image-sato-sdk that has the pieces needed by Eclipse, so we add/change the following in our conf/local.conf:
** We add the following to EXTRA_IMAGE_FEATURES in conf/local.conf
*** <code>EXTRA_IMAGE_FEATURES += " eclipse-debug "</code>
**** This adds gdbserver,tcf-agent (for Target Communication Framework), and openssh-sftp-server)
*** <code>EXTRA_IMAGE_FEATURES += " tools-sdk "</code>
**** This adds the build requirements on the target rootfs. This is needed since the Yocto Eclipse plugin is assuming that the qemu rootfs and the sysroot are synonymous.
*** <code>EXTRA_IMAGE_FEATURES += " ssh-server-openssh "</code>
**** This defaults to the openssl ssh server rather than dropbear. You can use either so this line can be omitted since the sftp server works with either.
** Execute: <code> $bitbake core-image-sato</code>
*** This makes a rootfs in tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2
* We need to build the toolchain eclipse will use
** Execute <code> $bitbake core-image-sato -c populate_sdk </code>
*** This makes sdk stuff such as the toolchains eclipse will use to build,
* We need to build a version of qemu that can run natively on our workstation as well as a userspace nfs daemon.
** Execute <code> $bitbake meta-ide-support </code>
* We need an extracted rootfs that can be used by the userspace nfs daemon to boot qemu.
** Execute <code> $mkdir MY_QEMU_ROOTFS </code>
*** Execute <code> $runqemu-extract-sdk tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2 MY_QEMU_ROOTFS</code>
**** This will result in a fully extracted rootfs in MY_QEMU_ROOTFS and a set of permissions maintained by pseudo (a yocto tool similar to but more functional than fakeroot) for the rootfs in MY_QEMU_ROOTFS.pseudo_state

=== Running Eclipse Against the Built Qemux86 Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Built Image ====
* Goto Windows->Preferences->Yocto Project SDK
** Select Build System Derived Toolchain
** enter Toolchain Root Location: <build dir>
*** this is where you have been running bitbake core-image-sato, bitbake etc.
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: <path>/tmp/deploy/images/qemux86/bzImage-qemux86.bin
*** this is the kernel we built
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Alternatively, Downloading what we need for Qemux86===
Sometimes, you may not want to build everything yourself but just want to download what you need. These instructions will target the Yocto 2.1 release, but you should be able to translate it into other releases pretty easily.
* Download the image http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/core-image-sato-sdk-qemux86.tar.bz2 Note, the image MUST be an SDK image. Images without sdk in the name will NOT work.
* Download the toolchain: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh Note, the toolchain must NOT be one with -ext in the name. The ext toolchains are for the extensible SDK which is useful for CLI and recipe development but which is not currently compatible with Eclipse.
* unpack the toolchain to a dir by running the shell script file you downloaded:
<code>$chmod a+x poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
<code>$sh poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
 
This will put the toolchain in /opt/poky/2.1 by default.
* source the toolchain environment file
<code> $source environment-setup-i586-poky-linux </code>
* extract the image you downloaded so that Qemu can use it
<code> $runqemu-extract-sdk core-image-sato-sdk-qemux86.tar.bz2 MY_QEMU_ROOTFS </code>
* Downloads a kernel Qemu can use to boot: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/bzImage-qemux86.bin

=== Running Eclipse Against the Downloaded Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Downloaded Image====
* Goto Windows->Preferences->Yocto Project SDK
** Select Standalone pre built toolchain
** enter Toolchain Root Location: /opt/poky/2.1
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: bzImage-qemux86.bin
*** this is the kernel you downloaded
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Run Qemu ===
* goto <code> Run->External Tools-> External Tools Configurations... </code> and select qemu_i586-poky-linux (in our example; in general it will be architecture-distro-os)
** click Run - this will bring up a Qemu instance.
==== Put the binary on the Qemu instance and debug it ====
* goto <code>Run->Debug Configurations...</code> and select your project (looks like hello_gdb_i586-poky-linux in our case if you named the project hello)
* make a new connection - I prefer SSH so I'll show that here. TCF should also work.
** Click New... on Connection line
** put 192.168.7.2 (probably, look at the end of the qemu xterm that launched and find ip=192.168.7.XX::192.168.7.XX-1 ) in the Host name
** name it
** click finish
===== Eclipse Oxygen and TCF =====
* In Mars, TCF would appear as a connection type alongside ssh.
* In Oxygen, this is no longer true. If you want to use tcf as the connection type you need to do the following:
** At the bottom of the DEbug Configuration page, by default, you will be in "Using GDB (DSF) Automatic Remote Debugging Launcher" - Select other...
** Click on "Select other..." and you can choose GDB (DSF) Automatic REmove Debugging via TCF/TE Launcher. This is the oxygen equivalent of a TCF connection.
*** at this point, your connection will be of type tcf
* Browse to where the prog will go. I tend to use /home/root.
** Remote Absolute File Path for C/C++ Application -> click Browse...
** Goto Root->/->home->root remember the user is root and the password is empty by default.
** double click root
** click ok
** If the browse to Remote Absolute File Path for C/C++ Application hangs or errors out Check
*** qemu is running
*** you can ssh root@<address> from a terminal window
*** make sure that the proxies eclipse knows about in Window->Preferences->Network Connections is not "eating" the ssh connection traffic. On Oxygen, for instance, I need to switch the network connection to manual and add 192.168.7.2 as a Proxy bypass host.
* click Apply
* click Debug and open the Debug Perspective.

== Other Useful Info ==
=== Why SDK Image ===
The current poky-eclipse plugin mashes together 2 concepts that should have been disjoint: namely, the target sysroot and the rootfs image qemu will boot. Under <code> Window->Preferences->Yocto Project SDK </code>, you choose a sysroot location. Up above I told you to point this to the MY_QEMU_ROOTFS you extracted with runqemu-extract-sdk. This will work, but only if you have built the sdk image. If you built the core-image-sato instead, then the MY_QEMU_ROOTFS will be missing a lot of important development files like stdio.h...
==== How to Work around this ====
Let's say you build core-image-sato (this is a standin for your particular image that can't have the sdk parts tossed in for some reason). You can still use eclipse and qemu. You need to do the same things as above with the following changes:
* <code>Sysroot in Window->Preferences->Yocto Project SDK </code> points to <builddir>/tmp/sysroots/qemux86-64. This will give the build system access to the development files it needs.
* In the dialog for <code> Run->External Tools-> External Tools Configurations... </code> you will need to read the long Argument string in the big box. It will have something like <code> -e "source env-setup-<sdkarch>; runqemu <kernel path> <rootfs path> </code> Because the current plugin munges the sysroot and qemu image concepts, the <rootfs path> will incorrectly be pointing to your sysroot. Just delete the <rootfs path> and put in the ABSOLUTE path to the MY_QEMU_ROOTFS you extracted with the runqemu-extract-sdk tool. This way, you can build & debug the image even if it's not an sdk image.

=== Adding static tap devices ===
It is often easier to preconfigure several tap devices so that qemu can run and use them without needing to be root or use sudo. To do this, run <code> runqemu-gen-tapdevs `id -u <username>` `id -g <username>` 4 `find ./tmp/sysroots -iname tunctl`</code> as root. This will make the inet devices tap0,tap1,tap2,tap3. You can see them with <code> ifconfig -a | grep tap </code>. Typically, tap0 is 192.168.7.1 and so when the first qemu is run it's address will be 192.168.7.2 and 192.168.7.1 will be that qemu instance's gateway. The runqemu-gen-tapdevs is located in bitbake/scripts.

scripts/runqemu-gen-tapdevs

=== Switching autotools C/C++ project from one toolchain/sysroot to another and build failing===
Sometimes you will have an existing project and want to switch to a different toolchain or sysroot or machine (x86->x86-64 for instance). If you find that you are getting errors like <code>install-sh not found</code> it may mean that your autotools files have gotten out of sync. Here's how you can force them back. All of these are found by Right clicking on the project and going to Invoke Autotools->:
<code> Invoke Libtoolize option = "--force"</code>
<code> Invoke Aclocal</code>
<code> Invoke Autoheader</code>
<code> Invoke Automake option = "--force-missing --add-missing"</code>
<code> Invoke Autoconf</code>
Then Right click on the project and select <code> Reconfigure Project </code>. You should now be able to build the project again.

=== Dealing with java.lang.NullPointerException ===
Eclipse upstream has some issues with *some* of the GTK3 libraries shipped with some of the distributions(https://bugs.eclipse.org/bugs/show_bug.cgi?id=430736). If you are seeing a NullPointer Exception try adding the following to your eclipse.ini file:
<code>
--launcher.GTK_version
2
</code>

Here's an in place snippet of a happy ini file:
<code>
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
--launcher.GTK_version
2
--launcher.appendVmargs
-vmargs
</code>
Alternatively, you can set
<code>
$export SWT_GTK3=0
</code>
But this might affect other programs as well.

=== XWayland workaround with <code>gtk3</code> (Fedora-25) ===

If you see bizarre screen layout of controls in Eclipse Neon, it is probably because the Neon (4.6.x) release of Eclipse is not fully compatible with straight Wayland.

Fedora-25 now has Wayland by default. To enable the XWayland backend:

<code>export GDK_BACKEND=x11</code>

This will still use gtk3, but will use the XWayland bridge rather than straight Wayland. This is expected to be fixed in the Oxygen (4.7) release of Eclipse to support straight Wayland.(https://bugs.eclipse.org/bugs/show_bug.cgi?id=496923)

Thanks to Roland Grunber (Red Hat Linux Tools team) for pointing me in the right direction on #fedora-linux (freenode IRC).

[Source] https://coffeeorientedprogramming.com/2016/10/06/make-applications-eclipse-use-x11-backend-on-wayland-fedora-25/#more-502

==== How to know what version of <code>gtk</code> is being used in your session? ====
<code>Help -> About -> Installation Details -> Configuration Tab</code>

Look for something like:

<code>org.eclipse.swt.internal.gtk.version=3.22.6</code>

[Source] https://coffeeorientedprogramming.com/2014/10/27/how-to-tell-if-you-are-running-eclipse-on-gtk2-or-on-gtk3/

==== How do you know you are running Wayland? ====
<code>loginctl</code> which should show you a list of sessions, one of which will be <code>gdm</code>, remember that session number.

<code>loginctl show-session <session number from above> -p Type</code>

Result should be:

<code>Type=wayland</code>

[Source] http://unix.stackexchange.com/questions/202891/how-to-know-whether-wayland-or-x11-is-being-used

TipsAndTricks/RunningEclipseAgainstBuiltImage

2017-09-13T23:19:52Z

Bavery:

= Cookbook guide to Making an Eclipse Debug Capable Image =
Suppose you are building images and would like to be able to use Eclipse and the Yocto Eclipse plugin to develop/debug a C/C++ application on either a remote hardware target or on qemu. This cookbook will explain the small number of steps needed to accomplish this.
 
So that the commands are specific and can be cut and pasted to try, we will assume the following:
* Target Image-> core-image-sato-sdk
** see section[[#Why SDK Image]] for why you have to build the sdk image and how to work around it if you really don't want to.
* Target Machine -> qemux86
 
=== Making a Suitable Qemux86 Image===
* We need to build a core-image-sato-sdk that has the pieces needed by Eclipse, so we add/change the following in our conf/local.conf:
** We add the following to EXTRA_IMAGE_FEATURES in conf/local.conf
*** <code>EXTRA_IMAGE_FEATURES += " eclipse-debug "</code>
**** This adds gdbserver,tcf-agent (for Target Communication Framework), and openssh-sftp-server)
*** <code>EXTRA_IMAGE_FEATURES += " tools-sdk "</code>
**** This adds the build requirements on the target rootfs. This is needed since the Yocto Eclipse plugin is assuming that the qemu rootfs and the sysroot are synonymous.
*** <code>EXTRA_IMAGE_FEATURES += " ssh-server-openssh "</code>
**** This defaults to the openssl ssh server rather than dropbear. You can use either so this line can be omitted since the sftp server works with either.
** Execute: <code> $bitbake core-image-sato</code>
*** This makes a rootfs in tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2
* We need to build the toolchain eclipse will use
** Execute <code> $bitbake core-image-sato -c populate_sdk </code>
*** This makes sdk stuff such as the toolchains eclipse will use to build,
* We need to build a version of qemu that can run natively on our workstation as well as a userspace nfs daemon.
** Execute <code> $bitbake meta-ide-support </code>
* We need an extracted rootfs that can be used by the userspace nfs daemon to boot qemu.
** Execute <code> $mkdir MY_QEMU_ROOTFS </code>
*** Execute <code> $runqemu-extract-sdk tmp/deploy/images/qemux86/core-image-sato-qemux86.tar.bz2 MY_QEMU_ROOTFS</code>
**** This will result in a fully extracted rootfs in MY_QEMU_ROOTFS and a set of permissions maintained by pseudo (a yocto tool similar to but more functional than fakeroot) for the rootfs in MY_QEMU_ROOTFS.pseudo_state

=== Running Eclipse Against the Built Qemux86 Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Built Image ====
* Goto Windows->Preferences->Yocto Project SDK
** Select Build System Derived Toolchain
** enter Toolchain Root Location: <build dir>
*** this is where you have been running bitbake core-image-sato, bitbake etc.
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: <path>/tmp/deploy/images/qemux86/bzImage-qemux86.bin
*** this is the kernel we built
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Alternatively, Downloading what we need for Qemux86===
Sometimes, you may not want to build everything yourself but just want to download what you need. These instructions will target the Yocto 2.1 release, but you should be able to translate it into other releases pretty easily.
* Download the image http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/core-image-sato-sdk-qemux86.tar.bz2 Note, the image MUST be an SDK image. Images without sdk in the name will NOT work.
* Download the toolchain: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh Note, the toolchain must NOT be one with -ext in the name. The ext toolchains are for the extensible SDK which is useful for CLI and recipe development but which is not currently compatible with Eclipse.
* unpack the toolchain to a dir by running the shell script file you downloaded:
<code>$chmod a+x poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
<code>$sh poky-glibc-x86_64-core-image-sato-i586-toolchain-2.1.sh </code>
 
This will put the toolchain in /opt/poky/2.1 by default.
* source the toolchain environment file
<code> $source environment-setup-i586-poky-linux </code>
* extract the image you downloaded so that Qemu can use it
<code> $runqemu-extract-sdk core-image-sato-sdk-qemux86.tar.bz2 MY_QEMU_ROOTFS </code>
* Downloads a kernel Qemu can use to boot: http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/machines/qemu/qemux86/bzImage-qemux86.bin

=== Running Eclipse Against the Downloaded Image ===
This Cookbook assumes you have already installed the Eclipse Poky plugin following the directions in http://www.yoctoproject.org/docs/latest/sdk-manual/sdk-manual.html. Assuming that is true, you can
==== Set up the Configuration For the Downloaded Image====
* Goto Windows->Preferences->Yocto Project SDK
** Select Standalone pre built toolchain
** enter Toolchain Root Location: /opt/poky/2.1
** enter Sysroot Location: <path>/MY_QEMU_ROOTFS
*** this is where you extracted the core-image-sato-qemux86.tar.bz2 to by running runqemu-extract-sdk
** enter Qemu/Kernel: bzImage-qemux86.bin
*** this is the kernel you downloaded
** You should now see i586-poky-linux in the Target Architecture dropdown list.
** hit apply and ok

=== Run Qemu ===
* goto <code> Run->External Tools-> External Tools Configurations... </code> and select qemu_i586-poky-linux (in our example; in general it will be architecture-distro-os)
** click Run - this will bring up a Qemu instance.
==== Put the binary on the Qemu instance and debug it ====
* goto <code>Run->Debug Configurations...</code> and select your project (looks like hello_gdb_i586-poky-linux in our case if you named the project hello)
* make a new connection - I prefer SSH so I'll show that here. TCF should also work.
** Click New... on Connection line
** put 192.168.7.2 (probably, look at the end of the qemu xterm that launched and find ip=192.168.7.XX::192.168.7.XX-1 ) in the Host name
** name it
** click finish
* Browse to where the prog will go. I tend to use /home/root.
** Remote Absolute File Path for C/C++ Application -> click Browse...
** Goto Root->/->home->root remember the user is root and the password is empty by default.
** double click root
** click ok
** If the browse to Remote Absolute File Path for C/C++ Application hangs or errors out Check
*** qemu is running
*** you can ssh root@<address> from a terminal window
*** make sure that the proxies eclipse knows about in Window->Preferences->Network Connections is not "eating" the ssh connection traffic. On Oxygen, for instance, I need to switch the network connection to manual and add 192.168.7.2 as a Proxy bypass host.
* click Apply
* click Debug and open the Debug Perspective.

== Other Useful Info ==
=== Why SDK Image ===
The current poky-eclipse plugin mashes together 2 concepts that should have been disjoint: namely, the target sysroot and the rootfs image qemu will boot. Under <code> Window->Preferences->Yocto Project SDK </code>, you choose a sysroot location. Up above I told you to point this to the MY_QEMU_ROOTFS you extracted with runqemu-extract-sdk. This will work, but only if you have built the sdk image. If you built the core-image-sato instead, then the MY_QEMU_ROOTFS will be missing a lot of important development files like stdio.h...
==== How to Work around this ====
Let's say you build core-image-sato (this is a standin for your particular image that can't have the sdk parts tossed in for some reason). You can still use eclipse and qemu. You need to do the same things as above with the following changes:
* <code>Sysroot in Window->Preferences->Yocto Project SDK </code> points to <builddir>/tmp/sysroots/qemux86-64. This will give the build system access to the development files it needs.
* In the dialog for <code> Run->External Tools-> External Tools Configurations... </code> you will need to read the long Argument string in the big box. It will have something like <code> -e "source env-setup-<sdkarch>; runqemu <kernel path> <rootfs path> </code> Because the current plugin munges the sysroot and qemu image concepts, the <rootfs path> will incorrectly be pointing to your sysroot. Just delete the <rootfs path> and put in the ABSOLUTE path to the MY_QEMU_ROOTFS you extracted with the runqemu-extract-sdk tool. This way, you can build & debug the image even if it's not an sdk image.

=== Adding static tap devices ===
It is often easier to preconfigure several tap devices so that qemu can run and use them without needing to be root or use sudo. To do this, run <code> runqemu-gen-tapdevs `id -u <username>` `id -g <username>` 4 `find ./tmp/sysroots -iname tunctl`</code> as root. This will make the inet devices tap0,tap1,tap2,tap3. You can see them with <code> ifconfig -a | grep tap </code>. Typically, tap0 is 192.168.7.1 and so when the first qemu is run it's address will be 192.168.7.2 and 192.168.7.1 will be that qemu instance's gateway. The runqemu-gen-tapdevs is located in bitbake/scripts.

scripts/runqemu-gen-tapdevs

=== Switching autotools C/C++ project from one toolchain/sysroot to another and build failing===
Sometimes you will have an existing project and want to switch to a different toolchain or sysroot or machine (x86->x86-64 for instance). If you find that you are getting errors like <code>install-sh not found</code> it may mean that your autotools files have gotten out of sync. Here's how you can force them back. All of these are found by Right clicking on the project and going to Invoke Autotools->:
<code> Invoke Libtoolize option = "--force"</code>
<code> Invoke Aclocal</code>
<code> Invoke Autoheader</code>
<code> Invoke Automake option = "--force-missing --add-missing"</code>
<code> Invoke Autoconf</code>
Then Right click on the project and select <code> Reconfigure Project </code>. You should now be able to build the project again.

=== Dealing with java.lang.NullPointerException ===
Eclipse upstream has some issues with *some* of the GTK3 libraries shipped with some of the distributions(https://bugs.eclipse.org/bugs/show_bug.cgi?id=430736). If you are seeing a NullPointer Exception try adding the following to your eclipse.ini file:
<code>
--launcher.GTK_version
2
</code>

Here's an in place snippet of a happy ini file:
<code>
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
--launcher.GTK_version
2
--launcher.appendVmargs
-vmargs
</code>
Alternatively, you can set
<code>
$export SWT_GTK3=0
</code>
But this might affect other programs as well.

=== XWayland workaround with <code>gtk3</code> (Fedora-25) ===

If you see bizarre screen layout of controls in Eclipse Neon, it is probably because the Neon (4.6.x) release of Eclipse is not fully compatible with straight Wayland.

Fedora-25 now has Wayland by default. To enable the XWayland backend:

<code>export GDK_BACKEND=x11</code>

This will still use gtk3, but will use the XWayland bridge rather than straight Wayland. This is expected to be fixed in the Oxygen (4.7) release of Eclipse to support straight Wayland.(https://bugs.eclipse.org/bugs/show_bug.cgi?id=496923)

Thanks to Roland Grunber (Red Hat Linux Tools team) for pointing me in the right direction on #fedora-linux (freenode IRC).

[Source] https://coffeeorientedprogramming.com/2016/10/06/make-applications-eclipse-use-x11-backend-on-wayland-fedora-25/#more-502

==== How to know what version of <code>gtk</code> is being used in your session? ====
<code>Help -> About -> Installation Details -> Configuration Tab</code>

Look for something like:

<code>org.eclipse.swt.internal.gtk.version=3.22.6</code>

[Source] https://coffeeorientedprogramming.com/2014/10/27/how-to-tell-if-you-are-running-eclipse-on-gtk2-or-on-gtk3/

==== How do you know you are running Wayland? ====
<code>loginctl</code> which should show you a list of sessions, one of which will be <code>gdm</code>, remember that session number.

<code>loginctl show-session <session number from above> -p Type</code>

Result should be:

<code>Type=wayland</code>

[Source] http://unix.stackexchange.com/questions/202891/how-to-know-whether-wayland-or-x11-is-being-used

TipsAndTricks/CropsCLIContainers

2017-08-29T22:25:12Z

Bavery:

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container. This wiki assumes you have followed the basic documentation on the CROPS github Readme's (like https://github.com/crops/poky-container) and are looking for more tips.

 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk. It works equally well with an old fashioned sdk (e.g. one without devtool but with a sysroot and a toolchain).

=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Instructions and explanations for Mac and Windows can be found here:
https://github.com/crops/docker-win-mac-docs/wiki

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busybox
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.
== Workflow ==
I tend to run this under tmux. I keep one tmux panel inside the container for commands and another outside the container to do host side editing/git operations.
== Git issues ==
Sometimes i try to do something in the container like git stash save "my stash" and it fails telling me that pokyuser does not have a default git user or email setup. There are 2 ways to work around this.

TipsAndTricks/CropsCLIContainers

2017-08-29T21:01:47Z

Bavery:

TipsAndTricks/CropsCLIContainers

2017-08-29T21:00:27Z

Bavery: /* eSDK */

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container. This wiki assumes you have followed the basic documentation on the CROPS github Readme's (like https://github.com/crops/poky-container) and are looking for more tips.

 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk. It works equally well with an old fashioned sdk (e.g. one without devtool but with a sysroot and a toolchain).

=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busybox
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.

TipsAndTricks/CropsCLIContainers

2017-08-29T20:59:34Z

Bavery:

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container. This wiki assumes you have followed the basic documentation on the CROPS github Readme's (like https://github.com/crops/poky-container) and are looking for more tips.

 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk.
=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busybox
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.

TipsAndTricks/CropsCLIContainers

2017-08-29T20:58:51Z

Bavery: /* Introduction */

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container. This wiki assumes you have followed the basic documentation on the CROPS github Readme's (like https://github.com/crops/poky-container) and are looking for more tips.

== How to Get Started ==
The setup on Mac and Windows requires some extra steps before you get up and running. This is due to the fact that neither the Mac nor the Windows file systems support all the features we need to do a build. To deal with this issue on these OS's we put the build into a persistent docker volume (https://docs.docker.com/engine/tutorials/dockervolumes/) and expose it to the host OS using a Samba container. The detailed instructions for doing this are located at https://github.com/crops/docker-win-mac-docs/wiki

If you plan to use these containers on Linux, you can skip the docker volume and the Samba container instructions as they are unnecessary. The detailed instructions for running the various containers are located in the README.md for each container source on github. Pointers to these instructions for the Poky,eSDK, and Toaster containers are located at the bottom of the above referenced wiki.
 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk.
=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busybox
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.

TipsAndTricks/CropsCLIContainers

2017-08-29T20:57:18Z

Bavery: /* Issues with File Paths */

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container.

== How to Get Started ==
The setup on Mac and Windows requires some extra steps before you get up and running. This is due to the fact that neither the Mac nor the Windows file systems support all the features we need to do a build. To deal with this issue on these OS's we put the build into a persistent docker volume (https://docs.docker.com/engine/tutorials/dockervolumes/) and expose it to the host OS using a Samba container. The detailed instructions for doing this are located at https://github.com/crops/docker-win-mac-docs/wiki

If you plan to use these containers on Linux, you can skip the docker volume and the Samba container instructions as they are unnecessary. The detailed instructions for running the various containers are located in the README.md for each container source on github. Pointers to these instructions for the Poky,eSDK, and Toaster containers are located at the bottom of the above referenced wiki.
 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk.
=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busybox
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.

TipsAndTricks/CropsCLIContainers

2017-08-29T17:28:21Z

Bavery:

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container.

== How to Get Started ==
The setup on Mac and Windows requires some extra steps before you get up and running. This is due to the fact that neither the Mac nor the Windows file systems support all the features we need to do a build. To deal with this issue on these OS's we put the build into a persistent docker volume (https://docs.docker.com/engine/tutorials/dockervolumes/) and expose it to the host OS using a Samba container. The detailed instructions for doing this are located at https://github.com/crops/docker-win-mac-docs/wiki

If you plan to use these containers on Linux, you can skip the docker volume and the Samba container instructions as they are unnecessary. The detailed instructions for running the various containers are located in the README.md for each container source on github. Pointers to these instructions for the Poky,eSDK, and Toaster containers are located at the bottom of the above referenced wiki.
 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk.
=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

== Issues with File Paths ==
While the default way to run the containers (using crops/poky as an example but this applies equally well to the others).
docker run --rm -it -v /home/myuser/mystuff:/workdir crops/poky --workdir=/workdir
highlights the fact that the directory path inside the container is independent of the directory path on the host, this can sometimes be confusing. So, it is also quite reasonable to run the container like so:
cd /home/myuser/mystuff
docker run --rm -it -v $(pwd):$(pwd) crops/poky --workdir=$(pwd)
This way, inside the container you have the same directory path as you do outside of the container. For things like
devtool modify busyboxl
devtool find-recipe busybox
this can make cutting and pasting easier from the "in container terminal" which lacks an editor into the editor or terminal with an editor on the host system.

TipsAndTricks/CropsCLIContainers

2017-08-29T17:14:34Z

Bavery: /* Poky */

=CROPS or How to Run bitbake/devtool/eSDK/Toaster on Your MacWinux Host=
== Introduction ==
CROPS provides CLI containers that allow you to run the set of tools listed in the title on Mac OSX, Windows or Linux.
The containers themselves are available on Dockerhub at https://hub.docker.com/u/crops/dashboard/

You may wonder why you would use these containers on Linux. The main reason is it gives you a guaranteed clean environment from which to build. Also it means you don't need to install the dependent packages required by the build tools. Measurements show '''no''' speed degradation when running in the container.

== How to Get Started ==
The setup on Mac and Windows requires some extra steps before you get up and running. This is due to the fact that neither the Mac nor the Windows file systems support all the features we need to do a build. To deal with this issue on these OS's we put the build into a persistent docker volume (https://docs.docker.com/engine/tutorials/dockervolumes/) and expose it to the host OS using a Samba container. The detailed instructions for doing this are located at https://github.com/crops/docker-win-mac-docs/wiki

If you plan to use these containers on Linux, you can skip the docker volume and the Samba container instructions as they are unnecessary. The detailed instructions for running the various containers are located in the README.md for each container source on github. Pointers to these instructions for the Poky,eSDK, and Toaster containers are located at the bottom of the above referenced wiki.
 
 NOTE: ALL the CROPS containers take a --help argument in case you forget the options! For example:
<code> docker run -it --rm crops/poky --help

usage: poky-entry.py [-h] [--workdir WORKDIR] [--id ID]
optional arguments:
-h, --help show this help message and exit
--workdir WORKDIR The active directory once the container is running. In
the abscence of the "id" argument, the uid and gid of the
workdir will also be used for the user in the container.
--id ID uid and gid to use for the user inside the container. It
should be in the form uid:gid
</code>

== Variations Explained ==
=== Poky ===
The default Poky container is quite generic and will work with most recent releases of YP. Despite the name, it simply contains all the dependent packages required by Poky but not the meta-data. It will work equally well for an OE Core setup. The default container is based on the latest Ubuntu LTS that YP supports. We are in the process of creating ones based on the rest of the standard supported Linux Distro's. For the common usage; however, which Distro is inside is irrelevant since everything is already set up for the user.

=== eSDK ===
The eSDK container is also generic in that it takes an extensible sdk install script as an argument so it can be used with a fairly arbitrary ext-sdk.
=== Toaster ===
Toaster is more complicated. To facilitate quick startup, the Toaster containers include a database prepopulated from the layer index https://layers.openembedded.org. Because of this, the toaster containers come in a couple of varieties.
* Toaster - https://hub.docker.com/r/crops/toaster/ - This is the last official release YP has available. At the time of writing this T&T, for instance, this is krogoth and will soon be morty.
* Toaster-krogoth - https://hub.docker.com/r/crops/toaster-krogoth - The krogoth release.
* Toaster -morty - https://hub.docker.com/r/crops/toaster-morty/ - The RC release. We created this once the morty branch was named/made.
* Toaster-master - https://hub.docker.com/r/crops/toaster-master/ - This is the ever changing master version. A new image is built for every commit to http://git.yoctoproject.org/cgit/cgit.cgi/poky/ on the master branch. We keep ~ 2 weeks back. A <code>docker pull toaster-master</code> will give you the latest commit. If you want a particular commit, look at the tags page: https://hub.docker.com/r/crops/toaster-master/tags/ and you see a number of options.
** latest - the last successful (passed the Travis test viewable at https://travis-ci.org/crops/toaster-container )
** latest-<date>-<commitish> - the latest tag points to the last of these. We also keep old ones around for ~ 2 weeks.
** toaster-next-<date>-<commitish> - This is built using the toaster-next branch of poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=toaster-next. This is intended for people actively developing toaster source code itself.

==Operating System Oddities==
While the purpose of the CROPS cli containers is to insulate the user from the vagaries of the host os, sometimes the oddness creeps in. Here's some of the host specific issues we've run into so far in case they affect you too.
=== Mac OSX ===
==== Samba/Files ====
Sometimes you may find yourself unable to delete files from you docker volume from the host os. If you see something like:
<code>
$ rm /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
override rwxrwxrwx bavery/staff arch,uchg for/Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack? y
</code>
it means that a flag has been set that is preventing you from deleting the file. To see what flags are set on the file do:
<code>
$ls -lO /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
-rwxrwxrwx 1 bavery staff arch,uchg 34188262 Oct 31 16:07 /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
In the above example , the uchg (user immutable) and arch (arvhived) flags are set. These need to be cleared using the command [https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/chflags.1.html chflags]
To clear them do:
<code>
$chflags noarch,nouchg /Volumes/workdir/.git/objects/pack/pack-67ce6b1222786a14656905bd779637f00fb7225e.pack
</code>
Then you can rm the file. Alternatively, you could do the rm from with in a container and the linux side is less persnickety.
=== Windows ===
The biggest issue for Windows so far is that Windows 10 Pro, Enterprise, and Education can use [https://docs.docker.com/docker-for-windows/ Docker for Windows ] while earlier versions of Windows need to use [https://docs.docker.com/toolbox/overview/ Docker Toolbox]. Docker for Windows matches what is used on the Mac and is very similar to the experience on Linux. Docker Toolbox insets a VM (by default Virtualbox) in between the host and the cotainers.

TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds

2017-07-18T02:08:00Z

Bavery:

TipsAndTricks/DockerOnImage

2017-07-14T19:54:25Z

Bavery: /* Current Hacks Needed */

This assumes you'd like to build an image that supports docker on a YP image. This is pretty straightforward due to the great work done on the meta-virtualization layer, mostly by people at Wind River. This document will show how to get docker running on a Nuc. I have run the same image on a Minnowboard (Turbot) successfully as well. Note, these layers change, so this is accurate as of July 2017.
== What Layers Do You Need? ==
The main layer is meta-virtualization. This can be found using the layers.openembedded search app, Here's the [[https://layers.openembedded.org/layerindex/branch/master/layer/meta-virtualization/ meta-virtualiation]] entry. As you can see from the dependency list, this layer requires oe-core,meta-oe, meta-networking, and meta-filesystems. Some of these (like meta-networking) have additional dependencies. Here's the complete list of additional layers you need:
* [[https://git.yoctoproject.org/git/meta-virtualization meta-virtualization]]
* [[git://git.openembedded.org/meta-openembedded meta-openembedded]]
** The meta-openembedded repo contains the rest of the needed layers:
*** meta-python
*** meta-networking
*** meta-filesystems
*** meta-oe
You need to add these (with absolute paths) to your build/conf/bblayers.conf file. BBLAYERS should look something like this:
BBLAYERS ?= " \
<AbsPath>/poky/meta \
<AbsPath>/poky/meta-poky \
<AbsPath>/poky/meta-yocto-bsp \
<AbsPath>/meta-openembedded/meta-oe \
<AbsPath>/meta-openembedded/meta-python \
<AbsPath>/meta-openembedded/meta-networking \
<AbsPath>/meta-openembedded/meta-filesystems \
<AbsPath>/meta-virtualization \

== What Configuration Changes Do you Need? ==
There are a number of configuration settings you need in your build/conf/local.conf in order to make this work. I'll list them below and try to explain why you need them:
# this gives us the linux-yocto kernel for an x86-64 machine like minnowboard
MACHINE = "genericx86-64"

# Docker presumes systemd
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

# we need space for images
IMAGE_ROOTFS_EXTRA_SPACE = "10000000"

# The extra space takes us above 4gb, so
# turn off hdd and iso images so they do
# not break
NOHDD="1"
NOISO="1"

# pick a kernel I know works:
PREFERRED_PROVIDER_virtual/kernel="linux-yocto"
PREFERRED_VERSION_linux-yocto="4.9%"

# add docker to the image
# connman to manage the networking
IMAGE_INSTALL_append = " docker docker-contrib connman connman-client \
"
== Current Hacks Needed ==
Some of this is not upstreamed yet, so here's some patches you may need to apply. Check and see of they have been accepted upstream or not, first...
=== Proxy problems ===
Once on the target board, you need to tell docker about your proxy (if you have one). This is a little hard to do by hand, so here are a couple of files you can insert into meta-virtualization to help you along.

meta-virtualization$ cat recipes-containers/docker/docker_git.bbappend

SRC_URI += "file://http-proxy.conf"
do_install_append() {
if [ -n "${http_proxy}" ]; then
docker_config_dir=${sysconfdir}/systemd/system/docker.service.d
install -d ${D}/$docker_config_dir
sed -e s_{URL}_${http_proxy}_ ${WORKDIR}/http-proxy.conf > ${D}/$docker_config_dir/http-proxy.conf
fi
}

meta-virtualization$ cat recipes-containers/docker/files/http-proxy.conf

[Service]
Environment="HTTP_PROXY={URL}/"

== What To Build ==
Once these are all in place, you are ready to make your image. I'd suggest building core-image-base. Note: core-image-mininal will NOT work since it does not include the kernel module packages by default.
$bitbake core-image-base
== How to get the image onto your usb key ==
You need to check (lsusb or dmesg) to see that /dev/sdX your usb key showed up as.
* sudo umount /dev/sdX1;sudo umount /dev/sdX2; <unmount any others as well>
** slow way
*** sudo dd if=tmp/deploy/images/genericx86-64/core-image-base-genericx86-64.wic of=/dev/sdX
** fast way
*** bitbake bmap-tools-native -caddto_recipe_sysroot
*** sudo chmod 666 /dev/sdX
*** oe-run-native bmap-tools-native bmaptool copy ./tmp/deploy/images/genericx86-64/core-image-base-genericx86-64.wic /dev/sdX
== How to Test ==
You can now log onto your target Minnowboard, and you can test out docker, which should be running.
$root@genericx86-64:~# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.0
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Kernel Version: 4.9.31-yocto-standard
Operating System: Poky (Yocto Project Reference Distro) 2.3 (pyro)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.618 GiB
Name: genericx86-64
ID: JIJP:ZUBZ:BC3J:JNVQ:3EOV:SVUP:RDO3:XGFN:WB5X:XIUH:34PX:EHBM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://foo.com:123/
Registry: https://index.docker.io/v1/
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
27144aa8f1b9: Pull complete
Digest: sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642
Status: Downloaded newer image for busybox:latest

$ docker run -it --rm busybox sh
/ #
If there are issues try <code> systemctl status docker </code> to see what went wrong.

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-13T22:01:13Z

Bavery: /* Additional Dependencies to Bitbake */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want by doing a
IMAGE_INSTALL_append += libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T18:06:57Z

Bavery: /* Debugging */

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want.
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T18:06:09Z

Bavery:

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want.
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
This is the success we've been aiming for all along!!! Yay!
 
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T18:05:03Z

Bavery:

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want.
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running Clear Containers ==
You will need a couple of logins onto the target.
=== More configuration ===
We have the CC runtime, now we need to let docker know about it. We change the startup of the docker systemd service to know about the new cor runtime and we also turn on debugging. Note, for faster performance, we would drop the -D and do <code> --add-runtime cor=/usr/bin/cc-oci-runtime</code> the .sh script is a clever interceptor that let's us turn on additional debugging features in case we need to.

# diff /lib/systemd/system/docker.service ~/docker.service
12c12
< ExecStart=/usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-oci-runtime.sh -H fd://
---
> ExecStart=/usr/bin/dockerd -H fd://

Once the systemd service has been modified, you need to tell systemd to reload, and then restart docker
# systemctl daemon-reload
# systemctl stop docker
# systemctl start docker

=== The Proxy ===
* by hand:
/usr/libexec/cc-proxy -v 3
This will show the connections and is quite reassuring early. This can also be run using the systemd service that got installed while building the runtime.
# ls /lib/systemd/system/cc-proxy.s*
/lib/systemd/system/cc-proxy.service /lib/systemd/system/cc-proxy.socket
=== A Container ===
Now all the pieces are in place for the big finale. This will run a "normal" docker container
# docker run -it --rm busybox sh
and
# ps augxww | grep qemu
should be empty.
Now, let's run a Clear Container:
# docker run -it --rm --runtime cor busybox sh
/ #
How do we know it worked?
* ps This will show us a qemu instance running:
#ps augxww | grep qemu
root 10192 0.7 0.9 716620 73864 ? Ssl 18:00 0:00 /usr/local/bin/qemu-system-x86_64 -name 0966e97e19a2 -machine
q35,accel=kvm,kernel_irqchip,nvdimm ..... and much more ....
* our proxy will have some nifty information, here's a snippet:
I0708 18:00:35.512724 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_read
I0708 18:00:35.512813 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.512933 10139 vm.go:114] [vm e9276e31 hyperstart] get length 41
I0708 18:00:35.513016 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 14, len 4
I0708 18:00:35.513148 10139 vm.go:114] [vm e9276e31 hyperstart] 0 0 0 b 0 0 0 29 7b 22 73 65 71 22 3a 31 2c 20
22 72 6f 77 22 3a 34 38 2c 20 22 63 6f 6c 75 6d 6e 22 3a 31 31 33 7d
I0708 18:00:35.513213 10139 vm.go:114] [vm e9276e31 hyperstart] hyper_channel_handle, type 11, len 41
I0708 18:00:35.513295 10139 vm.go:114] [vm e9276e31 hyperstart] call hyper_win_size, json {"seq":1, "row":48, "
column":113}, len 33
I0708 18:00:35.513429 10139 vm.go:114] [vm e9276e31 hyperstart] exec seq 1, seq 1
I0708 18:00:35.513525 10139 vm.go:114] [vm e9276e31 hyperstart] hyper send type 9, len 0

* and , since we are running the debug version (Remember, we set docker up with -D and the runtime to point to /usr/bin/cc-oci-runtime.sh, we also have a CC logfile in /run/cc-oci-runtime/cc-oci-runtime.log
# cat /run/cc-oci-runtime/cc-oci-runtime.log
... much stuff ...
2017-07-08T18:00:35.487831Z:10216:cc-oci-runtime:debug:proxy msg length: 16
2017-07-08T18:00:35.487868Z:10216:cc-oci-runtime:debug:message read from proxy socket: {"success":true}
2017-07-08T18:00:35.487940Z:10216:cc-oci-runtime:debug:msg received: {"success":true}
2017-07-08T18:00:35.487954Z:10216:cc-oci-runtime:debug:disconnecting from proxy
2017-07-08T18:00:35.488579Z:10216:cc-oci-runtime:debug:created state file /var/run/cc-oci-
runtime/e9276e3130766473ecf58edaf76fea76c3df46db9c1dff5dc5113e6e7a85a205/state.json
* as you can see, we also have a state file. This shows things like
"vm" : {
"pid" : 10192,
"hypervisor_path" : "/usr/local/bin/qemu-system-x86_64",
"image_path" : "/usr/share/clear-containers/clear-16050-containers.img",
"kernel_path" : "/usr/share/clear-containers/vmlinux-4.9.33-74.container",
"workload_path" : "",
....

=== Debugging ===
There is a lot of debugging information on the CC site, so I won't repeat it here. I used 3 main things when I was debugging this:
* the stderr/stdout for the proxy.
* the CC runtime log /run/cc-oci-runtime/cc-oci-runtime.log
* I also added an additional debug to the /usr/bin/cc-oci-runtime.sh. I added debugging for the hypervisor itself.
# mkdir /tmp/hypervisor
# diff /usr/bin/cc-oci-runtime.sh /usr/bin/cc-oci-runtime.sh.orig
64c64
< runtime_args="$runtime_args --global-log=\"$global_log\" --hypervisor-log-dir=/tmp/hypervisor "
---
> runtime_args="$runtime_args --global-log=\"$global_log\""

The hypervisor logs are empty unless there's a problem..,

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T17:22:13Z

Bavery:

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want.
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon. Now we can do:
make
make install
# qemu-system-x86_64 -machine help| grep q35
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)

The last line shows we can run qemu and have the machine available.

== Running It ==
=== Configuration ===
=== Programs ===
=== Example ===
=== Debugging ===

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T17:17:46Z

Bavery:

== What & Why ==
Clear containers (CC) offer a hybrid solution that encompasses the advantages of hypervisor security and container deployment. So, we wanted to see if they could be used in a YP environment. This was done for Clear Containers 2.2 based on YP master around the time of 2.4 RC1/2.
 
Note: this is a Proof of Concept, done by building on target. The eventual goal would be to create a standard recipe to allow the clear containers to be built in the standard way. Hopefully, this guide will help with that by outlining the parts, dependencies, and configuration steps. This guide assumes you already have docker running on your target by having followed [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. The target example is being done with an Intel Nuc. I have successfully run the same code on a Minnowboard Turbot.

== Dependencies you need ==
=== Layers ===
The layers I am using:
meta-openembedded/meta-oe
meta-openembedded/meta-python
meta-openembedded/meta-networking
meta-openembedded/meta-filesystems
meta-virtualization
meta-clear

All of these layers can be found on [https://layers.openembedded.org/layerindex/branch/master/layers/ layer.openembedded.org] except the meta-clear. The meta-clear layer was created with the script yocto-layer. It's only purpose is to turn on CONFIG_VHOST_NET=m for the kernel. Here's a tree of the layer:

├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-kernel
└── linux
├── linux-yocto
│ ├── clear.cfg
│ └── clear.scc
├── linux-yocto_4.10.bbappend
└── linux-yocto_4.9.bbappend

I am using the 4.9 kernel. Here's the linux-yocto_4.9.bbappend:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://clear.scc \
"
KERNEL_MODULE_AUTOLOAD += "vhost-net"

The scc file:
define KFEATURE_DESCRIPTION "Enable clearcon support"
define KFEATURE_COMPATIBILITY board
kconf non-hardware clear.cfg

And finally the cfg file:
CONFIG_VHOST_NET=m

=== Conf Changes ===
This guide presumes you have the setup in your conf file described in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage Running Docker on your image ]. In addition, to make on target building easier, I add the following to my conf/local.conf:
EXTRA_IMAGE_FEATURES += " dev-pkgs tools-sdk tools-debug tools-profile "

=== Additional Dependencies to Bitbake ===
These are the additional recipes I built in addition to the base I outlined above. They could be added all at once in the local.conf, if you want.
bitbake libcheck mdadm psmisc json-glib libmnl ossp-uuid autoconf-archive python-setuptools libcap-ng tunctl go wget sudo
These are additional packages I built for convenience, but they are not required:
bitbake less zile ntp rsync minicom
Once built these can be installed on the board. Note that we need the dev pkgs as we are mostly completing build requirements for pieces of CC.
dnf install tunctl python-dev python-setuptools-dev libcap-ng-dev libcheck-dev libmnl-dev libjson-glib-1.0-dev autoconf-archive-dev libcap-ng-dev python-setuptools-dev go wget sudo
and the convenient ones:
dnf install zile less ntp rsync minicom

=== The Image to Build ===
bitbake core-image--base

== The Pieces of CC==
Clear Containers are comprised of a set of software and binaries. The main code is a slightly forked (2.9 currently) qemu hypervisor configured to be minimal, a command proxy, a shim, and the oci runtime. The command proxy is written in go. The rest is c/c++. We build the hypervisor itself, but the binaries for the hypervisor are downloaded from the CC site.
=== The Runtime,Shim & Proxy ===
This comes from [[https://github.com/01org/cc-oci-runtime clear oci runtime]]. While getting it to work, I followed the development model outlined in [https://wiki.yoctoproject.org/wiki/TipsAndTricks/OnTargetWorkFlowLeveragingRPMPackagefeeds Leveraging Rpm Package Feeds]. Here I will list the dependencies to make it shorter.
 
Which Clear was this?
cc-oci-runtime version: 2.2.0
spec version: 1.0.0-rc1
commit: f92d50ad54003298c139de59777f07588683cdc2
==== Getting the Source Code ====
We will pretty much follow the (very good) instructions in the [https://github.com/01org/cc-oci-runtime README]. Because it is a go project we will follow the go flow...
go get -d github.com/01org/cc-oci-runtime/...
The ... is necessary. If you are behind a proxy, make sure you export http_proxy and https_proxy into your shell.
 
This will put the src in ~/go/src/github.com/01org/cc-oci-runtime/ by default.

==== Building It ====
Again, following their [https://github.com/01org/cc-oci-runtime README] do
./autogen.sh --disable-functional-tests
We disable functional tests because I was not able to easily find a recipe for BATS (Bash automated testing).
make
The make install is quite clean and show where everything is going:
# make install
make[1]: Entering directory '/home/root/go/src/github.com/01org/cc-oci-runtime'
/bin/mkdir -p '/usr/bin'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-oci-runtime '/usr/bin'
libtool: install: /usr/bin/install -c cc-oci-runtime /usr/bin/cc-oci-runtime
/bin/mkdir -p '/usr/bin'
/usr/bin/install -c data/cc-oci-runtime.sh '/usr/bin'
/bin/mkdir -p '/usr/libexec'
/bin/sh ./libtool --mode=install /usr/bin/install -c cc-shim '/usr/libexec'
libtool: install: /usr/bin/install -c cc-shim /usr/libexec/cc-shim
/bin/mkdir -p '/usr/libexec'
/usr/bin/install -c cc-proxy '/usr/libexec'
/bin/mkdir -p '/usr/share/defaults/cc-oci-runtime'
/usr/bin/install -c -m 644 data/vm.json data/hypervisor.args data/kernel-cmdline '/usr/share/defaults/cc-oci-runtime'
/bin/mkdir -p '/lib/systemd/system'
/usr/bin/install -c -m 644 proxy/cc-proxy.service proxy/cc-proxy.socket '/lib/systemd/system'
==== Getting the Artifacts ====
We need a kernel and an image for the hypervisor.
* The kernel comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] For example, I used release 16050 ~ June 2017. This can be found in the following rpm [[https://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm kernel rpm 16050]]
* The image (rootfs) also comes from [[https://download.clearlinux.org/releases/ Clear Linux Releases.]] I again picked the [[https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz 16050 release image ]]
* Here's the commands to download the above and install it.
wget http://download.clearlinux.org/releases/16050/clear/x86_64/os/Packages/linux-container-4.9.33-74.x86_64.rpm
wget https://download.clearlinux.org/releases/16050/clear/clear-16050-containers.img.xz
rpm --install ./linux-container-4.9.33-74.x86_64.rpm
xz --decompress clear-16050-containers.img.xz
cp clear-16050-containers.img /usr/share/clear-containers/
pushd /usr/share/clear-containers/
ln -s clear-16050-containers.img clear-containers.img
popd
After this, you should have a /usr/share/clear-containers directory that looks like this:
|-- clear-16050-containers.img
|-- clear-containers.img -> clear-16050-containers.img
|-- vmlinux-4.9.33-74.container
|-- vmlinux.container -> vmlinux-4.9.33-74.container
|-- vmlinuz-4.9.33-74.container
`-- vmlinuz.container -> vmlinuz-4.9.33-74.container

==== Configuring It ====
The configuration is located in /usr/share/defaults/cc-oci-runtime. There are 3 files.
* vm.json defines which hypervisor, kernel and rootfs we use:
{
"vm": {
"path": "/usr/bin/qemu-system-x86_64",
"image": "/usr/share/clear-containers/clear-containers.img",
"kernel": {
"path": "/usr/share/clear-containers/vmlinux.container",
"parameters": "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check
rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1
initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service
systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0"
}
}
}
* kernel-cmdline is well, the kernel command line
root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1
i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd
systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket
systemd.show_status=false cryptomgr.notests net.ifnames=0
* hypervisor.args defines how we start up the qemu hypervisor
/usr/bin/qemu-system-x86_64
-name
@NAME@
-machine
pc-lite,accel=kvm,kernel_irqchip,nvdimm
-device
nvdimm,memdev=mem0,id=nv0
-object
memory-backend-file,id=mem0,mem-path=@IMAGE@,size=@SIZE@
-m
2G,slots=2,maxmem=3G
-kernel
@KERNEL@
-append
@KERNEL_PARAMS@ @KERNEL_NET_PARAMS@
-smp
2,sockets=1,cores=2,threads=1
-cpu
host

This is the file we need to change to accommodate our small memory availability on the Minnowboard Turbot (4 G total) . We are also using a newer qemu and machine type than the hypervisor.args file assumes. This helps to make the Atom processor happy. Here's the diff:
# diff hypervisor.args.orig hypervisor.args
5c5
< pc-lite,accel=kvm,kernel_irqchip,nvdimm
---
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,nofw
11c11
< 2G,slots=2,maxmem=3G
---
> 256M,slots=2,maxmem=1G

I capped it at 1G for now. This setting would definitely need to be tweaked depending on needs/usage...

===The Qemu Hypervisor ===
==== Getting the Source Code ====
git clone https://github.com/clearcontainers/qemu qemu-cc
cd qemu-cc
git checkout -b qemu-lite-v2.9.0 origin/qemu-lite-v2.9.0

==== Building It ====
mkdir build
cd build
../configure '--disable-tools' '--disable-libssh2' '--disable-tcmalloc' '--disable-glusterfs' '--disable-seccomp' '--disable-bzip2' '--disable-snappy' '--disable-lzo' '--disable-usb-redir' '--disable-libusb' '--disable-libnfs' '--disable-tcg-interpreter' '--disable-debug-tcg' '--disable-libiscsi' '--disable-rbd' '--disable-spice' '--disable-attr' '--disable-cap-ng' '--disable-linux-aio' '--disable-brlapi' '--disable-vnc-jpeg' '--disable-vnc-png' '--disable-vnc-sasl' '--disable-rdma' '--disable-bluez' '--disable-fdt' '--disable-curl' '--disable-curses' '--disable-sdl' '--disable-gtk' '--disable-tpm' '--disable-vte' '--disable-vnc' '--disable-xen' '--disable-opengl' '--disable-slirp' '--enable-trace-backend=nop' '--enable-virtfs' '--enable-attr' '--enable-cap-ng' '--target-list=x86_64-softmmu' "$@"

In order to build it with gcc-7 (current YP compiler which is ahead of most/all std distros, I needed to HACK the following files:
modified: ../block/blkdebug.c
modified: ../block/blkverify.c
modified: ../hw/usb/bus.c
Here are the individual diffs:
* block/blkdebug.c
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 67e8024e36..e3ab19b947 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -668,7 +668,7 @@ static int blkdebug_truncate(BlockDriverState *bs, int64_t offset)

static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
{
- BDRVBlkdebugState *s = bs->opaque;
+// BDRVBlkdebugState *s = bs->opaque;
QDict *opts;
const QDictEntry *e;
bool force_json = false;
@@ -688,11 +688,13 @@ static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
return;
}

+#if 0
if (!force_json && bs->file->bs->exact_filename[0]) {
snprintf(bs->exact_filename, sizeof(bs->exact_filename),
"blkdebug:%s:%s", s->config_file ?: "",
bs->file->bs->exact_filename);
}
+#endif

opts = qdict_new();
qdict_put_obj(opts, "driver", QOBJECT(qstring_from_str("blkdebug")));

* block/blkverify.c
diff --git a/block/blkverify.c b/block/blkverify.c
index 9a1e21c6ad..aa5d5ccdad 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -302,6 +302,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->full_open_options = opts;
}

+#if 0
if (bs->file->bs->exact_filename[0]
&& s->test_file->bs->exact_filename[0])
{
@@ -310,6 +311,7 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
bs->file->bs->exact_filename,
s->test_file->bs->exact_filename);
}
+#endif
}

static BlockDriver bdrv_blkverify = {

* hw/usb/bus.c
diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 24f1608b4b..2f2abc1824 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -407,8 +407,8 @@ void usb_register_companion(const char *masterbus, USBPort *ports[],
void usb_port_location(USBPort *downstream, USBPort *upstream, int portnr)
{
if (upstream) {
- snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
- upstream->path, portnr);
+// snprintf(downstream->path, sizeof(downstream->path), "%s.%d",
+// upstream->path, portnr);
downstream->hubcount = upstream->hubcount + 1;
} else {
snprintf(downstream->path, sizeof(downstream->path), "%d", portnr);

I verified with fprintf(stderr....) that these are not hit in typical usage. I also expect they will be upstream patched soon.

==== How to Debug - Simple Version ====

=== The Hypervisor's Binaries ===

== Running It ==
=== Configuration ===
=== Programs ===
=== Example ===
=== Debugging ===

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T17:15:01Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T17:04:17Z

Bavery: /* Getting the Source Code */

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T17:03:43Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:57:27Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:46:46Z

Bavery: /* Getting the Artifacts */

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:44:49Z

Bavery: /* Getting the Artifacts */

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:42:18Z

Bavery: /* Getting the Artifacts */

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:32:45Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:22:50Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:21:06Z

Bavery:

TipsAndTricks/BuildingAndRunningClearContainersonTarget

2017-07-08T16:14:32Z

Bavery: /* Building It */