Yocto Project - User contributions [en]

Build Appliance Design

2012-05-29T08:34:42Z

Dcui: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Preferred: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
<del>Improve self-hosted performance</del> - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch already in poky master'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
<del> Transfer to vmware image - '''done''' </del>
* [P2][D2] <del>try vmware workstation</del> - Dexuan
* [P2][D3] <del>disk image format translating from qemu to vmware </del> - Dexuan

'''5.
Reduce image size - '''image size is low priority, performance is the key'''
* [P2][D5] <del>identify and remove unnecessary recipes</del> - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size & improve performance - Edwin

'''6.
<del>Deploying image - '''done''' </del>
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

'''7
Documentation
* [P2][D2] Readme also include instructions to setup sstate image - Edwin

'''8
Others
*<del> [P2][D2] one recipe to install source in the the self-hosted-image.(build from another new disk image)</del> - done by Dexuan and Saul
*<del> [P2] Fix "Multiple X provider" </del> - done by RP
*<del> [P2] Fix the PATH issue to find all utility </del> - done
*<del> [P2] hicolor issue </del>- done by Dexuan
* [P2] check the disk pass-through in vmware, ask for ESC license to check performance - Edwin

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Build Appliance Design

2012-05-29T08:28:08Z

Dcui: /* Goals */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Preferred: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
<del>Improve self-hosted performance</del> - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch already in poky master'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image - '''almost done'''
* [P2][D2] <del>try vmware workstation</del> - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan '''WIP'''

'''5.
Reduce image size - '''image size is low priority, performance is the key'''
* [P2][D5] <del>identify and remove unnecessary recipes</del> - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size & improve performance - Edwin

'''6.
Deploying image - '''Dexuan will update this'''
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

'''7
Documentation
* [P2][D2] Readme also include instructions to setup sstate image - Edwin

'''8
Others
* [P2][D2] one recipe to install source in the the self-hosted-image.(build from another new disk image) - Edwin
* [P2] Fix "Multiple X provider" - Edwin
* [P2] Fix the PATH issue to find all utility - Edwin
* [P2] hicolor issue - Dexuan
* [P2] check the disk pass-through in vmware, ask for ESC license to check performance - Edwin

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Working Behind a Network Proxy

2012-01-12T08:41:26Z

Dcui: /* CVS Setup (this seems useless since we don't have any recipe requiring cvs now */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2
Then run the following command:

git config --global core.gitProxy git-proxy

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup (this seems useless since we don't have any recipe requiring cvs now) ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:41:09Z

Dcui: /* CVS Setup */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2
Then run the following command:

git config --global core.gitProxy git-proxy

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup (this seems useless since we don't have any recipe requiring cvs now ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:37:15Z

Dcui: /* Git Setup (with socat) */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2
Then run the following command:

git config --global core.gitProxy git-proxy

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:36:08Z

Dcui: /* Git Setup (with socat) */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2

Then run the following command:

git config --global core.gitProxy git-proxy

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:34:34Z

Dcui:

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

git config --global core.gitProxy git-proxy

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:33:50Z

Dcui: /* Git Setup (with socat) */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Git Setup (with socat)==

First make sure you have the socat utility installed on your host (in Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

git config --global core.gitProxy git-proxy

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:32:58Z

Dcui:

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Git Setup (with socat)==

First make sure you have the socat utility (nc) installed on your host (In Ubuntu, this should be a simple command "sudo apt-get install socat")

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash
# $1 = hostname, $2 = port
PROXY=myproxy.example.com
exec socat STDIO SOCKS4:$proxy:$1:$2

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

git config --global core.gitProxy git-proxy

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Working Behind a Network Proxy

2012-01-12T08:19:35Z

Dcui: /* Git Setup */

This page lists some configuration tips for working behind a proxy.

== HTTP/HTTPS/FTP Setup ==

Set the following environment variables in your ~/.bashrc file. This example uses the same proxy server and port number for all three protocols.

export http_proxy='http://myproxy.example.com:1080/'
export https_proxy='https://myproxy.example.com:1080/'
export ftp_proxy='http://myproxy.example.com:1080/'
export no_proxy = '.example.com'

== Git Setup (with nc)==

First make sure you have the netcat utility (nc) installed on your host.

Create a script named ''git-proxy'' and put it in /usr/local/bin:

#!/bin/bash

PROXY=myproxy.example.com
PORT=1080

case $1 in
# list internal git servers here that you do not want to use
# the proxy with, separated by a pipe character '|' as below:
internalgit1.example.com|internalgit2.example.com)
METHOD="-X connect"
;;
*)
METHOD="-X 5 -x ${PROXY}:${PORT}"
;;
esac

/usr/bin/nc $METHOD $*

Note that on some Linux distros, the nc binary is in /bin. You can also change the '5' in the second METHOD line to '4' if your proxy server only supports SOCKS v4.

Then set the environment variable GIT_PROXY_COMMAND in your ~/.bashrc file and point it to this script:

export GIT_PROXY_COMMAND=/usr/local/bin/git-proxy
export GIT_PROXY_IGNORE="example.com"

== Subversion Setup ==

You'll need to have the following in your ~/.subversion/servers file:

[global]
http-proxy-exceptions = *.exception.com, www.internal-site.org
http-proxy-host = myproxy.example.com
http-proxy-port = 1080

You can also set ''http-proxy-username'' and ''http-proxy-password'' if your proxy requires authentication.

== CVS Setup ==

For CVS checkouts to work correctly, you need to add some options in your Poky ''local.conf'' file.

CVS_PROXY_HOST = "myproxy.example.com"
CVS_PROXY_PORT = "1080"

Run postinst during rootfs generation

2011-07-06T03:04:11Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:37:37Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:35:59Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:35:37Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:34:56Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:32:46Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:29:56Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:25:26Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:15:06Z

Dcui: /* 1.1 tasks */

Run postinst during rootfs generation

2011-07-03T14:14:22Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:11:28Z

Dcui:

Run postinst during rootfs generation

2011-07-03T14:09:12Z

Dcui:

== Background ==

In poky we have 2 types of postinst scripts: one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.

== 1.1 tasks ==

* We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.

* I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
** Add metadata to layers - can go in layer.conf
*** Layer name
*** Layer version
*** Layer dependencies (as a list of layer names + versions)
** Add dependency/version check
* Consider integrating Jeremy Puhlman's remote layers patch (see comments below) [Paul]
* Tool to manage combination repos (e.g. the poky repo) - see below [Paul/Ke]
* Tool to merge (flatten) several layers into one [Paul]
** Take current layer config and apply all .bbappend / remove overridden .bb files and output to a separate directory

==== Combination repo tool ====

* Separate tool - task specific and most people won't need to use it
* Assume current dir is within the combination repo
* Define repo components in a separate config file (conf/combolayer.conf ?). For each component:
** Source repo
** Destination subdir within combination repo tree
** Local repo workdir
** Pinned revision (optional)
** Last revision merged
* Operations:
** ''Create''
**# Check git initialised and working tree is clean
**# Clone component repositories
** ''Update'': merge changes in from upstream components
**# Ensure working tree is clean
**# Update combination repo
**# Update component repos
**# Apply patches from component repo to the combination repo (should this be done in a branch which we can throw away in case of error?)
**## Get list of revisions since last merged revision, then iterate:
**## Append upstream revision to commit message
**## Optionally append signed-off-by (-s)?
**## Apply revision to combination repo
** ''Splitpatch'': split a patch up over the components
**# Use the config file to understand which subdirs belong to which component
**# Output split patches to component-named subdirs of some destination folder (taken as an argument to the command)

=== Future tasks ===

Nice to have - may be added in 1.1 time allowing but not mandatory:

* Break out / make accessible functionality from OE's newcollection.bbclass to allow splitting recipes out to another layer
* Able to visualise dependencies between layers
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)
* Recipe maintenance tools - this is long term

== Brainstorming submissions ==

=== Richard Purdie ===
* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
** Multi SCM?
** Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseudo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.
** Paul says: am guessing a blocker for submodules (other than the fact that they look painful to deal with) is that they don't help us with the "part of another repo" thing; as well as being read-only.

=== Martin Jansa ===
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

=== Paul Eggleton / Chris Larson ===

* Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides across a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand. We have newcollection.bbclass in OE which does this, may be useful to break it out to a separate script somehow.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.
** Chris says: This is non-trivial, but doable given zecke's ast implementation. I expect we need an API function in the parse package to parse a string/file and hand back the AST nodes, which we could then manipulate, and write code to generate the file back out of the ast.

=== Jeremy Puhlman ===

[http://lists.linuxtogo.org/pipermail/openembedded-core/2011-April/001708.html Patch to add remote layers]

Paul says: a good start but naturally we want to avoid the bits that hard-code variable values. I wonder about situations where people want their own versions of layers or to share remote layers (e.g. between an oe-core setup and a poky setup); however people can just use local layers for this.

Jeremy says: In reality they are setting "reasonable" defaults. Hardcoding implies something that is unchangeable. All the values are "set if unset", so they can be changed in the bblayers.conf
file. Prior to the introduction of layers, all the values were hardcoded, just in a bitbake.conf file, so you always had working fetchers. With out that the fetchers are basically broken until
after you have loaded up the layers. Given the fact that nearly every bitbake.conf file contains the same settings for defining fetchcmd and a like, simply having a reasonable default setting for
the things the fetchers need is not really that terrible an idea irrespective of the remote layering.

As for the second question. I have some content tools that I am cleaning up that basically provide methods for defining layers and groups of layers together for mirroring and sharing. That might
address what your talking about there. They work along with the patch posted here, but could be modified to output things a bit differently if needed.

Richard says:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-May/001940.html

(discussion moved back to mailing list)

Run postinst during rootfs generation

2011-07-03T14:08:17Z

Dcui:

== Background ==

In poky we have 2 types of postinst scripts: one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.

=== 1.1 tasks ===

* We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.

* I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
** Add metadata to layers - can go in layer.conf
*** Layer name
*** Layer version
*** Layer dependencies (as a list of layer names + versions)
** Add dependency/version check
* Consider integrating Jeremy Puhlman's remote layers patch (see comments below) [Paul]
* Tool to manage combination repos (e.g. the poky repo) - see below [Paul/Ke]
* Tool to merge (flatten) several layers into one [Paul]
** Take current layer config and apply all .bbappend / remove overridden .bb files and output to a separate directory

==== Combination repo tool ====

* Separate tool - task specific and most people won't need to use it
* Assume current dir is within the combination repo
* Define repo components in a separate config file (conf/combolayer.conf ?). For each component:
** Source repo
** Destination subdir within combination repo tree
** Local repo workdir
** Pinned revision (optional)
** Last revision merged
* Operations:
** ''Create''
**# Check git initialised and working tree is clean
**# Clone component repositories
** ''Update'': merge changes in from upstream components
**# Ensure working tree is clean
**# Update combination repo
**# Update component repos
**# Apply patches from component repo to the combination repo (should this be done in a branch which we can throw away in case of error?)
**## Get list of revisions since last merged revision, then iterate:
**## Append upstream revision to commit message
**## Optionally append signed-off-by (-s)?
**## Apply revision to combination repo
** ''Splitpatch'': split a patch up over the components
**# Use the config file to understand which subdirs belong to which component
**# Output split patches to component-named subdirs of some destination folder (taken as an argument to the command)

=== Future tasks ===

Nice to have - may be added in 1.1 time allowing but not mandatory:

* Break out / make accessible functionality from OE's newcollection.bbclass to allow splitting recipes out to another layer
* Able to visualise dependencies between layers
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)
* Recipe maintenance tools - this is long term

== Brainstorming submissions ==

=== Richard Purdie ===
* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
** Multi SCM?
** Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseudo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.
** Paul says: am guessing a blocker for submodules (other than the fact that they look painful to deal with) is that they don't help us with the "part of another repo" thing; as well as being read-only.

=== Martin Jansa ===
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

=== Paul Eggleton / Chris Larson ===

* Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides across a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand. We have newcollection.bbclass in OE which does this, may be useful to break it out to a separate script somehow.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.
** Chris says: This is non-trivial, but doable given zecke's ast implementation. I expect we need an API function in the parse package to parse a string/file and hand back the AST nodes, which we could then manipulate, and write code to generate the file back out of the ast.

=== Jeremy Puhlman ===

[http://lists.linuxtogo.org/pipermail/openembedded-core/2011-April/001708.html Patch to add remote layers]

Paul says: a good start but naturally we want to avoid the bits that hard-code variable values. I wonder about situations where people want their own versions of layers or to share remote layers (e.g. between an oe-core setup and a poky setup); however people can just use local layers for this.

Jeremy says: In reality they are setting "reasonable" defaults. Hardcoding implies something that is unchangeable. All the values are "set if unset", so they can be changed in the bblayers.conf
file. Prior to the introduction of layers, all the values were hardcoded, just in a bitbake.conf file, so you always had working fetchers. With out that the fetchers are basically broken until
after you have loaded up the layers. Given the fact that nearly every bitbake.conf file contains the same settings for defining fetchcmd and a like, simply having a reasonable default setting for
the things the fetchers need is not really that terrible an idea irrespective of the remote layering.

As for the second question. I have some content tools that I am cleaning up that basically provide methods for defining layers and groups of layers together for mirroring and sharing. That might
address what your talking about there. They work along with the patch posted here, but could be modified to output things a bit differently if needed.

Richard says:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-May/001940.html

(discussion moved back to mailing list)

Run postinst during rootfs generation

2011-07-03T13:40:42Z

Dcui: Created page with '== Tasks == Extracted from the brainstorming info below. === 1.1 tasks === * Show a parse error if a bbappend matches no existing bb [Dexuan] * bitbake-layers: stop it display…'

== Tasks ==

Extracted from the brainstorming info below.

=== 1.1 tasks ===

* Show a parse error if a bbappend matches no existing bb [Dexuan]
* bitbake-layers: stop it displaying the warning message about not being run from the wrapper [Dexuan]
* bitbake-layers: extend it to show overlayed recipes [Paul]
* Update banner info to show layers being used and revisions/branches for each one if available [Dexuan]
* Dependency handling [Paul]
** Add metadata to layers - can go in layer.conf
*** Layer name
*** Layer version
*** Layer dependencies (as a list of layer names + versions)
** Add dependency/version check
* Consider integrating Jeremy Puhlman's remote layers patch (see comments below) [Paul]
* Tool to manage combination repos (e.g. the poky repo) - see below [Paul/Ke]
* Tool to merge (flatten) several layers into one [Paul]
** Take current layer config and apply all .bbappend / remove overridden .bb files and output to a separate directory

==== Combination repo tool ====

* Separate tool - task specific and most people won't need to use it
* Assume current dir is within the combination repo
* Define repo components in a separate config file (conf/combolayer.conf ?). For each component:
** Source repo
** Destination subdir within combination repo tree
** Local repo workdir
** Pinned revision (optional)
** Last revision merged
* Operations:
** ''Create''
**# Check git initialised and working tree is clean
**# Clone component repositories
** ''Update'': merge changes in from upstream components
**# Ensure working tree is clean
**# Update combination repo
**# Update component repos
**# Apply patches from component repo to the combination repo (should this be done in a branch which we can throw away in case of error?)
**## Get list of revisions since last merged revision, then iterate:
**## Append upstream revision to commit message
**## Optionally append signed-off-by (-s)?
**## Apply revision to combination repo
** ''Splitpatch'': split a patch up over the components
**# Use the config file to understand which subdirs belong to which component
**# Output split patches to component-named subdirs of some destination folder (taken as an argument to the command)

=== Future tasks ===

Nice to have - may be added in 1.1 time allowing but not mandatory:

* Break out / make accessible functionality from OE's newcollection.bbclass to allow splitting recipes out to another layer
* Able to visualise dependencies between layers
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)
* Recipe maintenance tools - this is long term

== Brainstorming submissions ==

=== Richard Purdie ===
* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
** Multi SCM?
** Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseudo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.
** Paul says: am guessing a blocker for submodules (other than the fact that they look painful to deal with) is that they don't help us with the "part of another repo" thing; as well as being read-only.

=== Martin Jansa ===
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

=== Paul Eggleton / Chris Larson ===

* Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides across a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand. We have newcollection.bbclass in OE which does this, may be useful to break it out to a separate script somehow.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.
** Chris says: This is non-trivial, but doable given zecke's ast implementation. I expect we need an API function in the parse package to parse a string/file and hand back the AST nodes, which we could then manipulate, and write code to generate the file back out of the ast.

=== Jeremy Puhlman ===

[http://lists.linuxtogo.org/pipermail/openembedded-core/2011-April/001708.html Patch to add remote layers]

Paul says: a good start but naturally we want to avoid the bits that hard-code variable values. I wonder about situations where people want their own versions of layers or to share remote layers (e.g. between an oe-core setup and a poky setup); however people can just use local layers for this.

Jeremy says: In reality they are setting "reasonable" defaults. Hardcoding implies something that is unchangeable. All the values are "set if unset", so they can be changed in the bblayers.conf
file. Prior to the introduction of layers, all the values were hardcoded, just in a bitbake.conf file, so you always had working fetchers. With out that the fetchers are basically broken until
after you have loaded up the layers. Given the fact that nearly every bitbake.conf file contains the same settings for defining fetchcmd and a like, simply having a reasonable default setting for
the things the fetchers need is not really that terrible an idea irrespective of the remote layering.

As for the second question. I have some content tools that I am cleaning up that basically provide methods for defining layers and groups of layers together for mirroring and sharing. That might
address what your talking about there. They work along with the patch posted here, but could be modified to output things a bit differently if needed.

Richard says:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-May/001940.html

(discussion moved back to mailing list)