Yocto Project - User contributions [en]

Build Appliance Design

2012-01-10T06:01:18Z

Gzhai: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Required: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
<del>Improve self-hosted performance</del> - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch already in poky master'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image - '''almost done'''
* [P2][D2] <del>try vmware workstation</del> - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan '''WIP'''

'''5.
Reduce image size - '''image size is low priority, performance is the key'''
* [P2][D5] <del>identify and remove unnecessary recipes</del> - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size & improve performance - Edwin

'''6.
Deploying image - '''Dexuan will update this'''
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

'''7
Documentation
* [P2][D2] Readme also include instructions to setup sstate image - Edwin

'''8
Others
* [P2][D2] one recipe to install source in the the self-hosted-image.(build from another new disk image) - Edwin
* [P2] Fix "Multiple X provider" - Edwin
* [P2] Fix the PATH issue to find all utility - Edwin
* [P2] hicolor issue - Dexuan
* [P2] check the disk pass-through in vmware, ask for ESC license to check performance - Edwin

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Build Appliance Design

2011-12-15T11:53:59Z

Gzhai: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Required: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
Improve self-hosted performance - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch will be out soon'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image
* [P2][D2] <del>try vmware workstation</del> - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan '''WIP'''

'''5.
Reduce image size - '''image size is low priority, performance is the key'''
* [P2][D5] <del>identify and remove unnecessary recipes</del> - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size & improve performance - Edwin

'''6.
Deploying image - '''Dexuan will update this'''
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Build Appliance Design

2011-12-15T11:38:41Z

Gzhai: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Required: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
Improve self-hosted performance - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch will be out soon'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image
* [P2][D2] <del>try vmware workstation</del> - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan

'''5.
Reduce image size - '''image size is low priority, performance is the key'''
* [P2][D5] <del>identify and remove unnecessary recipes</del> - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size & improve performance - Edwin

'''6.
Deploying image
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Build Appliance Design

2011-12-15T11:34:47Z

Gzhai: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Required: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
Improve self-hosted performance - '''done on KVM except pass-through, lower priority as vmware performance is acceptable'''
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
<del>Integrate hob</del> - '''patch will be out soon'''
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image
* [P2][D2] try vmware workstation - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan

'''5.
Reduce image size
* [P2][D5] identify and remove unnecessary recipes - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size - Edwin

'''6.
Deploying image
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

Build Appliance Design

2011-12-15T11:31:33Z

Gzhai: /* Plan */

This page holds the design for the Yocto Project build appliance.

=== Usage Model ===

This feature is designed to make the Yocto Project much more appealing to the developer who wants to check Yocto out but may not have a recent (and supported) Linux distro installed with all of the proxies set up correctly.

The developer will download a virtual image and boot it. This image is a Linux OS which will allow the user to do a build, boot the resulting Linux in an emulator. This gives a quick experience with the system without fear of dependencies being missing. (This is needed because of the general difficulty in having something as complex as the Yocto Project be totally compatible with every conceivable Linux system.

It is a non-goal that a developer would continue to use this appliance for all day-to-day development tasks.

=== Goals ===

# Required: Total size of the image must not exceed 100 Mbytes, to make it feasible to download the image.
# Preferred: Have a second, larger image which includes all source and sstate-cache preinstalled, but may be much larger.
# Preferred: VMWare ESX image. VMWare is known to correctly build whereas recent versions of Virtual Box and others are not.
# Required: Must have Linux plus all prerequisite packages installed to make a build work.
# Preferred: Generate the OS in the appliance with Yocto. (Thus, make it a self-hosted build appliance)
# Preferred: When the image boots, it boots up Hob and also has a terminal for launching QEMU or deploying the image
# Preferred: In addition to Hob, there is GUI support for deploying the image on a board and / or boot it into QEMU

=== Design Notes ===

First step is to build a non-graphical image that can provide a user with the needed tools to correctly build an image

Provide a simple X-Desktop with the HOB (pyGTK based) and a terminal (X-Term)

Create an APP or extend HOB to support deployment of images,
- this could be to a USB Device
- HDD
- USB Mem Stick
- SD Card or similar
- Burn a CD/DVD is available
- Other deployment option
- Network to real hardware (talk to Darren/Tom)?

=== Plan ===

'''1.
<del>Build an qemu image, in which "bitbake core-image-minimal" work</del>
* [P1][D2] check saul's branch to see how close it meet the goal - Dexuan/Edwin
* [P1][D3, depends on checking result] identify required host recipes and port them to yocto including
bitbake, wegt... Dexuan/Edwin
Use n450 to speed up debug process, and nfs for source/sstate at this stage

'''2.
Improve self-hosted performance
* [P2][D2] pre-install another disk image for source/sstate - Edwin
* [P2][D5] build performance improvement/test: <del>enabling KVM, SMP, virtio</del> or device pass-through for network/disk - Dexuan

'''3.
Integrate hob
* [P1][D5] minimal X system and required LIB to start terminal - Edwin
* [P1][D5] integrate hob - Edwin

'''4.
Transfer to vmware image
* [P2][D2] try vmware workstation - Dexuan
* [P2][D3] disk image format translating from qemu to vmware - Dexuan

'''5.
Reduce image size
* [P2][D5] identify and remove unnecessary recipes - Edwin
* [P2][D5] tune features for big recipes, like kernel/glibc to reduce image size - Edwin

'''6.
Deploying image
* [P1][D10] creating live hdd image/or ISO - Dexuan
* [P2][D10] extend HOB to deploy image - Dexuan

=== KVM Performance ===

See the following Page for details: [[BKM:_improve_qemu_performance]]

BKM: improve qemu performance

2011-12-06T13:43:14Z

Gzhai: /* Enable SMP */

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

''qemu built from yocto may have some bug that only 1 vcpu is truly used by guest(You can use top -d1 inside guest to see only 100% CPU usage). Pls. switch to the qemu in the linux distribution, say ubuntu, as one workaround.''

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
* enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
* change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''
* modify qemu NIC parameter: change the -net nic option to include '''model=virtio'''
Following is one example
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -drive file=/home/source.img,if=virtio -drive file=/home/build.img,if=virtio

== Enable VTd to for NIC/Block device ==
Enable VTd with KVM can directed assign PCI device(like NIC or USB disk) to guest, then achieve almost native performance. Pls. make sure your chipset and bios support VTd.
Details is to be coming soon:)

BKM: improve qemu performance

2011-12-06T13:41:35Z

Gzhai: /* Enable SMP */

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.
----
qemu built from yocto may have some bug that only 1 vcpu is truly used by guest(You can use top -d1 inside guest to see only 100% CPU usage). Pls. switch to the qemu in the linux distribution, say ubuntu, as one workaround.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
* enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
* change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''
* modify qemu NIC parameter: change the -net nic option to include '''model=virtio'''
Following is one example
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -drive file=/home/source.img,if=virtio -drive file=/home/build.img,if=virtio

== Enable VTd to for NIC/Block device ==
Enable VTd with KVM can directed assign PCI device(like NIC or USB disk) to guest, then achieve almost native performance. Pls. make sure your chipset and bios support VTd.
Details is to be coming soon:)

BKM: improve qemu performance

2011-12-06T13:38:02Z

Gzhai:

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
* enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
* change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''
* modify qemu NIC parameter: change the -net nic option to include '''model=virtio'''
Following is one example
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -drive file=/home/source.img,if=virtio -drive file=/home/build.img,if=virtio

== Enable VTd to for NIC/Block device ==
Enable VTd with KVM can directed assign PCI device(like NIC or USB disk) to guest, then achieve almost native performance. Pls. make sure your chipset and bios support VTd.
Details is to be coming soon:)

BKM: improve qemu performance

2011-12-06T13:34:30Z

Gzhai: /* Enable virtio block and network device */

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
* enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
* change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''
* modify qemu NIC parameter: change the -net nic option to include '''model=virtio'''
Following is one example
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -drive file=/home/source.img,if=virtio -drive file=/home/build.img,if=virtio

BKM: improve qemu performance

2011-12-06T13:26:54Z

Gzhai:

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
* enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
* change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''

BKM: improve qemu performance

2011-12-06T13:24:44Z

Gzhai:

[[Link title]]= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== Enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

== Create network environment ==
Default qemu in yocto talk with host via simple 192.168.7.X, you can use bridge to make the guest act as one true machine, E.g download source code from outside. Pls. see http://www.linux-kvm.org/page/Networking#public_bridge for details

== Enable virtio block and network device ==
Virtio (block/network) device is a para-virtualized device for kvm guest. It is different from normal emulated hard drive, because it is simply faster.
# enable yocto kernel configuration
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_BLK=y
# change qemu block device parameter from "-hda <your_disk_image" to '''-drive file=<your_disk_image>,if=virtio'''

BKM: improve qemu performance

2011-12-06T13:14:25Z

Gzhai:

BKM: improve qemu performance

2011-12-06T08:31:45Z

Gzhai:

= improve qemu performance =
Qemu in yocto is emulator, and is slow when running huge task inside. E.g. running yocto build inside qemu. This article illustrate possible performance improvement to accelerate it

== Enable KVM ==
If you have processor with VTx support, you can enable KVM so that virtualization rather than emulation is used for performance. Pls. refer following:
* [[How to enable KVM for Poky qemu]]

== Add big memory ==
Sometimes, workload inside qemu requires huge memory. Fail to do this lead performance drop. So pls. add enough memory to qemu via '''-m 2048''' (2G).

== enable SMP ==
If workload inside qemu is CPU-intensive, you can enable smp:
* enable smp configuration in yocto kernel
* enable smp option for qemu, like '''-smp 4''' to give 4 vcpu to guest.

== Add extra disk space ==
If workload inside qemu is disk-intensive, it may requires huge disk space. You can set up NFS server or add extra disk image, which is preferred as NFS server introduce extra network workload.
Following steps to create a new disk image(10G) with ext3 file system:
$ sudo dd if=/dev/zero of=b.img bs=1G count=10
$ sudo losetup /dev/loop0 b.img
$ sudo fdisk /dev/loop0
Then create one new single partition
$ sudo mkfs.ext3 /dev/loop0
to create ext3 file system
$ sudo losetup -d /dev/loop0

append this disk image to qemu via '''-hdb'''(need further changes if need virtio block device)

BKM: improve qemu performance

2011-12-06T06:44:01Z

Gzhai:

BKM: improve qemu performance

2011-12-06T06:01:36Z

Gzhai:

BKM: improve qemu performance

2011-12-06T05:58:40Z

Gzhai:

BKM: improve qemu performance

2011-12-06T05:53:09Z

Gzhai:

BKM: improve qemu performance

2011-12-06T05:52:15Z

Gzhai:

BKM: improve qemu performance

2011-12-06T05:43:44Z

Gzhai:

BKM: improve qemu performance

2011-12-06T05:38:37Z

Gzhai: Created page with "* How to enable KVM for Poky qemu test"

* [[How to enable KVM for Poky qemu]]

test

BSPs

2011-12-06T05:38:01Z

Gzhai:

* [[Yocto BSP One-Stop Shop (Documentation Overview, Getting Started, FAQs, and more)]]
* [[Poky Contributions]]
* [[Poky NFS Root]]
* [[Wind River Kernel]]
* [[Merging Packages from OpenEmbedded]]
* [[How to turn on Poky Audio on Netbook]]
* [[How to Build Target Application in the Host Machine]]
* [[How to enable KVM for Poky qemu]]
* [[Transcript: from git checkout to qemu desktop]]
* [[Transcript: from git checkout to meta-intel BSP]]
* [[BKM: starting a new BSP]]
* [[Transcript: creating one generic Atom BSP from another]]
* [[BKM: improve qemu performance]]

Enable sstate cache

2011-02-17T14:53:53Z

Gzhai: /* know issues */

== sstate background ==
To speed up the build process, sstate provides a cache mechanism, where sstate files from server can be reused to avoid build from scratch if the producer and consumer of the sstate has same environment. With perfect case, we can achieve more than 80% time decreasing.

== use sstate cache server ==
Another build machine is needed to produce the sstate file periodically. The default sstate files directory is '''sstate-cache''' under build dir, which is needed to be exported via NFS, http or ftp server.

On your local machine, change the conf/local.conf as following:

SSTATE_MIRRORS ?= "\
file://.* http://someserver.tld/share/sstate/"

for NFS server
SSTATE_MIRRORS ?= "\
file://.* file:///local/mounted/dir/sstate/"

== Use local sstate cache ==

You can make all the builds pointing to the same SSTATE_DIR to share sstate files between them, like this:

SSTATE_DIR ?= "/share/sstate-cache

So that each build would consume the sstate file if match, and produce new sstate file if not match.

'''Pls. do not start multiple builds at one time, because of race condition issue'''

== Verify the sstate is working ==
If sstate works, you can see something like:

NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 118 of 155 (virtual:native:/home/lulianhao/poky-build/edwin/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot_setscene)
NOTE: Running setscene task 119 of 155 (/home/lulianhao/poky-build/edwin/poky/meta/recipes-devtools/quilt/quilt-native_0.48.bb:do_populate_sysroot_setscene)

If there is no setscene tasks running, sstate cache can't be used because sstate function got broken or some environment value changes invalidate sstate cache. You can use following tools to check any changes between sstate cache and new produced one.

#bitbake-diffsigs /mnt/sstate/sstate-packagename-checksum1-tgz.siginfo builddir/sstate-cache/sstate-packagename-checksum2-tgz.siginfo

== know issues ==
* Various build distributions may lead to checksum mismatch
* New changeset may invalidate whole sstate cache occasionally
* Native package’s sstate cache can’t be reused among 32bit and 64bit host
* Missing sstate file on http/ftp server cause wget hang for a long time due to the retries and timeout

Enable sstate cache

2011-02-17T14:51:55Z

Gzhai: /* know issues */

Enable sstate cache

2011-02-17T14:51:18Z

Gzhai: /* know issues */

Enable sstate cache

2011-02-17T14:50:59Z

Gzhai: /* know issues */

Enable sstate cache

2011-02-17T14:48:47Z

Gzhai: /* Verify the sstate is working */

Enable sstate cache

2011-02-17T14:47:37Z

Gzhai: /* Verify the sstate is working */

Enable sstate cache

2011-02-17T14:37:32Z

Gzhai: /* Verify the sstate is working */

Enable sstate cache

2011-02-17T14:32:25Z

Gzhai: /* Verify the sstate is working */

Enable sstate cache

2011-02-17T14:26:13Z

Gzhai: /* Use local sstate cache */

Enable sstate cache

2011-02-17T14:16:31Z

Gzhai: /* Use local sstate cache */

Enable sstate cache

2011-02-17T14:15:01Z

Gzhai: /* Use local sstate cache */

Enable sstate cache

2011-02-17T14:12:40Z

Gzhai: /* Use local sstate cache */

Enable sstate cache

2011-02-17T14:08:36Z

Gzhai:

Enable sstate cache

2011-02-17T13:31:02Z

Gzhai: /* sstate background */

Enable sstate cache

2011-02-17T13:28:19Z

Gzhai: /* */

Enable sstate cache

2011-02-17T13:26:35Z

Gzhai: Created page with '== sstate background == To speed up the build process, sstate provides a cache mechanism, where sstate files from server can be reused to avoid build from scratch if the producer…'

Processes and Activities

2011-02-17T12:51:36Z

Gzhai:

Welcome to the Processes and Activities Page!

* [[Best Known Methods (BKMs) for Package Updating]]
* [[Working Behind a Network Proxy]]
* [[SDK Generator]]
* [[QA]]
* [[Kernel]]
* [[Core]]
* [[Bugzilla Configuration and Bug Tracking]]
* [[Yocto Release Engineering]]
* [[Performance]]
* [[Program Management Plan]]
* [[Enable sstate cache]]

How to enable KVM for Poky qemu

2010-12-22T06:04:01Z

Gzhai: /* Change the kvm dev ownership for non-root user */

= enable KVM for poky qemu =
== KVM introduction ==

''KVM'' (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, '''kvm.ko''', that provides the core virtualization infrastructure and a processor specific module, '''kvm-intel.ko''' or kvm-amd.ko. KVM also requires a modified QEMU although work is underway to get the required changes upstream.

Compared with native qemu, a pure emulator, KVM has better performance based on virtualization as most of guest instruction can be executed directly on the host processor.

== detect VT support ==
You need make sure your x86 processor support VT before using KVM.

With a recent enough Linux kernel, run the command:

$ egrep '^flags.*(vmx|svm)' /proc/cpuinfo

If something shows up, you have VT. You can also check the processor model name (in `/proc/cpuinfo`) in the vendor's web site.

Note:

* You'll never see (vmx|svm) in /proc/cpuinfo if you're currently running in in a dom0 or domU. The Xen hypervisor suppresses these flags in order to prevent hijacking.

* Some manufacturers disable VT in the machine's BIOS, in such a way that it cannot be re-enabled.

* `/proc/cpuinfo` only shows virtualization capabilities starting with Linux 2.6.15 (Intel) and Linux 2.6.16 (AMD). Use the `uname -r` command to query your kernel version.

In case of doubt, contact your hardware vendor.

== get the KVM module ==

The quick & easy way for this is just using it from your distribution. As of now, all major community and enterprise distributions contain kvm kernel modules; these are either installed by default or require installing a kvm package. For someone looking for stability, these are the best choice. No effort is needed to build or install the modules, support is provided by the distribution, and the distribution/module combination is well tested.

In other cases, you need refer [http://www.linux-kvm.org/page/Getting_the_kvm_kernel_modules getting kvm modules].

After getting the modules, you can install them into kernel by

$ sudo modprobe kvm-intel

== make KVM aware QEMU ==
Upstream QEMU has already supported KVM, and I have already checked in one patch to enable it. So you get a KVM capable qemu after poky build.

== Change the kvm dev ownership for non-root user ==
qemu is started as non-root user in poky, but ''/dev/kvm'' from some distribution remains root:root that only allow root to use KVM. To work around this, we need create a new group named '''kvm''', and make both ''/dev/kvm'' and the non-root user belongs to it.

$ sudo addgroup --system kvm
$ sudo adduser $USER kvm
$ sudo chown root:kvm /dev/kvm
$ sudo chmod 0660 /dev/kvm

The output of "ls -l /dev/kvm" should be like this:
crw-rw----+ 1 root kvm 10, 232 2010-07-02 09:27 /dev/kvm

On a system that runs udev, you will probably need to add the following line somewhere in your udev configuration so it will automatically give the right group to the newly created device (i-e for ubuntu add a line to /etc/udev/rules.d/40-permissions.rules).

KERNEL=="kvm", GROUP="kvm"

Note
* You need log out then log in the non-root user to take the effect
* Some distributions have already made this, so that we can skip this step

== running qemu with KVM enabled ==
just append "kvm" parameter to the ''poky-qemu'' like this

$ poky-qemu qemux86 kvm

How to enable KVM for Poky qemu

2010-12-22T01:20:56Z

Gzhai: Created page with '= enable KVM for poky qemu = == KVM introduction == ''KVM'' (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualizat…'

= enable KVM for poky qemu =
== KVM introduction ==

''KVM'' (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, '''kvm.ko''', that provides the core virtualization infrastructure and a processor specific module, '''kvm-intel.ko''' or kvm-amd.ko. KVM also requires a modified QEMU although work is underway to get the required changes upstream.

Compared with native qemu, a pure emulator, KVM has better performance based on virtualization as most of guest instruction can be executed directly on the host processor.

== detect VT support ==
You need make sure your x86 processor support VT before using KVM.

With a recent enough Linux kernel, run the command:

$ egrep '^flags.*(vmx|svm)' /proc/cpuinfo

If something shows up, you have VT. You can also check the processor model name (in `/proc/cpuinfo`) in the vendor's web site.

Note:

* You'll never see (vmx|svm) in /proc/cpuinfo if you're currently running in in a dom0 or domU. The Xen hypervisor suppresses these flags in order to prevent hijacking.

* Some manufacturers disable VT in the machine's BIOS, in such a way that it cannot be re-enabled.

* `/proc/cpuinfo` only shows virtualization capabilities starting with Linux 2.6.15 (Intel) and Linux 2.6.16 (AMD). Use the `uname -r` command to query your kernel version.

In case of doubt, contact your hardware vendor.

== get the KVM module ==

The quick & easy way for this is just using it from your distribution. As of now, all major community and enterprise distributions contain kvm kernel modules; these are either installed by default or require installing a kvm package. For someone looking for stability, these are the best choice. No effort is needed to build or install the modules, support is provided by the distribution, and the distribution/module combination is well tested.

In other cases, you need refer [http://www.linux-kvm.org/page/Getting_the_kvm_kernel_modules getting kvm modules].

After getting the modules, you can install them into kernel by

$ sudo modprobe kvm-intel

== make KVM aware QEMU ==
Upstream QEMU has already supported KVM, and I have already checked in one patch to enable it. So you get a KVM capable qemu after poky build.

== Change the kvm dev ownership for non-root user ==
qemu is started as non-root user in poky, but ''/dev/kvm'' from some distribution remains root:root that only allow root to use KVM. To work around this, we need create a new group named '''kvm''', and make both ''/dev/kvm'' and the non-root user belongs to it.

$ sudo addgroup --system kvm
$ sudo adduser $USER kvm
$ sudo chown root:kvm /dev/kvm
$ sudo chmod 0660 /dev/kvm

The output of "ls -l /dev/kvm" should be like this:
crw-rw----+ 1 root kvm 10, 232 2010-07-02 09:27 /dev/kvm

Note
* You need log out then log in the non-root user to take the effect
* Some distributions have already made this, so that we can skip this step

== running qemu with KVM enabled ==
just append "kvm" parameter to the ''poky-qemu'' like this

$ poky-qemu qemux86 kvm

BSPs

2010-12-22T01:17:25Z

Gzhai:

* [[Poky Contributions]]
* [[Poky NFS Root]]
* [[Wind River Kernel]]
* [[Merging Packages from OpenEmbedded]]
* [[How to turn on Poky Audio on Netbook]]
* [[How to Build Target Application in the Host Machine]]
* [[How to enable KVM for Poky qemu]]