ESDK

From Yocto Project
Revision as of 17:40, 3 November 2023 by Rpurdie (talk | contribs) (Created page with "= eSDK and how it differs from a standard build or SDK = A full build can be daunting to users yet an SDK can have limitations that frustrate users, for example not being able to create an image. The eSDK is designed to bridge the worlds between an SDK user and a full build. == Standard SDK == A standard SDK provides a cross compiler and a sysroot corresponding to a target system which the user can compile against. There is an environment script the user can source an...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

eSDK and how it differs from a standard build or SDK

A full build can be daunting to users yet an SDK can have limitations that frustrate users, for example not being able to create an image. The eSDK is designed to bridge the worlds between an SDK user and a full build.

Standard SDK

A standard SDK provides a cross compiler and a sysroot corresponding to a target system which the user can compile against. There is an environment script the user can source and perhaps some tools are placed into the path to help with execution of things like qemu. The project one tried allowing extension of the SDK with package management but this didn't work well and generally frustrated users. It also led to two ways of doing things like image creation and this meant one of them was invariably broken or functioned slightly differently.

The SDK is usually shipped as a single installer script which prompts the user for an installation location and extracts/configures the SDK to run there.

eSDK

The eSDK was a different approach where the build environment was collected up and shipped as a single installer so the user is provided with something like the SDK but the tools therein could be used for more advanced workflows like image creation. Not all tools are exposed to the user and the build configuration is locked down, specifically to limit the potential rebuild of components and ensure the shipped ones are reused. This meant users should have fast builds and access to more advanced tooling than the original SDK.

An environment script is provided to allow it to function in the same way as the original SDK.

eSDK differences to a full build

  1. Environment and tooling availability
    • esdk sets a number of variables from its initialization script that aid with cross-compiling components directly (e.g. the core use case of SDKs). Normal mode doesn't do that, but recently added meta-ide-support will generate a similar initialization script that will set up the same environment from the normal mode. There are tests and documentation for it.
    • PATH. eSDK has a number of items in PATH that point to various locations inside tmp/sysroots/, collectively they provide the cross-toolchain.
    • eSDK also puts a selection of yocto tools into path - wic, devtool but not bitbake:
    alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
    total 48
    lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 devtool -> ../../../../layers/poky/scripts/devtool
    lrwxrwxrwx 1 alex alex 54 Oct 30 12:52 oe-find-native-sysroot -> ../../../../layers/poky/scripts/oe-find-native-sysroot
    lrwxrwxrwx 1 alex alex 42 Oct 30 12:52 recipetool -> ../../../../layers/poky/scripts/recipetool
    lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 runqemu -> ../../../../layers/poky/scripts/runqemu
    lrwxrwxrwx 1 alex alex 55 Oct 30 12:52 runqemu-addptable2image -> ../../../../layers/poky/scripts/runqemu-addptable2image
    lrwxrwxrwx 1 alex alex 53 Oct 30 12:52 runqemu-export-rootfs -> ../../../../layers/poky/scripts/runqemu-export-rootfs
    lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-extract-sdk -> ../../../../layers/poky/scripts/runqemu-extract-sdk
    lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-gen-tapdevs -> ../../../../layers/poky/scripts/runqemu-gen-tapdevs
    lrwxrwxrwx 1 alex alex 46 Oct 30 12:52 runqemu-ifdown -> ../../../../layers/poky/scripts/runqemu-ifdown
    lrwxrwxrwx 1 alex alex 44 Oct 30 12:52 runqemu-ifup -> ../../../../layers/poky/scripts/runqemu-ifup
    lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd -> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
    lrwxrwxrwx 1 alex alex 35 Oct 30 12:52 wic -> ../../../../layers/poky/scripts/wic
    • 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH. Cross-toolchain can be added by the same environment script made by meta-ide-support as mentioned in 1a.
  2. Configuration (e.g. local.conf).
    • eSDK local.conf is local.conf from the normal mode that was used to produce eSDK, stripped of all comments, and with a bunch of extra settings:
    INHERIT:remove = "buildhistory icecc"
    CONNECTIVITY_CHECK_URIS = ""
    SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
    SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
    BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
    BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
    BUILDCFG_HEADER = ""
    METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
    1. Provide a flag to indicate we are in the EXT_SDK Context
    WITHIN_EXT_SDK = "1"
    SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
    file://universal-4.9/(.*) file://universal-4.8/\1"
    require conf/locked-sigs.inc
    require conf/unlocked-sigs.inc
    • Devtool also has a special configuration:
    alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
    [General]
    bitbake_subdir = layers/poky/bitbake
    init_path = layers/poky/oe-init-build-env
    core_meta_subdir = layers/poky/meta
    [SDK]
    sdk_targets = core-image-minimal
    • There is currently no tooling to add/remove these extras in either esdk mode or normal mode as far as I understand. Their individual purposes and effects are also not exactly clear to me, and need to be investigated one by one.
  3. Locked signatures for sstate
    • the locked-sigs.inc file use used to lock the task signatures to fixed values to prevent anything within the eSDK changing/rebuilding

Setting up a normal mode in a eSDK installation

This is actually pretty easy: rather than sourcing the sdk environment script, source the poky/oe-init-build-env: 

alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .

### Shell environment set up for builds. ###
...

Bitbake will then simply run, albeit with all those extra esdk-specific items in local.conf (which can be easily taken out to achieve a pristine normal mode). There will be some warnings but builds will proceed: 

WARNING: You are using a local hash equivalence server but have
configured an sstate mirror. This will likely mean no sstate will
match from the mirror. You may wish to disable the hash equivalence
use (BB_HASHSERVE), or use a hash equivalence server alongside the
sstate mirror.
...
WARNING: spirv-headers-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
URL git://github.com/KhronosGroup/SPIRV-Headers;protocol=https;branch=main,
attempting MIRRORS if available
WARNING: libepoxy-1.5.10-r0 do_fetch: Failed to fetch URL
git://github.com/anholt/libepoxy;branch=master;protocol=https,
attempting MIRRORS if available
WARNING: spirv-tools-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
URL git://github.com/KhronosGroup/SPIRV-Tools.git;branch=main;protocol=https,
attempting MIRRORS if available
WARNING: shaderc-native-2023.6-r0 do_fetch: Failed to fetch URL
git://github.com/google/shaderc.git;protocol=https;branch=main,
attempting MIRRORS if available

Setting up esdk mode in a normal yocto installation

This is on one hand easy: cross-toolchain and unified sysroots are made available for direct component builds by meta-ide-support and build-sysroots.

On the other hand, setting up the exact same environment as eSDK sets up isn't currently possible: tooling to replicate all the local.conf tweaks (that includes locked-sigs items) would be necessary, at a minimum. A more pedantic approach would also ensure that

  • only the 'allowed' set of tools is in PATH, and not everything from bitbake/bin/ and scripts/
  • environment variables and toolchain sysroots match between what eSDK sets up, and what meta-ide-support sets up
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=ESDK&oldid=86056"