TipsAndTricks/Understanding what changed (diffsigs etc)

shared state checksum changes can result in tasks being re-run when one would expect the existing sstate objects to be used. BitBake provides several tools for debugging these scenarios to learn what's different between the signatures of tasks.

A recent example from OE-core was that the signature of some nativesdk tasks was changing when changing target MACHINE YOCTO #10320, this was narrowed down to happening when switching between a MIPS machine and any other.

Signature data was generated for the differing signatures with:

$ MACHINE=qemux86-64 bitbake -S none nativesdk-glibc-initial
$ MACHINE=qemumips64 bitbake -S none nativesdk-glibc-initial

I then used ls on the stamps directory to see which tasks were rerun, and pick the

$ ls tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/*.sigdata.*

I can see that the there are two sets of signature data (sigdata) for several tasks, including the do_install task (2.24-r0.do_install.sigdata.[CHECKSUM]).

Now that I've identified an early task which has a different signature and therefore invalidates sstate object reuse I can use bitbake-diffsigs to compare the signature files for the two different task invocation. bitbake-diffsigs can be invoked with specific sigdata files passed as arguments, but I'll instead invoke it with the recipename and taskname and let the tool determine the two most recent signatures to compare:

$ bitbake-diffsigs -t nativesdk-glibc-initial do_install
basehash changed from 18d262a7dd0189dfa8df63a7915417db to 5ef4c25ef31eece9f00f5a670a092182
Variable MIPSPKGSFX_ABI value changed from 'None' to '${@bb.utils.contains('TUNE_FEATURES', 'n32', '-n32', '', d)}
TUNE_FEATURES{n32} = Unset'

Here we can see that the variable MIPSPKGSFX_ABI is changing between the two different task signatures. This is a Mips specific variable which only affects target recipes, why is it changing for nativesdk recipes?

To answer that question we can use another tool, bitbake-dumpsigs, to look at the inputs affecting the signature and how MIPSPKGSFX_ABI is used. Often context is very useful when looking at bitbake-dumpsigs output so it's recommended to pipe the output through a pager which supports searching:

$ bitbake-dumpsig tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/2.24-r0.do_install.sigdata.b127a5b3176daffc0fb90391370b6844 | less

Alternatively we might even just pipe the output through grep for the variable. In this instance that's enough to give us a lead, though the relevant item is a little obscured in the output:

$ bitbake-dumpsig tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/2.24-r0.do_install.sigdata.b127a5b3176daffc0fb90391370b6844 | grep MIPSPKGSFX_ABI
Task dependencies: ['ACLOCALDIR', 'AR', 'AS', 'ASNEEDED', 'B', 'BUILDSDK_CFLAGS', 'BUILDSDK_CPPFLAGS', 'BUILDSDK_LDFLAGS', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_OS', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'BUILD_SYS', 'BUILD_VENDOR', 'CC', 'CCLD', 'CC_FOR_BUILD', 'CFLAGS', 'CFLAGS_FOR_BUILD', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CPPFLAGS_FOR_BUILD', 'CPP_FOR_BUILD', 'CXX', 'CXXFLAGS', 'CXXFLAGS_FOR_BUILD', 'CXX_FOR_BUILD', 'D', 'DEBUG_BUILD', 'DEBUG_FLAGS', 'DEBUG_OPTIMIZATION', 'DEBUG_PREFIX_MAP', 'DISTRO', 'EXTENDPE', 'EXTRA_OEMAKE', 'FC', 'FULL_OPTIMIZATION', 'HOST_ARCH', 'HOST_AS_ARCH', 'HOST_CC_ARCH', 'HOST_LD_ARCH', 'HOST_OS', 'HOST_PREFIX', 'INIT_D_DIR', 'LC_ALL', 'LD', 'LDFLAGS', 'LDFLAGS_FOR_BUILD', 'LD_FOR_BUILD', 'LINKER_HASH_STYLE', 'LOGFIFO', 'MAKE', 'MIPSPKGSFX_ABI', 'NM', 'OBJCOPY', 'OBJDUMP', 'PE', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PN', 'PR', 'PSEUDO_DISABLED', 'PV', 'RANLIB', 'READELF', 'S', 'SDKPATH', 'SDKPATHNATIVE', 'SDK_ARCH', 'SDK_AS_ARCH', 'SDK_CC_ARCH', 'SDK_LD_ARCH', 'SDK_OS', 'SDK_PREFIX', 'SDK_SYS', 'SDK_VENDOR', 'SDK_VERSION', 'SELECTED_OPTIMIZATION', 'SITEINFO_BITS', 'SITEINFO_EXTRA_DATAFUNCS', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DIR', 'STAGING_DIR_NATIVE', 'STAGING_DIR_TCBOOTSTRAP', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STRINGS', 'STRIP', 'T', 'TARGET_ARCH', 'TARGET_CFLAGS', 'TARGET_CPPFLAGS', 'TARGET_CXXFLAGS', 'TARGET_LDFLAGS', 'TARGET_LINK_HASH_STYLE', 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TOOLCHAIN_OPTIONS', 'USE_LDCONFIG', 'USRBINPATH', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'baselib', 'bberror', 'bbfatal_log', 'bbnote', 'bindir', 'datadir', 'die', 'do_install[fakeroot]', 'do_install[umask]', 'docdir', 'exec_prefix', 'ident', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe_multilib_header', 'oe_runmake', 'oe_runmake_call', 'oldincludedir', 'prefix', 'prefix_native', 'prefix_nativesdk', 'rm_systemd_unitdir', 'rm_sysvinit_initddir', 'sbindir', 'servicedir', 'sharedstatedir', 'siteinfo_data', 'siteinfo_get_files', 'stem', 'sysconfdir', 'systemd_system_unitdir', 'systemd_unitdir', 'systemd_user_unitdir']
List of dependencies for variable oe_multilib_header is {'CPP', 'PSEUDO_DISABLED', 'LDFLAGS_FOR_BUILD', 'BUILD_AR', 'OBJDUMP', 'CFLAGS', 'SITEINFO_BITS', 'BUILD_CPPFLAGS', 'HOST_OS', 'infodir', 'LD_FOR_BUILD', 'systemd_unitdir', 'CXX_FOR_BUILD', 'libdir', 'RANLIB', 'datadir', 'nonarch_base_libdir', 'D', 'CPPFLAGS', 'docdir', 'base_prefix', 'base_sbindir', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'OBJCOPY', 'CCLD', 'LD', 'BUILD_LDFLAGS', 'PKG_CONFIG_LIBDIR', 'servicedir', 'TARGET_CPPFLAGS', 'BUILD_STRIP', 'CPP_FOR_BUILD', 'TARGET_CFLAGS', 'includedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'LDFLAGS', 'AR', 'CXXFLAGS', 'BUILD_CXXFLAGS', 'BUILD_FC', 'READELF', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'sharedstatedir', 'TARGET_ARCH', 'bindir', 'systemd_user_unitdir', 'PKG_CONFIG_DIR', 'CXXFLAGS_FOR_BUILD', 'FC', 'CONFIG_SITE', 'BUILD_NM', 'STRINGS', 'BUILD_CCLD', 'exec_prefix', 'libexecdir', 'AS', 'sbindir', 'CFLAGS_FOR_BUILD', 'BUILD_LD', 'CC', 'localstatedir', 'CXX', 'STRIP', 'stem', 'bberror', 'nonarch_libdir', 'NM', 'BUILD_RANLIB', 'systemd_system_unitdir', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'CC_FOR_BUILD', 'TARGET_CXXFLAGS', 'base_libdir', 'BUILD_CXX', 'CPPFLAGS_FOR_BUILD', 'oldincludedir', 'BUILD_AS', 'prefix', 'sysconfdir', 'mandir', 'PKG_CONFIG_PATH', 'BUILD_CC', 'ident', 'MAKE', 'TARGET_LDFLAGS', 'LC_ALL', 'BUILD_CPP', 'BUILD_CFLAGS', 'base_bindir', 'MIPSPKGSFX_ABI'}
List of dependencies for variable MIPSPKGSFX_ABI is set()
        mips*)  case "${MIPSPKGSFX_ABI}" in
Variable MIPSPKGSFX_ABI value is ${@bb.utils.contains('TUNE_FEATURES', 'n32', '-n32', '', d)}

We see that the oe_multilib_header variable (a function provided by multilib_header.bbclass) depends on the MIPSPKGSFX_ABI variable. Here's the relevant part of the bitbake-dumpsig output:

List of dependencies for variable oe_multilib_header is {'CPP', 'PSEUDO_DISABLED', 'LDFLAGS_FOR_BUILD', 'BUILD_AR', 'OBJDUMP', 'CFLAGS', 'SITEINFO_BITS', 'BUILD_CPPFLAGS', 'HOST_OS', 'infodir', 'LD_FOR_BUILD', 'systemd_unitdir', 'CXX_FOR_BUILD', 'libdir', 'RANLIB', 'datadir', 'nonarch_base_libdir', 'D', 'CPPFLAGS', 'docdir', 'base_prefix', 'base_sbindir', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'OBJCOPY', 'CCLD', 'LD', 'BUILD_LDFLAGS', 'PKG_CONFIG_LIBDIR', 'servicedir', 'TARGET_CPPFLAGS', 'BUILD_STRIP', 'CPP_FOR_BUILD', 'TARGET_CFLAGS', 'includedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'LDFLAGS', 'AR', 'CXXFLAGS', 'BUILD_CXXFLAGS', 'BUILD_FC', 'READELF', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'sharedstatedir', 'TARGET_ARCH', 'bindir', 'systemd_user_unitdir', 'PKG_CONFIG_DIR', 'CXXFLAGS_FOR_BUILD', 'FC', 'CONFIG_SITE', 'BUILD_NM', 'STRINGS', 'BUILD_CCLD', 'exec_prefix', 'libexecdir', 'AS', 'sbindir', 'CFLAGS_FOR_BUILD', 'BUILD_LD', 'CC', 'localstatedir', 'CXX', 'STRIP', 'stem', 'bberror', 'nonarch_libdir', 'NM', 'BUILD_RANLIB', 'systemd_system_unitdir', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'CC_FOR_BUILD', 'TARGET_CXXFLAGS', 'base_libdir', 'BUILD_CXX', 'CPPFLAGS_FOR_BUILD', 'oldincludedir', 'BUILD_AS', 'prefix', 'sysconfdir', 'mandir', 'PKG_CONFIG_PATH', 'BUILD_CC', 'ident', 'MAKE', 'TARGET_LDFLAGS', 'LC_ALL', 'BUILD_CPP', 'BUILD_CFLAGS', 'base_bindir', 'MIPSPKGSFX_ABI'}

Armed with this knowledge we were then able to look at the class where oe_multilib_header is defined and see that indeed, the oe_multilib_header function does reference MIPSPKGSFX_ABI. Furthermore there's already a -native override of the function which simply returns with a comment noting that this prevents architecture specific variables affecting the sstate signature of -native recipes. The fix then for this changing sstate signature was to replicate the empty function override for -nativesdk recipes too.

TODO

• clean up & edit
• talk about the common "ERROR: Taskhash mismatch dc3543e66b52acc379ec340f0c3f7703 verses 581ae57e28e43d6c41e72240a2c01f04 for ..."