Joshua Lock: Add to do items before this article is ready

2017-02-15T12:10:20Z

Add to do items before this article is ready

← Older revision		Revision as of 12:10, 15 February 2017
Line 44:		Line 44:

	Armed with this knowledge we were then able to look at the class where oe_multilib_header is defined and see that indeed, the ''oe_multilib_header'' function does reference ''MIPSPKGSFX_ABI''. Furthermore there's already a -native override of the function which simply returns with a comment noting that this prevents architecture specific variables affecting the sstate signature of -native recipes. The fix then for this changing sstate signature was to replicate the empty function override for -nativesdk recipes too.		Armed with this knowledge we were then able to look at the class where oe_multilib_header is defined and see that indeed, the ''oe_multilib_header'' function does reference ''MIPSPKGSFX_ABI''. Furthermore there's already a -native override of the function which simply returns with a comment noting that this prevents architecture specific variables affecting the sstate signature of -native recipes. The fix then for this changing sstate signature was to replicate the empty function override for -nativesdk recipes too.


			== TODO ==
			* clean up & edit
			* talk about the common "ERROR: Taskhash mismatch dc3543e66b52acc379ec340f0c3f7703 verses 581ae57e28e43d6c41e72240a2c01f04 for ..."

Joshua Lock: use -S to dump signatures

2016-09-30T13:45:54Z

use -S to dump signatures

← Older revision		Revision as of 13:45, 30 September 2016
Line 4:		Line 4:

	Signature data was generated for the differing signatures with:		Signature data was generated for the differing signatures with:
	$ MACHINE=qemux86-64 bitbake nativesdk-glibc-initial		$ MACHINE=qemux86-64 bitbake -S none nativesdk-glibc-initial
	$ MACHINE=qemumips64 bitbake nativesdk-glibc-initial		$ MACHINE=qemumips64 bitbake -S none nativesdk-glibc-initial

	I then used ls on the stamps directory to see which tasks were rerun, and pick the		I then used ls on the stamps directory to see which tasks were rerun, and pick the

Joshua Lock: First draft of a tips & tricks discussing bitbake-diffsigs and bitbake-dumpsig

2016-09-30T13:17:26Z

First draft of a tips & tricks discussing bitbake-diffsigs and bitbake-dumpsig

New page

shared state checksum changes can result in tasks being re-run when one would expect the existing sstate objects to be used. BitBake provides several tools for debugging these scenarios to learn what's different between the signatures of tasks.

A recent example from OE-core was that the signature of some nativesdk tasks was changing when changing target ''MACHINE'' [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10320 YOCTO #10320], this was narrowed down to happening when switching between a MIPS machine and any other.

Signature data was generated for the differing signatures with:
$ MACHINE=qemux86-64 bitbake nativesdk-glibc-initial
$ MACHINE=qemumips64 bitbake nativesdk-glibc-initial

I then used ls on the stamps directory to see which tasks were rerun, and pick the

$ ls tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/*.sigdata.*

I can see that the there are two sets of signature data (sigdata) for several tasks, including the ''do_install'' task (2.24-r0.do_install.sigdata.[CHECKSUM]).

Now that I've identified an early task which has a different signature and therefore invalidates sstate object reuse I can use <tt>bitbake-diffsigs</tt> to compare the signature files for the two different task invocation. <tt>bitbake-diffsigs</tt> can be invoked with specific sigdata files passed as arguments, but I'll instead invoke it with the recipename and taskname and let the tool determine the two most recent signatures to compare:

<nowiki>
$ bitbake-diffsigs -t nativesdk-glibc-initial do_install
basehash changed from 18d262a7dd0189dfa8df63a7915417db to 5ef4c25ef31eece9f00f5a670a092182
Variable MIPSPKGSFX_ABI value changed from 'None' to '${@bb.utils.contains('TUNE_FEATURES', 'n32', '-n32', '', d)}
TUNE_FEATURES{n32} = Unset'
</nowiki>

Here we can see that the variable ''MIPSPKGSFX_ABI'' is changing between the two different task signatures. This is a Mips specific variable which only affects target recipes, why is it changing for nativesdk recipes?

To answer that question we can use another tool, <tt>bitbake-dumpsigs</tt>, to look at the inputs affecting the signature and how ''MIPSPKGSFX_ABI'' is used. Often context is very useful when looking at <tt>bitbake-dumpsigs</tt> output so it's recommended to pipe the output through a pager which supports searching:

$ bitbake-dumpsig tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/2.24-r0.do_install.sigdata.b127a5b3176daffc0fb90391370b6844 | less

Alternatively we might even just pipe the output through grep for the variable. In this instance that's enough to give us a lead, though the relevant item is a little obscured in the output:

<nowiki>$ bitbake-dumpsig tmp/stamps/x86_64-nativesdk-pokysdk-linux/nativesdk-glibc-initial/2.24-r0.do_install.sigdata.b127a5b3176daffc0fb90391370b6844 | grep MIPSPKGSFX_ABI
Task dependencies: ['ACLOCALDIR', 'AR', 'AS', 'ASNEEDED', 'B', 'BUILDSDK_CFLAGS', 'BUILDSDK_CPPFLAGS', 'BUILDSDK_LDFLAGS', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_OS', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'BUILD_SYS', 'BUILD_VENDOR', 'CC', 'CCLD', 'CC_FOR_BUILD', 'CFLAGS', 'CFLAGS_FOR_BUILD', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CPPFLAGS_FOR_BUILD', 'CPP_FOR_BUILD', 'CXX', 'CXXFLAGS', 'CXXFLAGS_FOR_BUILD', 'CXX_FOR_BUILD', 'D', 'DEBUG_BUILD', 'DEBUG_FLAGS', 'DEBUG_OPTIMIZATION', 'DEBUG_PREFIX_MAP', 'DISTRO', 'EXTENDPE', 'EXTRA_OEMAKE', 'FC', 'FULL_OPTIMIZATION', 'HOST_ARCH', 'HOST_AS_ARCH', 'HOST_CC_ARCH', 'HOST_LD_ARCH', 'HOST_OS', 'HOST_PREFIX', 'INIT_D_DIR', 'LC_ALL', 'LD', 'LDFLAGS', 'LDFLAGS_FOR_BUILD', 'LD_FOR_BUILD', 'LINKER_HASH_STYLE', 'LOGFIFO', 'MAKE', 'MIPSPKGSFX_ABI', 'NM', 'OBJCOPY', 'OBJDUMP', 'PE', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PN', 'PR', 'PSEUDO_DISABLED', 'PV', 'RANLIB', 'READELF', 'S', 'SDKPATH', 'SDKPATHNATIVE', 'SDK_ARCH', 'SDK_AS_ARCH', 'SDK_CC_ARCH', 'SDK_LD_ARCH', 'SDK_OS', 'SDK_PREFIX', 'SDK_SYS', 'SDK_VENDOR', 'SDK_VERSION', 'SELECTED_OPTIMIZATION', 'SITEINFO_BITS', 'SITEINFO_EXTRA_DATAFUNCS', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DIR', 'STAGING_DIR_NATIVE', 'STAGING_DIR_TCBOOTSTRAP', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STRINGS', 'STRIP', 'T', 'TARGET_ARCH', 'TARGET_CFLAGS', 'TARGET_CPPFLAGS', 'TARGET_CXXFLAGS', 'TARGET_LDFLAGS', 'TARGET_LINK_HASH_STYLE', 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TOOLCHAIN_OPTIONS', 'USE_LDCONFIG', 'USRBINPATH', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'baselib', 'bberror', 'bbfatal_log', 'bbnote', 'bindir', 'datadir', 'die', 'do_install[fakeroot]', 'do_install[umask]', 'docdir', 'exec_prefix', 'ident', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe_multilib_header', 'oe_runmake', 'oe_runmake_call', 'oldincludedir', 'prefix', 'prefix_native', 'prefix_nativesdk', 'rm_systemd_unitdir', 'rm_sysvinit_initddir', 'sbindir', 'servicedir', 'sharedstatedir', 'siteinfo_data', 'siteinfo_get_files', 'stem', 'sysconfdir', 'systemd_system_unitdir', 'systemd_unitdir', 'systemd_user_unitdir']
List of dependencies for variable oe_multilib_header is {'CPP', 'PSEUDO_DISABLED', 'LDFLAGS_FOR_BUILD', 'BUILD_AR', 'OBJDUMP', 'CFLAGS', 'SITEINFO_BITS', 'BUILD_CPPFLAGS', 'HOST_OS', 'infodir', 'LD_FOR_BUILD', 'systemd_unitdir', 'CXX_FOR_BUILD', 'libdir', 'RANLIB', 'datadir', 'nonarch_base_libdir', 'D', 'CPPFLAGS', 'docdir', 'base_prefix', 'base_sbindir', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'OBJCOPY', 'CCLD', 'LD', 'BUILD_LDFLAGS', 'PKG_CONFIG_LIBDIR', 'servicedir', 'TARGET_CPPFLAGS', 'BUILD_STRIP', 'CPP_FOR_BUILD', 'TARGET_CFLAGS', 'includedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'LDFLAGS', 'AR', 'CXXFLAGS', 'BUILD_CXXFLAGS', 'BUILD_FC', 'READELF', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'sharedstatedir', 'TARGET_ARCH', 'bindir', 'systemd_user_unitdir', 'PKG_CONFIG_DIR', 'CXXFLAGS_FOR_BUILD', 'FC', 'CONFIG_SITE', 'BUILD_NM', 'STRINGS', 'BUILD_CCLD', 'exec_prefix', 'libexecdir', 'AS', 'sbindir', 'CFLAGS_FOR_BUILD', 'BUILD_LD', 'CC', 'localstatedir', 'CXX', 'STRIP', 'stem', 'bberror', 'nonarch_libdir', 'NM', 'BUILD_RANLIB', 'systemd_system_unitdir', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'CC_FOR_BUILD', 'TARGET_CXXFLAGS', 'base_libdir', 'BUILD_CXX', 'CPPFLAGS_FOR_BUILD', 'oldincludedir', 'BUILD_AS', 'prefix', 'sysconfdir', 'mandir', 'PKG_CONFIG_PATH', 'BUILD_CC', 'ident', 'MAKE', 'TARGET_LDFLAGS', 'LC_ALL', 'BUILD_CPP', 'BUILD_CFLAGS', 'base_bindir', 'MIPSPKGSFX_ABI'}
List of dependencies for variable MIPSPKGSFX_ABI is set()
mips*) case "${MIPSPKGSFX_ABI}" in
Variable MIPSPKGSFX_ABI value is ${@bb.utils.contains('TUNE_FEATURES', 'n32', '-n32', '', d)}
</nowiki>

We see that the ''oe_multilib_header'' variable (a function provided by ''multilib_header.bbclass'') depends on the ''MIPSPKGSFX_ABI'' variable. Here's the relevant part of the <tt>bitbake-dumpsig</tt> output:

<nowiki>List of dependencies for variable oe_multilib_header is {'CPP', 'PSEUDO_DISABLED', 'LDFLAGS_FOR_BUILD', 'BUILD_AR', 'OBJDUMP', 'CFLAGS', 'SITEINFO_BITS', 'BUILD_CPPFLAGS', 'HOST_OS', 'infodir', 'LD_FOR_BUILD', 'systemd_unitdir', 'CXX_FOR_BUILD', 'libdir', 'RANLIB', 'datadir', 'nonarch_base_libdir', 'D', 'CPPFLAGS', 'docdir', 'base_prefix', 'base_sbindir', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'OBJCOPY', 'CCLD', 'LD', 'BUILD_LDFLAGS', 'PKG_CONFIG_LIBDIR', 'servicedir', 'TARGET_CPPFLAGS', 'BUILD_STRIP', 'CPP_FOR_BUILD', 'TARGET_CFLAGS', 'includedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'LDFLAGS', 'AR', 'CXXFLAGS', 'BUILD_CXXFLAGS', 'BUILD_FC', 'READELF', 'PKG_CONFIG_SYSTEM_INCLUDE_PATH', 'sharedstatedir', 'TARGET_ARCH', 'bindir', 'systemd_user_unitdir', 'PKG_CONFIG_DIR', 'CXXFLAGS_FOR_BUILD', 'FC', 'CONFIG_SITE', 'BUILD_NM', 'STRINGS', 'BUILD_CCLD', 'exec_prefix', 'libexecdir', 'AS', 'sbindir', 'CFLAGS_FOR_BUILD', 'BUILD_LD', 'CC', 'localstatedir', 'CXX', 'STRIP', 'stem', 'bberror', 'nonarch_libdir', 'NM', 'BUILD_RANLIB', 'systemd_system_unitdir', 'PKG_CONFIG_SYSTEM_LIBRARY_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'CC_FOR_BUILD', 'TARGET_CXXFLAGS', 'base_libdir', 'BUILD_CXX', 'CPPFLAGS_FOR_BUILD', 'oldincludedir', 'BUILD_AS', 'prefix', 'sysconfdir', 'mandir', 'PKG_CONFIG_PATH', 'BUILD_CC', 'ident', 'MAKE', 'TARGET_LDFLAGS', 'LC_ALL', 'BUILD_CPP', 'BUILD_CFLAGS', 'base_bindir', 'MIPSPKGSFX_ABI'}
</nowiki>

Armed with this knowledge we were then able to look at the class where oe_multilib_header is defined and see that indeed, the ''oe_multilib_header'' function does reference ''MIPSPKGSFX_ABI''. Furthermore there's already a -native override of the function which simply returns with a comment noting that this prevents architecture specific variables affecting the sstate signature of -native recipes. The fix then for this changing sstate signature was to replicate the empty function override for -nativesdk recipes too.

TipsAndTricks/Understanding what changed (diffsigs etc) - Revision history

Joshua Lock: Add to do items before this article is ready

Joshua Lock: use -S to dump signatures

Joshua Lock: First draft of a tips & tricks discussing bitbake-diffsigs and bitbake-dumpsig