Weekly Status: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 10: Line 10:


'''Key Status/Updates:'''<br/>
'''Key Status/Updates:'''<br/>
*YP 3.4 M1 was built and is in the QA process.
*YP 3.4 M1 has been through QA pending release approval with two QA issues highlighted.
*The intermittent ltp hanging issue is continuing to be tracked down, currently thought to be a kernel bug introduced around the 5.0 -> 5.1 kernel version window. We ended up unblocking M1 since it is clear the LTP issue will take longer to unravel. Whilst aufs was initially thought to be a possible cause, it wasn’t the issue.
*YP 3.1.9 is being built ready for QA.
*We are working around the autobuilder centos8 issue which is an issue with the upstream centos kernel where syscalls were backported in the latest version and don’t work properly.
*Big thanks to Paul Gortmaker for tracking down the cause of an LTP null pointer dereference (and other errors) within the cgroups mount code which was responsible for several of our LTP hangs. The issue was introduced in new code between 5.0 and 5.1 in the kernel and had been present for a while. The fix is now making its way through various kernel trees upstream.
*Sadly, the above fix did not resolve the “rcu” autobuilder VM hangs we are seeing occasionally. These are odd in that they affect kvm and non-kvm builds (x86 and arm seen on x86-64 hosts), they pin the VM at 300-400% CPU usage and the VM will respond to pings but no ssh or console output. The rcu dumps from the kernel are likely a symptom that something is wrong rather than the cause and often look incomplete. It is as if some instantaneous host load breaks timers in a way the guest cannot recover or continue execution from. We’re continuing to try and narrow this down but it is proving elusive and progress is slow, any insight anyone may have would be welcome.
There are new manual sections that have recently been added on Reproducible Builds and Yocto Project Compatible.
*We continue to deal with an issue with centos8 kernels having what looks like bad bounds checking on the utimensat_time64 32 bit syscalls where the syscall was backported into a kernel point release. We’re working on reporting it upstream.
*We have a 10th anniversary T-shirt and some other Yocto Project items (hoody, stickers, mugs etc.) now available at https://yoctoproject.org/shop (EU and Americas sources)
*We have a 10th anniversary T-shirt and some other Yocto Project items (hoody, stickers, mugs etc.) now available at https://yoctoproject.org/shop (EU and Americas sources)
*The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.
*The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.

Revision as of 14:47, 22 June 2021

Yocto Project Weekly Status June 22, 2021

Current Dev Position: YP 3.4 M2
Next Deadline: 12th July 2021 YP 3.4 M2 build

Next Team Meetings:

Key Status/Updates:

  • YP 3.4 M1 has been through QA pending release approval with two QA issues highlighted.
  • YP 3.1.9 is being built ready for QA.
  • Big thanks to Paul Gortmaker for tracking down the cause of an LTP null pointer dereference (and other errors) within the cgroups mount code which was responsible for several of our LTP hangs. The issue was introduced in new code between 5.0 and 5.1 in the kernel and had been present for a while. The fix is now making its way through various kernel trees upstream.
  • Sadly, the above fix did not resolve the “rcu” autobuilder VM hangs we are seeing occasionally. These are odd in that they affect kvm and non-kvm builds (x86 and arm seen on x86-64 hosts), they pin the VM at 300-400% CPU usage and the VM will respond to pings but no ssh or console output. The rcu dumps from the kernel are likely a symptom that something is wrong rather than the cause and often look incomplete. It is as if some instantaneous host load breaks timers in a way the guest cannot recover or continue execution from. We’re continuing to try and narrow this down but it is proving elusive and progress is slow, any insight anyone may have would be welcome.

There are new manual sections that have recently been added on Reproducible Builds and Yocto Project Compatible.

  • We continue to deal with an issue with centos8 kernels having what looks like bad bounds checking on the utimensat_time64 32 bit syscalls where the syscall was backported into a kernel point release. We’re working on reporting it upstream.
  • We have a 10th anniversary T-shirt and some other Yocto Project items (hoody, stickers, mugs etc.) now available at https://yoctoproject.org/shop (EU and Americas sources)
  • The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.
  • Intermittent autobuilder issues continue to occur and are now at a record high level. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT We are working to identify the load pattern on the infrastructure that seems to trigger these.

Ways to contribute:

YP 3.4 Milestone Dates:

  • YP 3.4 M1 is in review
  • YP 3.4 M2 build date 2021/07/12
  • YP 3.4 M2 Release date 2021/07/23
  • YP 3.4 M3 build date 2021/08/23
  • YP 3.4 M3 Release date 2021/09/03
  • YP 3.4 M4 build date 2021/10/04
  • YP 3.4 M4 Release date 2021/10/29

Planned upcoming dot releases:

  • YP 3.1.9 is being built
  • YP 3.1.9 release date 2021/07/02
  • YP 3.3.2 build date 2021/07/19
  • YP 3.3.2 release date 2021/07/30
  • YP 3.1.10 build date 2021/07/26
  • YP 3.1.10 release date 2021/08/06
  • YP 3.1.11 build date 2021/09/13
  • YP 3.1.11 release date 2021/9/24

Tracking Metrics:

The Yocto Project’s technical governance is through its Technical Steering Committee, more information is available at:
https://wiki.yoctoproject.org/wiki/TSC

Archives

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Weekly_Status&oldid=84778"