Weekly Status: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 10: Line 10:


'''Key Status/Updates:'''<br/>
'''Key Status/Updates:'''<br/>
*YP 3.4 M1 has been through QA pending release approval with two QA issues highlighted.
*YP 3.1.9 has been through QA and is awaiting release approval, no blocking issues were identified.
*YP 3.1.9 is being built ready for QA.
*YP 3.4 M1 was released.
*Big thanks to Paul Gortmaker for tracking down the cause of an LTP null pointer dereference (and other errors) within the cgroups mount code which was responsible for several of our LTP hangs. The issue was introduced in new code between 5.0 and 5.1 in the kernel and had been present for a while. The fix is now making its way through various kernel trees upstream.
*Significantly for the project, we have now identified a kernel hang caused from RCU stalls which has been plaguing our autobuilder testing for months. The issue is still open in upstream mainline kernels but there is a patch queued in the rcu tree for it. Since applying this patch to linux-yocto, our builds have been noticeably greener and more stable. We have closed out many of the related AB-INT bugs and will be creating new ones for any remaining issues as they occur since it is hard to tell some other bugs apart from the RCU issue.
*Sadly, the above fix did not resolve the “rcu” autobuilder VM hangs we are seeing occasionally. These are odd in that they affect kvm and non-kvm builds (x86 and arm seen on x86-64 hosts), they pin the VM at 300-400% CPU usage and the VM will respond to pings but no ssh or console output. The rcu dumps from the kernel are likely a symptom that something is wrong rather than the cause and often look incomplete. It is as if some instantaneous host load breaks timers in a way the guest cannot recover or continue execution from. We’re continuing to try and narrow this down but it is proving elusive and progress is slow, any insight anyone may have would be welcome.
*The prserv rewrite to use asyncio is stuck on autobuilder hangs when testing at larger scale. We’d welcome help from anyone with more details asyncio knowledge/experience as it is looking like the new code isn’t entirely stable in python 3.5/3.6 and may have “at exit” issues, as well as perhaps issues mixing async io code on the client side with multiprocessing during parsing.
*There are new manual sections that have recently been added on http://docs.yoctoproject.org/test-manual/reproducible-builds.html and http://docs.yoctoproject.org/test-manual/yocto-project-compatible.html.
*There does still seem to be an arm specific LTP hang reading /proc/kmsg now tracked by bug 14460.
*We continue to deal with an issue with centos8 kernels having what looks like bad bounds checking on the utimensat_time64 32 bit syscalls where the syscall was backported into a kernel point release. We’re working on reporting it upstream.
*We have a 10th anniversary T-shirt and some other Yocto Project items (hoody, stickers, mugs etc.) now available at https://yoctoproject.org/shop (EU and Americas sources)
*The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.
*The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.
*Intermittent autobuilder issues continue to occur and are now at a record high level. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT We are working to identify the load pattern on the infrastructure that seems to trigger these.
*Intermittent autobuilder issues continue to occur, about 50% of the open issues are now ptest failures and the rest are various other races or timeouts. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT


'''Ways to contribute:'''<br/>
'''Ways to contribute:'''<br/>

Revision as of 14:47, 29 June 2021

Yocto Project Weekly Status June 29, 2021

Current Dev Position: YP 3.4 M2
Next Deadline: 12th July 2021 YP 3.4 M2 build

Next Team Meetings:

Key Status/Updates:

  • YP 3.1.9 has been through QA and is awaiting release approval, no blocking issues were identified.
  • YP 3.4 M1 was released.
  • Significantly for the project, we have now identified a kernel hang caused from RCU stalls which has been plaguing our autobuilder testing for months. The issue is still open in upstream mainline kernels but there is a patch queued in the rcu tree for it. Since applying this patch to linux-yocto, our builds have been noticeably greener and more stable. We have closed out many of the related AB-INT bugs and will be creating new ones for any remaining issues as they occur since it is hard to tell some other bugs apart from the RCU issue.
  • The prserv rewrite to use asyncio is stuck on autobuilder hangs when testing at larger scale. We’d welcome help from anyone with more details asyncio knowledge/experience as it is looking like the new code isn’t entirely stable in python 3.5/3.6 and may have “at exit” issues, as well as perhaps issues mixing async io code on the client side with multiprocessing during parsing.
  • There does still seem to be an arm specific LTP hang reading /proc/kmsg now tracked by bug 14460.
  • The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others. Richard is trying to fix but trying to fix autobuilder issues and other problems and these are slow builds to debug.
  • Intermittent autobuilder issues continue to occur, about 50% of the open issues are now ptest failures and the rest are various other races or timeouts. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT

Ways to contribute:

YP 3.4 Milestone Dates:

  • YP 3.4 M1 is released
  • YP 3.4 M2 build date 2021/07/12
  • YP 3.4 M2 Release date 2021/07/23
  • YP 3.4 M3 build date 2021/08/23
  • YP 3.4 M3 Release date 2021/09/03
  • YP 3.4 M4 build date 2021/10/04
  • YP 3.4 M4 Release date 2021/10/29

Planned upcoming dot releases:

  • YP 3.1.9 is under review
  • YP 3.3.2 build date 2021/07/19
  • YP 3.3.2 release date 2021/07/30
  • YP 3.1.10 build date 2021/07/26
  • YP 3.1.10 release date 2021/08/06
  • YP 3.1.11 build date 2021/09/13
  • YP 3.1.11 release date 2021/9/24

Tracking Metrics:

The Yocto Project’s technical governance is through its Technical Steering Committee, more information is available at:
https://wiki.yoctoproject.org/wiki/TSC

Archives

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Weekly_Status&oldid=84784"