Patchtest: Difference between revisions

From Yocto Project
Jump to navigationJump to search
mNo edit summary
No edit summary
 
(9 intermediate revisions by 2 users not shown)
Line 3: Line 3:
Patchtest is a tool designed for improving the quality of Yocto Project and OpenEmbedded contributors' patch submissions, while simultaneously reducing the level of review effort required from project maintainers. It does this by providing a set of scripts that users can test their patches with prior to submission ("host" mode), and which can be used as part of an automated service that provides periodic feedback for patches received by Patchwork ("guest" mode).
Patchtest is a tool designed for improving the quality of Yocto Project and OpenEmbedded contributors' patch submissions, while simultaneously reducing the level of review effort required from project maintainers. It does this by providing a set of scripts that users can test their patches with prior to submission ("host" mode), and which can be used as part of an automated service that provides periodic feedback for patches received by Patchwork ("guest" mode).


=== Using Patchtest===
=== Using Patchtest ===


See the [https://git.yoctoproject.org/poky/tree/scripts/patchtest.README patchtest README] for instructions on how to use patchtest in your local workspace, and the [https://git.yoctoproject.org/meta-patchtest/tree/README.md meta-patchtest README] for details on automated/guest mode setup.
See the [https://git.yoctoproject.org/poky/tree/scripts/patchtest.README patchtest README] for instructions on how to use patchtest in your local workspace, and the [https://git.yoctoproject.org/meta-patchtest/tree/README.md meta-patchtest README] for details on automated/guest mode setup.
Line 11: Line 11:
The patchtest feedback email is designed to provide concise instructions when one or more of the tests has failed, but additional information on specific failures is provided below:
The patchtest feedback email is designed to provide concise instructions when one or more of the tests has failed, but additional information on specific failures is provided below:


==== LIC_FILES_CHKSUM changed without "License-Update:" tag and description in commit message ====
{| class="wikitable" style="margin:auto"
|-
! Testcase !! Details !! Further information
|-
| test_upstream_status_presence_format || This testcase checks if newly introduced *.patch files contain a valid upstream status, if not included the testcase fails || https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=upstream+status#patch-upstream-status
|-
| test_signed_off_by_presence || This testcase checks if newly introduced *.patch files contain a Signed-off-by, if not included the testcase fails  || https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=signed+off#patch-upstream-status
|-
| test_cve_tag_format ||  Whenever a submission adds a CVE patch, patchtest will check both the patch '''and the parent mbox's commit message''' for a "CVE: CVE-YYYY-XXXX" tag. While this has historically not been required, including the tag in both locations aids review and maintenance. || https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=signed+off#cve-patches
|-
| test_signed_off_by_presence || This testcase checks if the tested patch contains a valid Signed-off-by || If it fails you can add it manually or with "git commit --amend -s", See also https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
|-
| test_shortlog_format || This testcases checks the format of the shortlog (header line in the commit message). The expected format is: <target>: <summary>, where component is a recipe name or the short form path to the file being changed || https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
|-
| test_shortlog_length || This failure indicates that the shortlog (i.e. the subject line) for the patch is so long that it may not be easily readable on some mail clients. Some subject prefixes (e.g. the branch name when submitting a patch for an older release) may unavoidably inflate the length, but the shortlog should be kept as short and concise as possible so that it is clear what the patch does and what it changes. Extra information can always be provided in the commit message. || https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
|-
| test_series_merge_on_head || Checks if series applies on top of target branch. Temporary disabled, but ensure your series applies cleanly || https://docs.yoctoproject.org/overview-manual/development-environment.html?highlight=git#git
|-
| test_target_mailing_list || Checks if series is sent to the correct mailing list for all files in the series. Poky is a combo layer combining bitbake, openembedded-core, meta-yocto and yocto-docs. Please create and submit patches against the individual components. || https://docs.yoctoproject.org/contributor-guide/identify-component.html
|-
| test_mbox_format || Checks if the patch has no malformed diff lines and can be applied cleanly. ||  Create the series again using git-format-patch and ensure it applies using git am. See also: https://docs.yoctoproject.org/overview-manual/development-environment.html?highlight=git#git
|-
| test_commit_message_presence || This testcases fails if a submission has an empty commit message. Provide for example detailed information on what you changed and why, how you tested it.  || https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
|-
| test_bugzilla_entry_format || Checks that the Bugzilla issue ID, if mentioned in the commit message, is correctly formatted. Format is: "[YOCTO #<bugzilla ID>]. || https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes 
|-
| test_author_valid || A patch authored by an invalid author was detected. || You can amend the author via git commit --amend
|-
| test_non_auh_upgrade || A patch authored by the Auto-Upgrade Helper (AUH) was detected. These patches are meant to provide maintainers with a sanity check for recipe upgrades and a starting point for testing them, so patches submitted with that authorship are marked as failures to indicate that they have not been fully validated. || For a overview of AUH see also: https://docs.yoctoproject.org/dev/dev-manual/upgrading-recipes.html#using-the-auto-upgrade-helper-auh
|-
| test_pylint || Fails if patch contains .py files and pylint detected new issues if the patch is applied || Check changes with pylint and amend patch
|-
| test_license_presence || Checks to ensure that LICENSE is set for a newly added recipe. || https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-LICENSE
|-
| test_lic_files_chksum_presence || Checks to ensure that LIC_FILES_CHKSUM is present, if not then this test will fail. || https://docs.yoctoproject.org/dev/dev-manual/licenses.html#tracking-license-changes
|-
| test_lic_files_chksum_modified_not_mentioned || Checks to ensure that any changes made to LIC_FILES_CHKSUM lines (including checksums for new files) are accounted for in the commit message with at least one "License-Update: <reason>" tag. If this is not present, then this test will fail. ||  https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=license+update#license-updates
|-
| test_max_line_length || Checks if any line in the meta-data changed in series contains no lines that are longer that 200 characters. || Please ensure that lines do not exceed this length and amend your patch 
|-
| test_src_uri_left_files || Checks if all files removed from SRC_URI are also removed from the tree. || Please remove files that are not included in the SRC_URI from the tree as wll, e.g. with git rm <file>
|-
| test_summary_presence || Checks if SUMMARY is set for newly added recipes. Should be set to a short description of the package created by the recipe. || https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SUMMARY 
|-
| test_cve_check_ignore || Fails if the deprecated CVE_CHECK_INGNORE is used instead of CVE_STATUS. || https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-CVE_STATUS
|}


Patchtest checks to ensure that any changes made to LIC_FILES_CHKSUM lines (including checksums for new files) are accounted for in the commit message with at least one "License-Update: <reason>" tag. If this is not present, then this test will fail.


==== A CVE tag should be provided in the commit message with format: "CVE: CVE-YYYY-XXXX" ====


Whenever a submission adds a CVE patch, patchtest will check both the patch '''and the parent mbox's commit message''' for a "CVE: CVE-YYYY-XXXX" tag. While this has historically not been required, including the tag in both locations improves review and maintenance.


==== Please include a commit message on your patch explaining the change ====
==== Tests failed for the patch, but the results log could not be processed due to excessive result line length. ====
 
This response is received whenever a submission has an empty commit message.
 
==== Edit shortlog so that it is 90 characters or less (currently X characters) ====


This failure indicates that the shortlog (i.e. the subject line) for the patch is so long that it may not be easily readable on some mail clients. Some subject prefixes (e.g. the branch name when submitting a patch for an older release) may unavoidably inflate the length, but the shortlog should be kept as short and concise as possible so that it is clear what the patch does and what it changes. Extra information can always be provided in the commit message.
The patchtest-send-results script checks the test results' maximum line length to help ensure that no unintended and/or malicious code is sent to the mailing list with the results. If this message is present, the user should re-test their patch(es) locally using the '''patchtest''' script in openembedded-core/scripts before further escalation. If the issue persists after re-testing and re-submission it should be reported on the Yocto Project [https://bugzilla.yoctoproject.org/ Bugzilla] under the "Yocto Project Subprojects" -> "Patchtest" category with a "Major" priority (include the patch file as an attachment).


==== Invalid author auh@auh.yoctoproject.org. Resend the series with a valid patch author ====
=== Submitting Bug Reports ===


A patch authored by the Auto-Upgrade Helper (AUH) was detected. These patches are meant to provide maintainers with a sanity check for recipe upgrades and a starting point for testing them, so patches submitted with that authorship are marked as failures to indicate that they have not been fully validated.
If you suspect there is an issue with the patchtest feedback emails, the service itself, or any of the documentation, please submit an issue on the Yocto Project Bugzilla [https://bugzilla.yoctoproject.org/ Bugzilla] under the "Yocto Project Subprojects" -> "Patchtest" category.


==== Tests failed for the patch, but the results log could not be processed due to excessive result line length. ====
=== Patchtest Architecture Notes ===


The patchtest-send-results script checks the test results' maximum line length to help ensure that no unintended and/or malicious code is sent to the mailing list with the results. If this message is present, the user should re-test their patch(es) locally using the '''patchtest''' script in openembedded-core/scripts before further escalation. If the issue persists after re-testing and re-submission it should be reported as a [https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Patchwork%2FPatchtest patchtest bug] with a "Major" priority (include the patch file as an attachment).
{| class="wikitable" style="margin:align-left"
|+ Test Suite Details
|-
! Test Name !! Category !! Uses Tinfoil !! Notes
|-
| test_upstream_status_presence_format || TestPatch || No ||
|-
| test_signed_off_by_presence || TestPatch || No || Same test name as similar test in TestMbox
|-
| test_cve_tag_format || TestPatch || No ||
|-
| test_signed_off_by_presence || TestMbox || No || Same test name as similar test in TestPatch
|-
| test_shortlog_format || TestMbox || No ||
|-
| test_shortlog_length || TestMbox || No ||
|-
| test_series_merge_on_head || TestMbox || No || Temporarily disabled
|-
| test_target_mailing_list || TestMbox || No ||
|-
| test_mbox_format || TestMbox || No ||
|-
| test_commit_message_presence || TestMbox || No ||
|-
| test_bugzilla_entry_format || TestMbox || No ||
|-
| test_author_valid || TestMbox || No ||
|-
| test_non_auh_upgrade || TestMbox || No ||
|-
| test_pylint || PyLint || No ||
|-
| test_license_presence || TestMetadata || Yes ||
|-
| test_lic_files_chksum_presence || TestMetadata || Yes ||
|-
| test_lic_files_chksum_modified_not_mentioned || TestMetadata || No ||
|-
| test_max_line_length || TestMetadata || No ||
|-
| test_src_uri_left_files || TestMetadata || Yes || Has a linked pretest: pretest_src_uri_left_files
|-
| test_summary_presence || TestMetadata || Yes ||
|-
| test_cve_check_ignore || TestMetadata || Yes ||
|}

Latest revision as of 13:45, 20 February 2024

Patchtest

Patchtest is a tool designed for improving the quality of Yocto Project and OpenEmbedded contributors' patch submissions, while simultaneously reducing the level of review effort required from project maintainers. It does this by providing a set of scripts that users can test their patches with prior to submission ("host" mode), and which can be used as part of an automated service that provides periodic feedback for patches received by Patchwork ("guest" mode).

Using Patchtest

See the patchtest README for instructions on how to use patchtest in your local workspace, and the meta-patchtest README for details on automated/guest mode setup.

Patchtest "FAIL:" Descriptions

The patchtest feedback email is designed to provide concise instructions when one or more of the tests has failed, but additional information on specific failures is provided below:

Testcase Details Further information
test_upstream_status_presence_format This testcase checks if newly introduced *.patch files contain a valid upstream status, if not included the testcase fails https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=upstream+status#patch-upstream-status
test_signed_off_by_presence This testcase checks if newly introduced *.patch files contain a Signed-off-by, if not included the testcase fails https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=signed+off#patch-upstream-status
test_cve_tag_format Whenever a submission adds a CVE patch, patchtest will check both the patch and the parent mbox's commit message for a "CVE: CVE-YYYY-XXXX" tag. While this has historically not been required, including the tag in both locations aids review and maintenance. https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=signed+off#cve-patches
test_signed_off_by_presence This testcase checks if the tested patch contains a valid Signed-off-by If it fails you can add it manually or with "git commit --amend -s", See also https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
test_shortlog_format This testcases checks the format of the shortlog (header line in the commit message). The expected format is: <target>: <summary>, where component is a recipe name or the short form path to the file being changed https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
test_shortlog_length This failure indicates that the shortlog (i.e. the subject line) for the patch is so long that it may not be easily readable on some mail clients. Some subject prefixes (e.g. the branch name when submitting a patch for an older release) may unavoidably inflate the length, but the shortlog should be kept as short and concise as possible so that it is clear what the patch does and what it changes. Extra information can always be provided in the commit message. https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
test_series_merge_on_head Checks if series applies on top of target branch. Temporary disabled, but ensure your series applies cleanly https://docs.yoctoproject.org/overview-manual/development-environment.html?highlight=git#git
test_target_mailing_list Checks if series is sent to the correct mailing list for all files in the series. Poky is a combo layer combining bitbake, openembedded-core, meta-yocto and yocto-docs. Please create and submit patches against the individual components. https://docs.yoctoproject.org/contributor-guide/identify-component.html
test_mbox_format Checks if the patch has no malformed diff lines and can be applied cleanly. Create the series again using git-format-patch and ensure it applies using git am. See also: https://docs.yoctoproject.org/overview-manual/development-environment.html?highlight=git#git
test_commit_message_presence This testcases fails if a submission has an empty commit message. Provide for example detailed information on what you changed and why, how you tested it. https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
test_bugzilla_entry_format Checks that the Bugzilla issue ID, if mentioned in the commit message, is correctly formatted. Format is: "[YOCTO #<bugzilla ID>]. https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html#implement-and-commit-changes
test_author_valid A patch authored by an invalid author was detected. You can amend the author via git commit --amend
test_non_auh_upgrade A patch authored by the Auto-Upgrade Helper (AUH) was detected. These patches are meant to provide maintainers with a sanity check for recipe upgrades and a starting point for testing them, so patches submitted with that authorship are marked as failures to indicate that they have not been fully validated. For a overview of AUH see also: https://docs.yoctoproject.org/dev/dev-manual/upgrading-recipes.html#using-the-auto-upgrade-helper-auh
test_pylint Fails if patch contains .py files and pylint detected new issues if the patch is applied Check changes with pylint and amend patch
test_license_presence Checks to ensure that LICENSE is set for a newly added recipe. https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-LICENSE
test_lic_files_chksum_presence Checks to ensure that LIC_FILES_CHKSUM is present, if not then this test will fail. https://docs.yoctoproject.org/dev/dev-manual/licenses.html#tracking-license-changes
test_lic_files_chksum_modified_not_mentioned Checks to ensure that any changes made to LIC_FILES_CHKSUM lines (including checksums for new files) are accounted for in the commit message with at least one "License-Update: <reason>" tag. If this is not present, then this test will fail. https://docs.yoctoproject.org/dev/contributor-guide/recipe-style-guide.html?highlight=license+update#license-updates
test_max_line_length Checks if any line in the meta-data changed in series contains no lines that are longer that 200 characters. Please ensure that lines do not exceed this length and amend your patch
test_src_uri_left_files Checks if all files removed from SRC_URI are also removed from the tree. Please remove files that are not included in the SRC_URI from the tree as wll, e.g. with git rm <file>
test_summary_presence Checks if SUMMARY is set for newly added recipes. Should be set to a short description of the package created by the recipe. https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SUMMARY
test_cve_check_ignore Fails if the deprecated CVE_CHECK_INGNORE is used instead of CVE_STATUS. https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-CVE_STATUS



Tests failed for the patch, but the results log could not be processed due to excessive result line length.

The patchtest-send-results script checks the test results' maximum line length to help ensure that no unintended and/or malicious code is sent to the mailing list with the results. If this message is present, the user should re-test their patch(es) locally using the patchtest script in openembedded-core/scripts before further escalation. If the issue persists after re-testing and re-submission it should be reported on the Yocto Project Bugzilla under the "Yocto Project Subprojects" -> "Patchtest" category with a "Major" priority (include the patch file as an attachment).

Submitting Bug Reports

If you suspect there is an issue with the patchtest feedback emails, the service itself, or any of the documentation, please submit an issue on the Yocto Project Bugzilla Bugzilla under the "Yocto Project Subprojects" -> "Patchtest" category.

Patchtest Architecture Notes

Test Suite Details
Test Name Category Uses Tinfoil Notes
test_upstream_status_presence_format TestPatch No
test_signed_off_by_presence TestPatch No Same test name as similar test in TestMbox
test_cve_tag_format TestPatch No
test_signed_off_by_presence TestMbox No Same test name as similar test in TestPatch
test_shortlog_format TestMbox No
test_shortlog_length TestMbox No
test_series_merge_on_head TestMbox No Temporarily disabled
test_target_mailing_list TestMbox No
test_mbox_format TestMbox No
test_commit_message_presence TestMbox No
test_bugzilla_entry_format TestMbox No
test_author_valid TestMbox No
test_non_auh_upgrade TestMbox No
test_pylint PyLint No
test_license_presence TestMetadata Yes
test_lic_files_chksum_presence TestMetadata Yes
test_lic_files_chksum_modified_not_mentioned TestMetadata No
test_max_line_length TestMetadata No
test_src_uri_left_files TestMetadata Yes Has a linked pretest: pretest_src_uri_left_files
test_summary_presence TestMetadata Yes
test_cve_check_ignore TestMetadata Yes