LAVA server vs Yocto HW automation testing: Difference between revisions
From Yocto Project
Jump to navigationJump to search
No edit summary |
No edit summary |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
{| class="wikitable" | {| class="wikitable" | ||
| style="background-color: #87CEEB; text-align:center;" ! colspan="4" | '''Lava server vs Yocto hardware automation''' | | style="background-color: #87CEEB; text-align:center;" ! colspan="4" | '''Lava server vs Yocto hardware automation testing''' | ||
|- | |- | ||
| style="background-color: #ffffcc; width: 40%" | '''Tested steps''' | | style="background-color: #ffffcc; text-align:center; width: 40%" | '''Tested steps''' | ||
| style="background-color: #ffffcc; | | style="background-color: #ffffcc; text-align:center;" | '''Lava''' | ||
| style="background-color: #ffffcc; | | style="background-color: #ffffcc; text-align:center;" | '''Yocto''' | ||
| style="background-color: #ffffcc;" | '''Comments''' | | style="background-color: #ffffcc; text-align:center;" | '''Comments''' | ||
|- | |- | ||
| Controlling a board via serial connection | | Controlling a board via serial connection | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| | | | ||
|- | |- | ||
| Booting a board via serial connection and bootloader using Yocto images artifacts | | Booting a board via serial connection and bootloader using Yocto images artifacts | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| | | | ||
|- | |- | ||
| Able to schedule a test jobs to run at certain time | | Able to schedule a test jobs to run at certain time | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| "Cron can be used in addition with lava CLI commands to trigger a test job" | | "Cron can be used in addition with lava CLI commands to trigger a test job" | ||
|- | |- | ||
| Able to perform a full automated test cycle for a non serial connection board | | Able to perform a full automated test cycle for a non serial connection board | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| "There is an issue with the test job which is exiting right after the master image ‘reboot’ command in order to boot into the test image as the master image ssh connection is lost and cannot run the next configu#FF6347 steps from the test job.Opened an enhancement for this at linaro but they told me that this feature will not be implemented for the dummy-ssh devices (https://bugs.linaro.org/show_bug.cgi?id=478)" | | "There is an issue with the test job which is exiting right after the master image ‘reboot’ command in order to boot into the test image as the master image ssh connection is lost and cannot run the next configu#FF6347 steps from the test job.Opened an enhancement for this at linaro but they told me that this feature will not be implemented for the dummy-ssh devices (https://bugs.linaro.org/show_bug.cgi?id=478)" | ||
|- | |- | ||
| Able to view the test jobs results as tables | | Able to view the test jobs results as tables | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| | | | ||
|- | |- | ||
| Able to view the test jobs results as graphical reports | | Able to view the test jobs results as graphical reports | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| | | | ||
|- | |- | ||
| Able to query old test jobs results | | Able to query old test jobs results | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| | | | ||
|- | |- | ||
| Can use existing connection scripts for the tested boards | | Can use existing connection scripts for the tested boards | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| | | | ||
|- | |- | ||
| Does not require additional packages for master image | | Does not require additional packages for master image | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| Aditional packages needs for master image for ssh - sshfs connection | | Aditional packages needs for master image for ssh - sshfs connection | ||
|- | |- | ||
| Can use existing test definitions | | Can use existing test definitions | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| Runtime tests definitions needs to be manually added | | Runtime tests definitions needs to be manually added | ||
|- | |- | ||
| Can use existing test jobs | | Can use existing test jobs | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| Tests jobs needs to be manually created | | Tests jobs needs to be manually created | ||
|- | |- | ||
| Web based interface support | | Web based interface support | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| | | | ||
|- | |- | ||
| Can accept test results from external sources | | Can accept test results from external sources | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| There is no way to submit results from the tested bsp to the Lava server | | There is no way to submit results from the tested bsp to the Lava server | ||
|- | |- | ||
| Authentication user support for the web based interface | | Authentication user support for the web based interface | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| | | | ||
|- | |- | ||
| Able to use Yocto images artifacts | | Able to use Yocto images artifacts | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| | | | ||
|- | |- | ||
| Can use the existing master image configurations | | Can use the existing master image configurations | ||
| style=" background-color: #FF6347;" | N | | style=" background-color: #FF6347; text-align:center;" | N | ||
| style=" background-color: lime;" | Y | | style=" background-color: lime; text-align:center;" | Y | ||
| "We need to generate a pair of authentication keys between lava server and the master image – password less authentication from lava-server to the master image – root is requi#FF6347;Master image and test image must have the .ssh/authorized_keys configu#FF6347." | | "We need to generate a pair of authentication keys between lava server and the master image – password less authentication from lava-server to the master image – root is requi#FF6347;Master image and test image must have the .ssh/authorized_keys configu#FF6347." | ||
|} | |} | ||
Latest revision as of 13:16, 12 September 2014
| Lava server vs Yocto hardware automation testing | |||
| Tested steps | Lava | Yocto | Comments |
| Controlling a board via serial connection | Y | Y | |
| Booting a board via serial connection and bootloader using Yocto images artifacts | Y | Y | |
| Able to schedule a test jobs to run at certain time | N | Y | "Cron can be used in addition with lava CLI commands to trigger a test job" |
| Able to perform a full automated test cycle for a non serial connection board | N | Y | "There is an issue with the test job which is exiting right after the master image ‘reboot’ command in order to boot into the test image as the master image ssh connection is lost and cannot run the next configu#FF6347 steps from the test job.Opened an enhancement for this at linaro but they told me that this feature will not be implemented for the dummy-ssh devices (https://bugs.linaro.org/show_bug.cgi?id=478)" |
| Able to view the test jobs results as tables | Y | N | |
| Able to view the test jobs results as graphical reports | Y | N | |
| Able to query old test jobs results | Y | N | |
| Can use existing connection scripts for the tested boards | N | Y | |
| Does not require additional packages for master image | N | Y | Aditional packages needs for master image for ssh - sshfs connection |
| Can use existing test definitions | N | Y | Runtime tests definitions needs to be manually added |
| Can use existing test jobs | N | Y | Tests jobs needs to be manually created |
| Web based interface support | Y | N | |
| Can accept test results from external sources | N | Y | There is no way to submit results from the tested bsp to the Lava server |
| Authentication user support for the web based interface | Y | N | |
| Able to use Yocto images artifacts | Y | Y | |
| Can use the existing master image configurations | N | Y | "We need to generate a pair of authentication keys between lava server and the master image – password less authentication from lava-server to the master image – root is requi#FF6347;Master image and test image must have the .ssh/authorized_keys configu#FF6347." |
