Checkbox certification software usage (Ubuntu 22.04)
Checkbox is a flexible test automation software. It’s the main tool used in Ubuntu Certification program. This document describes the usage of Checkbox on Ubuntu 22.04, but it can work on older versions of Ubuntu as well.
To run the maximum number of tests, it is advised to connect as many peripheral devices as possible. Before running the checkbox, connect peripherals to available connectors, such as:
- card reader
- USB ports
- display ports
- Ethernet ports
- headphone and microphone jacks
- Thunderbolt ports
- power supply and battery
- PCMCIA or ExpressCard slot
- DIMM/RAM slots
- SATA ports
- PCI/PCIe slots
- M.2 slots
- TPM header
- other slots
Set up the testing environment
For the tests that don't require any interference after their startup, perform the following steps:
To stop the screen from being locked on suspend, open the terminal and run the following command:
gsettings set org.gnome.desktop.lockdown disable-lock-screen 'true'
To execute all
sudocommands without a password, open the terminal and run the following command for a user:
echo "user ALL=(ALL:ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/user
To set up automatic login so that a password is not required during testing reboot, power off etc.:
- Press the
SUPER_KEY, type in
Usersand click Enter. A window with options to change should appear.
- Click the user name you want to enable automatic login for.
- On the top right side of the window, click
- You will be asked to enter your user password to continue.
- You should now be able to toggle the
Automatic Loginbutton to enable it.
- Restart your system for changes to take effect.
- Press the
Download and install Checkbox
Disable Secure Boot. To do this you can follow the steps described in Secure Boot test.
Disable Secure boot is required to properly download packages.
Open the terminal and type in the below commands to install Checkbox and all its dependencies:
sudo add-apt-repository ppa:hardware-certification/public sudo apt-get update sudo apt-get install checkbox-ng plainbox-provider-resource-generic \ plainbox-provider-certification-client plainbox-provider-checkbox \ canonical-certification-client
Run automatic validation
Start checkbox. To do this, open the terminal and run the following command:
After using the above-mentioned command, the following menu should appear:
┌──────────────────────────────────────────────────────────────────────────────┐ │ (X) (Deprecated) Fully Automatic Client Certification Tests │ │ ( ) After suspend Dock Cert blocker tests │ │ ( ) After suspend Dock Cert tests │ │ ( ) After suspend Dock Hot-plug Cert tests │ │ ( ) After suspend Dock Hot-plug tests │ │ ( ) After suspend LED and oops tests (using special sleep key) │ │ ( ) After suspend LED and oops tests (using special sleep key, cert. │ │ blockers only) │ │ ( ) After suspend automated USB 3 write/read/compare tests on storage │ │ devices │ │ ( ) After suspend automated USB write/read/compare tests on storage │ │ devices │ │ ( ) After suspend reference tests │ │ ( ) After suspend reference tests (automated) │ │ ( ) After suspend reference tests (certification blockers only) │ │ ( ) After suspend tests (discrete GPU automated) │ │ ( ) After suspend tests (discrete GPU manual) │ │ ( ) After suspend tests (discrete GPU) │ │ ( ) After suspend tests (discrete GPU, certification blockers only) | . . . etc. └──────────────────────────────────────────────────────────────────────────────┘
Select test suites (
Enter- go to the next test suite), which you want to run. Marking the first option starts the full automated check. After selecting the first option, all modules should be displayed and you can manually uncheck some modules, which you don't want to test.
┌──────────────────────────────────────────────────────────────────────────────┐ │[X] + Audio tests │ │[X] + Benchmarks tests │ │[X] + Bluetooth tests │ │[X] + CPU tests │ │[X] + Camera tests │ │[X] + Disk tests │ │[X] + Ethernet Device tests │ │[X] + Firmware tests │ │[X] + Graphics tests │ │[X] + Informational tests │ │[X] + Input Devices tests │ │[X] + Memory tests │ │[X] + Miscellaneous tests │ │[X] + Mobile broadband tests │ │[X] + Non-device specific networking tests │ │[X] + Optical Drive tests │ │[X] + Power Management tests │ │[X] + Suspend tests │ │[X] + TPM 2.0 (Trusted Platform Module) │ │[X] + Touchpad tests │ . . . etc. └──────────────────────────────────────────────────────────────────────────────┘
Tto start the testing procedure.
You might be asked about the hardware in your machine. Just tick it truthfully and press
┌──────────────────────────────────────────────────────────────────────────────┐ │Does this machine have this piece of hardware? │ │ An Ethernet Port (X) YES ( ) NO │ │ Camera/Capture Device (X) YES ( ) NO │ │ USB Storage Device Connected (X) YES ( ) NO │ └──────────────────────────────────────────────────────────────────────────────┘
When testing ends, a menu with test results should appear. Press
Rto rerun test cases, or
After all, paths to the test results should be displayed in the terminal. By default, they are placed in
runlets you run a particular test plan or a set of jobs, but it does save any results. Example of running one test plan:
sudo checkbox-cli run com.canonical.certification::smoke
launchercommand lets you customize the checkbox experience. To use it, you need to create
config_file.iniand start running the checkbox like below:
checkbox-cli launcher config_file.ini
For more details about these (and others) commands visit checkbox-cli.
When somehow the checkbox stop working you can resume the previous session. To do this, open the terminal and run the following command:
If at least one incomplete session is present on the machine, you should see:
Do you want to resume session 'session_title-2022-07-06T13.09.22'? r => resume this session n => next session c => create new session d => delete old sessions [rncd]:
r to resume the stopped session. You can also create a new session or
delete an old session by typing the appropriate letter.
What do you want to do with that job? s => skip that job p => mark it as passed and continue f => mark it as failed and continue r => run it [spfr]:
Decide what you want to do with the last test and type the appropriate letter. After this, your checkbox session will be resumed.