12.2. Test Controls¶
The tests in the RTEMS kernel testsuite can be configured for each BSP. The expected state of the test can be set as well as any configuration parameters a test needs to run on a BSP.
The test states are:
passed
The test start and end banners have been sent to the console.
failure
The test start banner has been sent to the console and no end banner has been seen when a target restart is detected.
excepted-fail
The test is tagged as
expected-fail
in the RTEMS sources for this BSP and outputs the banner*** TEST STATE: EXPECTED_FAIL
. The test is known not to pass on this BSP. The RTEMS Tester will let the test run as far as it can and if the test passes it is recorded as a pass in the test results otherwise it is recorded as expected-fail.
indeterminate
The test is tagged as
indeterminate
in the RTEMS sources for this BSP and outputs the banner*** TEST STATE: INDETERMINATE
. The test may or may not pass so the result is not able to be determined. The RTEMS Tester will let the test run as far as it can and record the result as indeterminate.
user-input
The test is tagged as
user-input
in the RTEMS sources and outputs the banner*** TEST STATE: USER_INPUT
. The RTEMS Tester will reset the target if the target’s configuration provides a target reset command.
benchmark
The test is tagged as
benchmark
in the RTEMS sources and outputs the banner*** TEST STATE: BENCHMARK
. Benchmarks can take a while to run and performance is not regression tested in RTEMS. The RTEMS Tester will reset the target if the target’s configuration provides a target reset command.
timeout
The test start banner has been sent to the console and no end banner is seen within the timeout period and the target has not restart. A default timeout can be set in a target configuration, a user configuration or provide on the RTEMS Tester’s command line using the
--timeout
option.
invalid
The test did not output a start banner and the RTEMS Tester has detected the target has restarted. This means the executable did not load correctly, the RTEMS kernel did not initialize or the RTEMS kernel configuration failed for this BSP.
12.2.1. Expected Test States¶
A test’s expected state is set in the RTEMS kernel’s testsuite. The default for
a tested is to pass
. If a test is known to fail it can have it’s state set
to expected-fail
. Setting tests that are known to fail to expected-fail
lets everyone know a failure is not to be countered and consider a regression.
Expected test states are listed in test configuration files
12.2.2. Test Configuration¶
Tests can be configured for each BSP using test configuration files. These
files exist under the spec
directory and have the form tst*.yml
. Global
default test states exist as part of the tests themselves and are applied to
all BSPs. BSPs can provide a test configuration that applies to just that BSP
and these files can include subsets of test configurations.
The configuration supports:
Including test configuration files to allow sharing of common configurations.
Excluding tests from being built that do not build for a BSP.
Setting the test state if it is not
passed
.Specifing a BSP specific build configuration for a test.
The test configuration file format is described in the engineering manual.
Expected test state is configured using set-test-state
action blocks in
build type YAML configuration files. These set-test-state
action blocks
contain one or more sub blocks that each describe a reason
, an expected
state
, and a list of tests
to which the state
and reason
apply.
The reason
is a text description of why the expected test state is being
changed from the default and will typically include a ticket number that further
documents the status of the listed tests.
The state
is one of:
include
The test list is the name of a test configuration file to include
exclude
The tests listed are not build. This can happen if a BSP cannot support a test. For example it does not have enough memory.
expected-fail
The tests listed are set to expected fail. The test will fail on the BSP being built.
user-input
The tests listed require user input to run and are not supported by automatic testers.
indeterminate
The tests listed may pass or may not, the result is not reliable.
benchmark
The tests listed are benchmarks. Benchmarks are flagged and not left to run to completion because they may take too long.
By default all tests are included, specific excluded tests using the
exclude
state are excluded and cannot be included again.
12.3. Test Builds¶
The test reports the build of RTEMS being tested. The build are:
default
The build is the default. No RTEMS configure options have been used.
posix
The build includes the POSIX API. The RTEMS configure option
--enable-posix
has been used. Thecpuopts.h
defineRTEMS_POSIX
has defined and it true.
smp
The build is an SMP kernel. The RTEMS configure option
--enable-smp
has been used. Thecpuopts.h
defineRTEMS_SMP
has defined and it true.
mp
The build is an MP kernel. The RTEMS configure option
--enable-multiprocessing
has been used. Thecpuopts.h
defineRTEMS_MULTIPROCESSING
has defined and it true.
paravirt
The build is a paravirtualization kernel. The
cpuopts.h
defineRTEMS_PARAVIRT
has defined and it true.
debug
The build includes kernel debugging support. The RTEMS configure option
--enable-debug
has been used. Thecpuopts.h
defineRTEMS_DEBUG
has defined and it true.
profiling
The build include profiling support. The RTEMS configure option
--enable-profiling
has been used. Thecpuopts.h
defineRTEMS_PROFILING
has defined and it true.