Documentation/dev-tools/ktap.rst

1 .. SPDX-License-Identifier: GPL-2.0
12 (KTAP) format is specified to extend and alter TAP to support these use-cases.
17 can have subtests), each of which can contain both diagnostic data -- e.g., log
18 lines -- and a final result. The test structure and results are
19 machine-readable, whereas the diagnostic data is unstructured and is there to
22 KTAP output is built from four different types of lines:
23 - Version lines
24 - Plan lines
25 - Test case result lines
26 - Diagnostic lines
34 Version lines
35 -------------
37 All KTAP-formatted results begin with a "version line" which specifies which
41 - "KTAP version 1"
42 - "TAP version 13"
43 - "TAP version 14"
53 Plan lines
54 ----------
58 Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
59 Plan lines follow version lines to indicate the number of nested tests.
61 While there are cases where the number of tests is not known in advance -- in
62 which case the test plan may be omitted -- it is strongly recommended one is
65 Test case result lines
66 ----------------------
68 Test case result lines indicate the final status of a test.
71 .. code-block:: none
73 	<result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
86 The directive and any diagnostic data is optional. If either are present, they
91 keyword preceding the diagnostic data. In the event that a parser encounters
97 - "SKIP", which indicates a test was skipped (note the result of the test case
99 - "TODO", which indicates that a test is not expected to pass at the moment,
102 - "XFAIL", which indicates that a test is expected to fail. This is similar
104 - “TIMEOUT”, which indicates a test has timed out (note the result of the test
106 - “ERROR”, which indicates that the execution of a test has failed due to a
107   specific error that is included in the diagnostic data. (note the result of
110 The diagnostic data is a plain-text field which contains any additional details
114 The diagnostic data field is optional, and results which have neither a
115 directive nor any diagnostic data do not need to include the "#" field
118 Example result lines include::
141 The test "test" timed out, with diagnostic data "30 seconds".
147 The test "check return code" passed, with additional diagnostic data “rcode=0”
150 Diagnostic lines
151 ----------------
154 "diagnostic lines". Diagnostic lines are optional, freeform text, and are
156 more detail than the final result and diagnostic data line provides.
158 Diagnostic lines are formatted as "# <diagnostic_description>", where the
159 description can be any string.  Diagnostic lines can be anywhere in the test
160 output. As a rule, diagnostic lines regarding a test are directly before the
163 Note that most tools will treat unknown lines (see below) as diagnostic lines,
168 Unknown lines
169 -------------
171 There may be lines within KTAP output that do not follow the format of one of
172 the four formats for lines described above. This is allowed, however, they will
178 messages from non-test code that is invoked by the test.  The kernel code
183 ------------
186 output an entire set of KTAP-formatted results. This can be used to categorize
194 Additionally, all lines in a subtest should be indented. One level of
198 "Unknown lines" are not considered to be lines in a subtest and thus are
233 --------------------------------------
241 "Unknown lines" are in category of "Anything else"   yes        no
242 "Unknown lines" are                                  incorrect  allowed
250 --------------------
279 - "example_test_1", which passes, and has one subtest:
281    - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1"
283 - "example_test_2", which passes, and has two subtests:
285    - "test_1", which is skipped, with the explanation "test_1 skipped"
286    - "test_2", which passes
288 - "example_test_3", which fails, and has three subtests
290    - "test_1", which passes
291    - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.
292    - "test_3", which is skipped with the explanation "test_3 skipped"
302 ---------
304 - The TAP specification:
305   https://testanything.org/tap-version-13-specification.html
306 - The (stagnant) TAP version 14 specification:
307   https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
308 - The kselftest documentation:
309   Documentation/dev-tools/kselftest.rst
310 - The KUnit documentation:
311   Documentation/dev-tools/kunit/index.rst