Lines Matching +full:fault +full:- +full:inject
2 Fault injection capabilities infrastructure
5 See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
8 Available fault injection capabilities
9 --------------------------------------
11 - failslab
15 - fail_page_alloc
19 - fail_usercopy
23 - fail_futex
25 injects futex deadlock and uaddr fault errors.
27 - fail_sunrpc
31 - fail_make_request
34 /sys/block/<device>/make-it-fail or
35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
37 - fail_mmc_request
42 - fail_function
48 - NVMe fault injection
50 inject NVMe status code and retry flag on devices permitted by setting
55 - Null test block driver fault injection
57 inject IO timeouts by setting config items under
59 inject requeue requests by setting config items under
61 inject init_hctx() errors by setting config items under
64 Configure fault-injection capabilities behavior
65 -----------------------------------------------
70 fault-inject-debugfs kernel module provides some debugfs entries for runtime
71 configuration of fault-injection capabilities.
73 - /sys/kernel/debug/fail*/probability:
79 Note that one-failure-per-hundred is a very high error rate
83 - /sys/kernel/debug/fail*/interval:
91 - /sys/kernel/debug/fail*/times:
93 specifies how many times failures may happen at most. A value of -1
96 - /sys/kernel/debug/fail*/space:
102 - /sys/kernel/debug/fail*/verbose
108 log line per failure; '2' will print a call trace too -- useful
109 to debug the problems revealed by fault injection.
111 - /sys/kernel/debug/fail*/task-filter:
117 /proc/<pid>/make-it-fail==1.
119 - /sys/kernel/debug/fail*/require-start,
120 /sys/kernel/debug/fail*/require-end,
121 /sys/kernel/debug/fail*/reject-start,
122 /sys/kernel/debug/fail*/reject-end:
131 - /sys/kernel/debug/fail*/stacktrace-depth:
134 for a caller within [require-start,require-end) OR
135 [reject-start,reject-end).
137 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
141 default is 'Y', setting it to 'N' will also inject failures into
144 - /sys/kernel/debug/failslab/cache-filter
147 default is 'N', setting it to 'Y' will only inject failures when
152 - /sys/kernel/debug/failslab/ignore-gfp-wait:
153 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
157 default is 'Y', setting it to 'N' will also inject failures
160 - /sys/kernel/debug/fail_page_alloc/min-order:
165 - /sys/kernel/debug/fail_futex/ignore-private:
172 - /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
179 - /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
186 - /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
193 - /sys/kernel/debug/fail_function/inject:
195 Format: { 'function-name' | '!function-name' | '' }
202 - /sys/kernel/debug/fail_function/injectable:
207 - NULL: retval must be 0.
208 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
209 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
211 - /sys/kernel/debug/fail_function/<function-name>/retval:
213 specifies the "error" return value to inject to the given function.
217 $ printf %#x -12 > retval
222 In order to inject faults while debugfs is not available (early boot time),
235 - /proc/<pid>/fail-nth,
236 /proc/self/task/<tid>/fail-nth:
238 Write to this file of integer N makes N-th call in the task fail.
240 that the fault setup with a previous write to this file was injected.
241 A positive integer N indicates that the fault wasn't yet injected.
244 like probability, interval, times, etc. But per-capability settings
245 (e.g. fail_futex/ignore-private) take precedence over it.
252 --------------------------
260 Since the function-level error injection forcibly changes the code path
265 - The function returns an error code if it fails, and the callers must check
268 - The function does not execute any code which can change any state before
275 (free objects) functions are usually harder to inject errors than allocate
291 There are 4 types of errors defined in include/asm-generic/error-injection.h
298 This function will return an `-errno` error code if it fails. e.g. return
299 -EINVAL if the input is wrong. This will include the functions which will
300 return an address which encodes `-errno` by ERR_PTR() macro.
303 This function will return an `-errno` or `NULL` if it fails. If the caller
308 This function will return `true` (non-zero positive value) if it fails.
315 How to add new fault injection capability
316 -----------------------------------------
318 - #include <linux/fault-inject.h>
320 - define the fault attributes
324 Please see the definition of struct fault_attr in fault-inject.h
327 - provide a way to configure fault attributes
329 - boot option
331 If you need to enable the fault injection capability from boot time, you can
336 - debugfs entries
343 - module parameters
345 If the scope of the fault injection capability is limited to a
347 configure the fault attributes.
349 - add a hook to insert failures
351 Upon should_fail() returning true, client code should inject a failure:
356 --------------------
358 - Inject slab allocation failures into module init/exit code::
363 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
366 echo -1 > /sys/kernel/debug/$FAILTYPE/times
369 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
373 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
376 if [ $# -eq 0 ]
388 faulty_system modprobe -r $m
391 ------------------------------------------------------------------------------
393 - Inject page allocation failures only for a specific module::
400 if [ -z $module ]
408 if [ ! -d /sys/module/$module/sections ]
414 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
415 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
417 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
420 echo -1 > /sys/kernel/debug/$FAILTYPE/times
423 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
424 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
425 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
432 ------------------------------------------------------------------------------
434 - Inject open_ctree error while btrfs mount::
438 rm -f testfile.img
440 DEVICE=$(losetup --show -f testfile.img)
441 mkfs.btrfs -f $DEVICE
442 mkdir -p tmpmnt
446 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
447 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
448 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
451 echo -1 > /sys/kernel/debug/$FAILTYPE/times
455 mount -t btrfs $DEVICE tmpmnt
456 if [ $? -ne 0 ]
464 echo > /sys/kernel/debug/$FAILTYPE/inject
467 losetup -d $DEVICE
470 ------------------------------------------------------------------------------
472 - Inject only skbuff allocation failures ::
477 echo 1 > /sys/kernel/debug/failslab/cache-filter
478 # Turn on fault injection
484 ----------------------------------------------------
486 tools/testing/fault-injection/failcmd.sh. Please run a command
487 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
492 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
495 # ./tools/testing/fault-injection/failcmd.sh \
496 -- make -C tools/testing/selftests/ run_tests
501 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
502 -- make -C tools/testing/selftests/ run_tests
504 Same as above except to inject page allocation failure instead of slab
508 ./tools/testing/fault-injection/failcmd.sh --times=100 \
509 -- make -C tools/testing/selftests/ run_tests
511 Systematic faults using fail-nth
512 ---------------------------------
514 The following code systematically faults 0-th, 1-st, 2-nd and so on
533 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
534 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
546 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
556 1-th fault Y: res=-1/23
557 2-th fault Y: res=-1/23
558 3-th fault Y: res=-1/12
559 4-th fault Y: res=-1/12
560 5-th fault Y: res=-1/23
561 6-th fault Y: res=-1/23
562 7-th fault Y: res=-1/23
563 8-th fault Y: res=-1/12
564 9-th fault Y: res=-1/12
565 10-th fault Y: res=-1/12
566 11-th fault Y: res=-1/12
567 12-th fault Y: res=-1/12
568 13-th fault Y: res=-1/12
569 14-th fault Y: res=-1/12
570 15-th fault Y: res=-1/12
571 16-th fault N: res=0/12