Documentation/admin-guide/kernel-per-CPU-kthreads.rst

2 Reducing OS jitter due to per-cpu kthreads
5 This document lists per-CPU kthreads in the Linux kernel and presents
6 options to control their OS jitter.  Note that non-per-CPU kthreads are
7 not listed here.  To reduce OS jitter from non-per-CPU kthreads, bind
13 -	Documentation/core-api/irq/irq-affinity.rst:  Binding interrupts to sets of CPUs.
15 -	Documentation/admin-guide/cgroup-v1:  Using cgroups to bind tasks to sets of CPUs.
17 -	man taskset:  Using the taskset command to bind tasks to sets
20 -	man sched_setaffinity:  Using the sched_setaffinity() system
23 -	/sys/devices/system/cpu/cpuN/online:  Control CPU N's hotplug state,
26 -	In order to locate kernel-generated OS jitter on CPU N:
41   Periodically process Infiniband-related work.
43 To reduce its OS jitter, do any of the following:
46 	that does not require per-CPU kthreads.  This will prevent these
50 2.	Do all eHCA-Infiniband-related work on other CPUs, including
52 3.	Rework the eHCA driver so that its per-CPU kthreads are
57   irq/%d-%s
62 To reduce its OS jitter, do the following:
73 To reduce its OS jitter, do one of the following:
77 2.	Use irq affinity to force Bluetooth-related interrupts to
87 To reduce its OS jitter, each softirq vector must be handled
91 -------------
96 	is non-idle, for example, by avoiding system calls and by forcing
107 ---------------------------------
113 3.	Once your application has started, prevent CPU-hotplug operations
114 	from being initiated from tasks that might run on the CPU to
115 	be de-jittered.  (It is OK to force this CPU offline and then
119 -------------
123 1.	Force block-device interrupts onto some other CPU.
125 3.	Once your application has started, prevent CPU-hotplug operations
126 	from being initiated from tasks that might run on the CPU to
127 	be de-jittered.  (It is OK to force this CPU offline and then
131 ----------------
135 1.	Force block-device interrupts onto some other CPU.
136 2.	Initiate any block I/O and block-I/O polling on other CPUs.
137 3.	Once your application has started, prevent CPU-hotplug operations
138 	from being initiated from tasks that might run on the CPU to
139 	be de-jittered.  (It is OK to force this CPU offline and then
143 ---------------
154 -------------
158 1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
160 	on that CPU.  If a thread that expects to run on the de-jittered
163 2.	CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered
164 	is marked as an adaptive-ticks CPU using the "nohz_full="
165 	boot parameter.  This reduces the number of scheduler-clock
166 	interrupts that the de-jittered CPU receives, minimizing its
170 	is non-idle, for example, by avoiding system calls and by
172 	This further reduces the number of scheduler-clock interrupts
173 	received by the de-jittered CPU.
176 ---------------
181 	is non-idle.  For example, avoid system calls and force both
192 -----------
196 1.	Offload callbacks and keep the CPU in either dyntick-idle or
197 	adaptive-ticks state by doing all of the following:
200 		de-jittered is marked as an adaptive-ticks CPU using the
202 		housekeeping CPUs, which can tolerate OS jitter.
204 		when it is non-idle, for example, by avoiding system
208 2.	Enable RCU to do its processing remotely via dyntick-idle by
218 		when it is non-idle, for example, by avoiding system
228 To reduce its OS jitter, do any of the following:
230 1.	Run your workload at a real-time priority, which will allow
246 	a.	Avoid using oprofile, thus avoiding OS jitter from
248 	b.	Limit your CPU frequency so that a CPU-frequency
253 		avoid the CPU-frequency governor periodically running
258 	c.	As of v3.18, Christoph Lameter's on-demand vmstat workers
259 		commit prevents OS jitter due to vmstat_update() on
261 		to entirely get rid of the OS jitter, but you can
265 		will make your virtual-memory statistics update more
267 		a real-time priority, thus preempting vmstat_update(),
268 		but if your workload is CPU-bound, this is a bad idea.
270 		(based on an earlier one from Gilad Ben-Yossef) that
272 …workloads at https://lore.kernel.org/r/00000140e9dfd6bd-40db3d4f-c1be-434f-8132-7820f81bb586-00000…
273 	d.	If running on high-end powerpc servers, build with
282 		CBE_CPUFREQ_SPU_GOVERNOR=n to avoid OS jitter from
287 		CONFIG_PMAC_RACKMETER=n to disable the CPU-meter,
288 		avoiding OS jitter from rackmeter_do_timer().
296 To reduce its OS jitter, do at least one of the following:
309 	to OS jitter.  This approach prevents the rcuc/%u kthreads from
323 To reduce its OS jitter, do at least one of the following:
329 	note that this will not eliminate OS jitter, but will instead