admin-guide/pm/cpuidle.rst

1 .. SPDX-License-Identifier: GPL-2.0
19 Modern processors are generally able to enter states in which the execution of
20 a program is suspended and instructions belonging to it are not fetched from
24 generally allows power drawn by the processor to be reduced and, in consequence,
25 it is an opportunity to save energy.
27 CPU idle time management is an energy-efficiency feature concerned about using
31 ------------
36 not be separate physical entities and may just be interfaces appearing to
37 software as individual single-core processors.  In other words, a CPU is an
38 entity which appears to be fetching instructions that belong to one sequence
42 First, if the whole processor can only follow one sequence of instructions (one
43 program) at a time, it is a CPU.  In that case, if the hardware is asked to
44 enter an idle state, that applies to the processor as a whole.
46 Second, if the processor is multi-core, each core in it is able to follow at
47 least one program at a time.  The cores need not be entirely independent of each
50 one program, those programs run mostly independently of each other at the same
51 time.  The entire cores are CPUs in that case and if the hardware is asked to
52 enter an idle state, that applies to the core that asked for it in the first
53 place, but it also may apply to a larger unit (say a "package" or a "cluster")
54 that the core belongs to (in fact, it may apply to an entire hierarchy of larger
56 except for one have been put into idle states at the "core level" and the
57 remaining core asks the processor to enter an idle state, that may trigger it
58 to put the whole larger unit into an idle state which also will affect the
61 Finally, each core in a multi-core processor may be able to follow more than one
62 program in the same time frame (that is, each core may be able to fetch
65 the cores present themselves to software as "bundles" each consisting of
66 multiple individual single-core "processors", referred to as *hardware threads*
67 (or hyper-threads specifically on Intel hardware), that each can follow one
69 time management perspective and if the processor is asked to enter an idle state
70 by one of them, the hardware thread (or CPU) that asked for it is stopped, but
72 core also have asked the processor to enter an idle state.  In that situation,
78 ---------
80 Logical CPUs, simply referred to as "CPUs" in what follows, are regarded as
81 *idle* by the Linux kernel when there are no tasks to run on them except for the
85 sequence of instructions to execute, or code, data to be manipulated while
86 running that code, and some context information that needs to be loaded into the
88 distributes work by assigning tasks to run to the CPUs present in the system.
93 events to occur or similar).  When a task becomes runnable, the CPU scheduler
94 assigns it to one of the available CPUs to run and if there are no more runnable
95 tasks assigned to it, the CPU will load the given task's context and run its
96 code (from the instruction following the last one executed so far, possibly by
97 another CPU).  [If there are multiple runnable tasks assigned to one CPU
98 simultaneously, they will be subject to prioritization and time sharing in order
99 to allow them to make some progress over time.]
102 assigned to the given CPU and the CPU is then regarded as idle.  In other words,
104 code may cause the processor to be put into one of its idle states, if they are
105 supported, in order to save energy, but if the processor does not support any
106 idle states, or there is not enough time to spend in an idle state before the
109 useless instructions in a loop until it is assigned a new task to run.
112 .. _idle-loop:
118 calls into a code module referred to as the *governor* that belongs to the CPU
119 idle time management subsystem called ``CPUIdle`` to select an idle state for
120 the CPU to ask the hardware to enter.  Second, it invokes another code module
121 from the ``CPUIdle`` subsystem, called the *driver*, to actually ask the
122 processor hardware to enter the idle state selected by the governor.
124 The role of the governor is to find an idle state most suitable for the
126 asked to enter by logical CPUs are represented in an abstract way independent of
127 the platform or the processor architecture and organized in a one-dimensional
128 (linear) array.  That array has to be prepared and supplied by the ``CPUIdle``
130 time.  This allows ``CPUIdle`` governors to be independent of the underlying
131 hardware and to work with any platforms that the Linux kernel can run on.
133 Each idle state present in that array is characterized by two parameters to be
134 taken into account by the governor, the *target residency* and the (worst-case)
136 spend in the given state, including the time needed to enter it (which may be
137 substantial), in order to save more energy than it would save by entering one of
139 corresponds to the power drawn by the processor in that state.]  The exit
141 hardware to enter an idle state to start executing the first instruction after a
143 the time needed to enter the given state in case the wakeup occurs when the
144 hardware is entering it and it must be entered completely to be exited in an
151 CPU depends on can spend in an idle state, including the time necessary to enter
152 and exit it.  However, the CPU may be woken up by a non-timer event at any time
155 was idle after it has been woken up (that time will be referred to as the *idle
157 time until the closest timer to estimate the idle duration in future.  How the
159 and that is the primary reason for having more than one governor in the
162 There are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_,
165 tick can be `stopped by the idle loop <idle-cpus-and-tick_>`_.  Available
173 platform the kernel is running on, but there are platforms with more than one
175 majority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with
176 hardcoded idle states information and the other able to read that information
179 decision on which one of them to use has to be made early (on Intel platforms
186 .. _idle-cpus-and-tick:
191 The scheduler tick is a timer that triggers periodically in order to implement
193 multiple runnable tasks assigned to one CPU at the same time, the only way to
194 allow them to make reasonable progress in a given time frame is to make them
196 given a slice of the CPU time to run its code, subject to the scheduling class,
198 switched over to running (the code of) another task.  The currently running task
199 may not want to give the CPU away voluntarily, however, and the scheduler tick
200 is there to make the switch happen regardless.  That is not the only role of the
206 Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense
207 for them to ask the hardware to enter idle states with target residencies above
210 exiting idle states due to the tick wakeups on idle CPUs will be wasted.
212 Fortunately, it is not really necessary to allow the tick to trigger on idle
213 CPUs, because (by definition) they have no tasks to run except for the special
214 "idle" one.  In other words, from the CPU scheduler perspective, the only user
217 tick goes away if the given CPU is idle.  Consequently, it is possible to stop
221 Whether or not it makes sense to stop the scheduler tick in the idle loop
223 (non-tick) timer due to trigger within the tick range, stopping the tick clearly
224 would be a waste of time, even though the timer hardware may not need to be
225 reprogrammed in that case.  Second, if the governor is expecting a non-timer
229 going to be relatively shallow.  The governor really cannot select a deep idle
232 waste of time and in this case the timer hardware would need to be reprogrammed,
237 tick range, it is better to allow the tick trigger.  Otherwise, however, the
242 or not to stop the scheduler tick belongs to it.  Still, if the tick has been
243 stopped already (in one of the previous iterations of the loop), it is better
244 to leave it as is and the governor needs to take that into account.
246 The kernel can be configured to disable stopping the scheduler tick in the idle
247 loop altogether.  That can be done through the build-time configuration of it
249 ``nohz=off`` to it in the command line.  In both cases, as the stopping of the
253 The systems that run kernels configured to allow the scheduler tick to be
254 stopped on idle CPUs are referred to as *tickless* systems and they are
255 generally regarded as more energy-efficient than the systems running kernels in
261 .. _menu-gov:
268 Namely, when invoked to select an idle state for a CPU (i.e. an idle state that
269 the CPU will ask the processor hardware to enter), it attempts to predict the
273 that the scheduler tick will be stopped.  That time, referred to as the *sleep
275 wakeup.  It is used to determine the sleep length range, which in turn is needed
276 to get the sleep length correction factor.
279 One of them is used when tasks previously running on the given CPU are waiting
280 for some I/O operations to complete and the other one is used when that is not
282 to different sleep length ranges organized so that each range represented in the
283 array is approximately 10 times wider than the previous one.
287 up and the closer the sleep length is to the observed idle duration, the closer
288 to 1 the correction factor becomes (it must fall between 0 and 1 inclusive).
290 falls into to obtain the first approximation of the predicted idle duration.
292 Next, the governor uses a simple pattern recognition algorithm to refine its
296 milliseconds) or it is small relative to the average (the average is greater
303 the "typical interval" is assumed to equal "infinity" (the maximum unsigned
308 Then, the governor computes an extra latency limit to help "interactive"
311 in that state probably will be very short and the amount of energy to save by
312 entering it will be relatively small, so likely it is better to avoid the
313 overhead related to entering that state and exiting it.  Thus selecting a
314 shallower state is likely to be a better option then.   The first approximation
317 previously ran on the given CPU and now they are waiting for I/O operations to
319 from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_,
323 Now, the governor is ready to walk the list of idle states and choose one of
326 limit.  It selects the state with the target residency closest to the predicted
330 In the final step the governor may still need to refine the idle state selection
331 if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_.  That
336 that time, the governor may need to select a shallower state with a suitable
340 .. _teo-gov:
346 for tickless systems.  It follows the same basic strategy as the ``menu`` `one
347 <menu-gov_>`_: it always tries to find the deepest idle state suitable for the
348 given conditions.  However, it applies a different approach to that problem.
350 .. kernel-doc:: drivers/cpuidle/governors/teo.c
351    :doc: teo-description
353 .. _idle-states-representation:
359 supported by the processor have to be represented as a one-dimensional array of
360 |struct cpuidle_state| objects each allowing an individual (logical) CPU to ask
361 the processor hardware to enter an idle state of certain properties.  If there
362 is a hierarchy of units in the processor, one |struct cpuidle_state| object can
365 of it <idle-loop_>`_, must reflect the properties of the idle state at the
369 For example, take a processor with two cores in a larger unit referred to as
370 a "module" and suppose that asking the hardware to enter a specific idle state
371 (say "X") at the "core" level by one core will trigger the module to try to
374 level gives the hardware a license to go as deep as to idle state "MX" at the
375 "module" level, but there is no guarantee that this is going to happen (the core
378 idle state "X" must reflect the minimum time to spend in idle state "MX" of
379 the module (including the time needed to enter it), because that is the minimum
380 time the CPU needs to be idle to save any energy in case the hardware enters
384 will start to execute the first new instruction (assuming that both cores in the
385 module will always be ready to execute instructions as soon as the module
393 entirely up to the driver, but still the physical properties of the idle state
399 In addition to the target residency and exit latency idle state parameters
401 parameters describing the idle state and a pointer to the function to run in
402 order to ask the hardware to enter that state.  Also, for each
404 :c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage
409 directory in ``sysfs``, where the number ``<N>`` is assigned to the given
411 called :file:`state0`, :file:`state1` and so on, up to the number of idle state
412 objects defined for the given CPU minus one.  Each of these directories
413 corresponds to one idle state object and the larger the number in its name, the
416 object corresponding to it, as follows:
420 	observed idle duration was certainly too short to match its target
455 	Total number of times the hardware has been asked by the given CPU to
459 	Total number of times a request to enter this idle state on the given
463 between them is that the name is expected to be more concise, while the
467 The :file:`disable` attribute is the only writeable one.  If it contains 1, the
470 driver will never ask the hardware to enter it for that CPU as a result.
471 However, disabling an idle state for one CPU does not prevent it from being
472 asked for by the other CPUs, so it must be disabled for all of them in order to
473 never be asked for by any of them.  [Note that, due to the way the ``ladder``
475 selecting any idle states deeper than the disabled one too.]
479 CPUs in the system at the same time.  Writing 1 to it causes the idle state to
480 be disabled for this particular CPU and writing 0 to it allows the governor to
481 take it into consideration for the given CPU and the driver to ask for it,
487 hierarchy of units in the processor, and it generally is hard to obtain idle
494 the kernel and it may not cover the cases in which the hardware refused to enter
495 this idle state and entered a shallower one instead of it (or even it did not
497 asking the hardware to enter an idle state and the subsequent wakeup of the CPU
502 particular case.  For these reasons, the only reliable way to find out how
504 it is to use idle state residency counters in the hardware, if available.
506 Generally, an interrupt received when trying to enter an idle state causes the
507 idle state entry request to be rejected, in which case the ``CPUIdle`` driver
508 may return an error code to indicate that this was the case. The :file:`usage`
512 .. _cpu-pm-qos:
518 allows kernel code and user space processes to set constraints on various
519 energy-efficiency features of the kernel to prevent performance from dropping
528 signed 32-bit integer) to it.  In turn, the resume latency constraint for a CPU
530 32-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under
545 PM QoS request to be created and added to a global priority list of CPU latency
548 number written to it will be associated with the PM QoS request represented by
550 used to determine the new effective value of the entire list of requests and
563 with that file descriptor to be removed from the global priority list of CPU
565 mechanism will be used again, to determine the new effective value for the whole
568 In turn, for each CPU there is one resume latency PM QoS request associated with
570 :file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes
571 this single PM QoS request to be updated regardless of which user space
573 user space, so access to the file associated with it needs to be arbitrated
574 to avoid confusion.  [Arguably, the only legitimate use of this mechanism in
575 practice is to pin a process to the CPU in question and let it use the
576 ``sysfs`` interface to control the resume latency constraint for it.]  It is
577 still only a request, however.  It is an entry in a priority list used to
578 determine the effective value to be set as the resume latency constraint for the
582 CPU idle time governors are expected to regard the minimum of the global
585 they are allowed to select for that CPU.  They should never select any idle
592 In addition to the ``sysfs`` interface allowing individual idle states to be
593 `disabled for individual CPUs <idle-states-representation_>`_, there are kernel
596 The ``cpuidle.off=1`` kernel command line option can be used to disable the
599 from being invoked.  If it is added to the kernel command line, the idle loop
600 will ask the hardware to enter idle states on idle CPUs via the CPU architecture
601 support code that is expected to provide a default mechanism for this purpose.
604 however, so it is rather crude and not very energy-efficient.  For this reason,
608 governor to use to be specified.  It has to be appended with a string matching
610 governor will be used instead of the default one.  It is possible to force
611 the ``menu`` governor to be used on the systems that use the ``ladder`` governor
616 to ``intel_idle`` affect Intel processors only.
619 options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
622 ``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the
623 architecture support code to deal with idle CPUs.  How it does that depends on
624 which of the two parameters is added to the kernel command line.  In the
627 and causes the hardware to attempt to enter the shallowest available idle state)
633 P-states (see |cpufreq|) that require any number of CPUs in a package to be
634 idle, so it very well may hurt single-thread computations performance as well as
635 energy-efficiency.  Thus using it for performance reasons may not be a good idea
639 the CPU to enter idle states. When this option is used, the ``acpi_idle``
646 In addition to the architecture-level kernel command line options affecting CPU
648 drivers that can be passed to them via the kernel command line.  Specifically,
652 `Representation of Idle States <idle-states-representation_>`_), causes the
653 ``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the
655 for any of those idle states or expose them to the governor.  [The behavior of
656 the two drivers is different for ``<n>`` equal to ``0``.  Adding
657 ``intel_idle.max_cstate=0`` to the kernel command line disables the
658 ``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas
659 ``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``.
661 can be loaded separately and ``max_cstate=<n>`` can be passed to it as a module