1.. SPDX-License-Identifier: GPL-2.0 2 3======================= 4Energy Model of devices 5======================= 6 71. Overview 8----------- 9 10The Energy Model (EM) framework serves as an interface between drivers knowing 11the power consumed by devices at various performance levels, and the kernel 12subsystems willing to use that information to make energy-aware decisions. 13 14The source of the information about the power consumed by devices can vary greatly 15from one platform to another. These power costs can be estimated using 16devicetree data in some cases. In others, the firmware will know better. 17Alternatively, userspace might be best positioned. And so on. In order to avoid 18each and every client subsystem to re-implement support for each and every 19possible source of information on its own, the EM framework intervenes as an 20abstraction layer which standardizes the format of power cost tables in the 21kernel, hence enabling to avoid redundant work. 22 23The power values might be expressed in micro-Watts or in an 'abstract scale'. 24Multiple subsystems might use the EM and it is up to the system integrator to 25check that the requirements for the power value scale types are met. An example 26can be found in the Energy-Aware Scheduler documentation 27Documentation/scheduler/sched-energy.rst. For some subsystems like thermal or 28powercap power values expressed in an 'abstract scale' might cause issues. 29These subsystems are more interested in estimation of power used in the past, 30thus the real micro-Watts might be needed. An example of these requirements can 31be found in the Intelligent Power Allocation in 32Documentation/driver-api/thermal/power_allocator.rst. 33Kernel subsystems might implement automatic detection to check whether EM 34registered devices have inconsistent scale (based on EM internal flag). 35Important thing to keep in mind is that when the power values are expressed in 36an 'abstract scale' deriving real energy in micro-Joules would not be possible. 37 38The figure below depicts an example of drivers (Arm-specific here, but the 39approach is applicable to any architecture) providing power costs to the EM 40framework, and interested clients reading the data from it:: 41 42 +---------------+ +-----------------+ +---------------+ 43 | Thermal (IPA) | | Scheduler (EAS) | | Other | 44 +---------------+ +-----------------+ +---------------+ 45 | | em_cpu_energy() | 46 | | em_cpu_get() | 47 +---------+ | +---------+ 48 | | | 49 v v v 50 +---------------------+ 51 | Energy Model | 52 | Framework | 53 +---------------------+ 54 ^ ^ ^ 55 | | | em_dev_register_perf_domain() 56 +----------+ | +---------+ 57 | | | 58 +---------------+ +---------------+ +--------------+ 59 | cpufreq-dt | | arm_scmi | | Other | 60 +---------------+ +---------------+ +--------------+ 61 ^ ^ ^ 62 | | | 63 +--------------+ +---------------+ +--------------+ 64 | Device Tree | | Firmware | | ? | 65 +--------------+ +---------------+ +--------------+ 66 67In case of CPU devices the EM framework manages power cost tables per 68'performance domain' in the system. A performance domain is a group of CPUs 69whose performance is scaled together. Performance domains generally have a 701-to-1 mapping with CPUFreq policies. All CPUs in a performance domain are 71required to have the same micro-architecture. CPUs in different performance 72domains can have different micro-architectures. 73 74To better reflect power variation due to static power (leakage) the EM 75supports runtime modifications of the power values. The mechanism relies on 76RCU to free the modifiable EM perf_state table memory. Its user, the task 77scheduler, also uses RCU to access this memory. The EM framework provides 78API for allocating/freeing the new memory for the modifiable EM table. 79The old memory is freed automatically using RCU callback mechanism when there 80are no owners anymore for the given EM runtime table instance. This is tracked 81using kref mechanism. The device driver which provided the new EM at runtime, 82should call EM API to free it safely when it's no longer needed. The EM 83framework will handle the clean-up when it's possible. 84 85The kernel code which want to modify the EM values is protected from concurrent 86access using a mutex. Therefore, the device driver code must run in sleeping 87context when it tries to modify the EM. 88 89With the runtime modifiable EM we switch from a 'single and during the entire 90runtime static EM' (system property) design to a 'single EM which can be 91changed during runtime according e.g. to the workload' (system and workload 92property) design. 93 94It is possible also to modify the CPU performance values for each EM's 95performance state. Thus, the full power and performance profile (which 96is an exponential curve) can be changed according e.g. to the workload 97or system property. 98 99 1002. Core APIs 101------------ 102 1032.1 Config options 104^^^^^^^^^^^^^^^^^^ 105 106CONFIG_ENERGY_MODEL must be enabled to use the EM framework. 107 108 1092.2 Registration of performance domains 110^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 111 112Registration of 'advanced' EM 113~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 114 115The 'advanced' EM gets its name due to the fact that the driver is allowed 116to provide more precised power model. It's not limited to some implemented math 117formula in the framework (like it is in 'simple' EM case). It can better reflect 118the real power measurements performed for each performance state. Thus, this 119registration method should be preferred in case considering EM static power 120(leakage) is important. 121 122Drivers are expected to register performance domains into the EM framework by 123calling the following API:: 124 125 int em_dev_register_perf_domain(struct device *dev, unsigned int nr_states, 126 struct em_data_callback *cb, cpumask_t *cpus, bool microwatts); 127 128Drivers must provide a callback function returning <frequency, power> tuples 129for each performance state. The callback function provided by the driver is free 130to fetch data from any relevant location (DT, firmware, ...), and by any mean 131deemed necessary. Only for CPU devices, drivers must specify the CPUs of the 132performance domains using cpumask. For other devices than CPUs the last 133argument must be set to NULL. 134The last argument 'microwatts' is important to set with correct value. Kernel 135subsystems which use EM might rely on this flag to check if all EM devices use 136the same scale. If there are different scales, these subsystems might decide 137to return warning/error, stop working or panic. 138See Section 3. for an example of driver implementing this 139callback, or Section 2.4 for further documentation on this API 140 141Registration of EM using DT 142~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 143 144The EM can also be registered using OPP framework and information in DT 145"operating-points-v2". Each OPP entry in DT can be extended with a property 146"opp-microwatt" containing micro-Watts power value. This OPP DT property 147allows a platform to register EM power values which are reflecting total power 148(static + dynamic). These power values might be coming directly from 149experiments and measurements. 150 151Registration of 'artificial' EM 152~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 153 154There is an option to provide a custom callback for drivers missing detailed 155knowledge about power value for each performance state. The callback 156.get_cost() is optional and provides the 'cost' values used by the EAS. 157This is useful for platforms that only provide information on relative 158efficiency between CPU types, where one could use the information to 159create an abstract power model. But even an abstract power model can 160sometimes be hard to fit in, given the input power value size restrictions. 161The .get_cost() allows to provide the 'cost' values which reflect the 162efficiency of the CPUs. This would allow to provide EAS information which 163has different relation than what would be forced by the EM internal 164formulas calculating 'cost' values. To register an EM for such platform, the 165driver must set the flag 'microwatts' to 0, provide .get_power() callback 166and provide .get_cost() callback. The EM framework would handle such platform 167properly during registration. A flag EM_PERF_DOMAIN_ARTIFICIAL is set for such 168platform. Special care should be taken by other frameworks which are using EM 169to test and treat this flag properly. 170 171Registration of 'simple' EM 172~~~~~~~~~~~~~~~~~~~~~~~~~~~ 173 174The 'simple' EM is registered using the framework helper function 175cpufreq_register_em_with_opp(). It implements a power model which is tight to 176math formula:: 177 178 Power = C * V^2 * f 179 180The EM which is registered using this method might not reflect correctly the 181physics of a real device, e.g. when static power (leakage) is important. 182 183 1842.3 Accessing performance domains 185^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 186 187There are two API functions which provide the access to the energy model: 188em_cpu_get() which takes CPU id as an argument and em_pd_get() with device 189pointer as an argument. It depends on the subsystem which interface it is 190going to use, but in case of CPU devices both functions return the same 191performance domain. 192 193Subsystems interested in the energy model of a CPU can retrieve it using the 194em_cpu_get() API. The energy model tables are allocated once upon creation of 195the performance domains, and kept in memory untouched. 196 197The energy consumed by a performance domain can be estimated using the 198em_cpu_energy() API. The estimation is performed assuming that the schedutil 199CPUfreq governor is in use in case of CPU device. Currently this calculation is 200not provided for other type of devices. 201 202More details about the above APIs can be found in ``<linux/energy_model.h>`` 203or in Section 2.5 204 205 2062.4 Runtime modifications 207^^^^^^^^^^^^^^^^^^^^^^^^^ 208 209Drivers willing to update the EM at runtime should use the following dedicated 210function to allocate a new instance of the modified EM. The API is listed 211below:: 212 213 struct em_perf_table __rcu *em_table_alloc(struct em_perf_domain *pd); 214 215This allows to allocate a structure which contains the new EM table with 216also RCU and kref needed by the EM framework. The 'struct em_perf_table' 217contains array 'struct em_perf_state state[]' which is a list of performance 218states in ascending order. That list must be populated by the device driver 219which wants to update the EM. The list of frequencies can be taken from 220existing EM (created during boot). The content in the 'struct em_perf_state' 221must be populated by the driver as well. 222 223This is the API which does the EM update, using RCU pointers swap:: 224 225 int em_dev_update_perf_domain(struct device *dev, 226 struct em_perf_table __rcu *new_table); 227 228Drivers must provide a pointer to the allocated and initialized new EM 229'struct em_perf_table'. That new EM will be safely used inside the EM framework 230and will be visible to other sub-systems in the kernel (thermal, powercap). 231The main design goal for this API is to be fast and avoid extra calculations 232or memory allocations at runtime. When pre-computed EMs are available in the 233device driver, than it should be possible to simply re-use them with low 234performance overhead. 235 236In order to free the EM, provided earlier by the driver (e.g. when the module 237is unloaded), there is a need to call the API:: 238 239 void em_table_free(struct em_perf_table __rcu *table); 240 241It will allow the EM framework to safely remove the memory, when there is 242no other sub-system using it, e.g. EAS. 243 244To use the power values in other sub-systems (like thermal, powercap) there is 245a need to call API which protects the reader and provide consistency of the EM 246table data:: 247 248 struct em_perf_state *em_perf_state_from_pd(struct em_perf_domain *pd); 249 250It returns the 'struct em_perf_state' pointer which is an array of performance 251states in ascending order. 252This function must be called in the RCU read lock section (after the 253rcu_read_lock()). When the EM table is not needed anymore there is a need to 254call rcu_real_unlock(). In this way the EM safely uses the RCU read section 255and protects the users. It also allows the EM framework to manage the memory 256and free it. More details how to use it can be found in Section 3.2 in the 257example driver. 258 259There is dedicated API for device drivers to calculate em_perf_state::cost 260values:: 261 262 int em_dev_compute_costs(struct device *dev, struct em_perf_state *table, 263 int nr_states); 264 265These 'cost' values from EM are used in EAS. The new EM table should be passed 266together with the number of entries and device pointer. When the computation 267of the cost values is done properly the return value from the function is 0. 268The function takes care for right setting of inefficiency for each performance 269state as well. It updates em_perf_state::flags accordingly. 270Then such prepared new EM can be passed to the em_dev_update_perf_domain() 271function, which will allow to use it. 272 273More details about the above APIs can be found in ``<linux/energy_model.h>`` 274or in Section 3.2 with an example code showing simple implementation of the 275updating mechanism in a device driver. 276 277 2782.5 Description details of this API 279^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 280.. kernel-doc:: include/linux/energy_model.h 281 :internal: 282 283.. kernel-doc:: kernel/power/energy_model.c 284 :export: 285 286 2873. Examples 288----------- 289 2903.1 Example driver with EM registration 291^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 292 293The CPUFreq framework supports dedicated callback for registering 294the EM for a given CPU(s) 'policy' object: cpufreq_driver::register_em(). 295That callback has to be implemented properly for a given driver, 296because the framework would call it at the right time during setup. 297This section provides a simple example of a CPUFreq driver registering a 298performance domain in the Energy Model framework using the (fake) 'foo' 299protocol. The driver implements an est_power() function to be provided to the 300EM framework:: 301 302 -> drivers/cpufreq/foo_cpufreq.c 303 304 01 static int est_power(struct device *dev, unsigned long *mW, 305 02 unsigned long *KHz) 306 03 { 307 04 long freq, power; 308 05 309 06 /* Use the 'foo' protocol to ceil the frequency */ 310 07 freq = foo_get_freq_ceil(dev, *KHz); 311 08 if (freq < 0); 312 09 return freq; 313 10 314 11 /* Estimate the power cost for the dev at the relevant freq. */ 315 12 power = foo_estimate_power(dev, freq); 316 13 if (power < 0); 317 14 return power; 318 15 319 16 /* Return the values to the EM framework */ 320 17 *mW = power; 321 18 *KHz = freq; 322 19 323 20 return 0; 324 21 } 325 22 326 23 static void foo_cpufreq_register_em(struct cpufreq_policy *policy) 327 24 { 328 25 struct em_data_callback em_cb = EM_DATA_CB(est_power); 329 26 struct device *cpu_dev; 330 27 int nr_opp; 331 28 332 29 cpu_dev = get_cpu_device(cpumask_first(policy->cpus)); 333 30 334 31 /* Find the number of OPPs for this policy */ 335 32 nr_opp = foo_get_nr_opp(policy); 336 33 337 34 /* And register the new performance domain */ 338 35 em_dev_register_perf_domain(cpu_dev, nr_opp, &em_cb, policy->cpus, 339 36 true); 340 37 } 341 38 342 39 static struct cpufreq_driver foo_cpufreq_driver = { 343 40 .register_em = foo_cpufreq_register_em, 344 41 }; 345 346 3473.2 Example driver with EM modification 348^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 349 350This section provides a simple example of a thermal driver modifying the EM. 351The driver implements a foo_thermal_em_update() function. The driver is woken 352up periodically to check the temperature and modify the EM data:: 353 354 -> drivers/soc/example/example_em_mod.c 355 356 01 static void foo_get_new_em(struct foo_context *ctx) 357 02 { 358 03 struct em_perf_table __rcu *em_table; 359 04 struct em_perf_state *table, *new_table; 360 05 struct device *dev = ctx->dev; 361 06 struct em_perf_domain *pd; 362 07 unsigned long freq; 363 08 int i, ret; 364 09 365 10 pd = em_pd_get(dev); 366 11 if (!pd) 367 12 return; 368 13 369 14 em_table = em_table_alloc(pd); 370 15 if (!em_table) 371 16 return; 372 17 373 18 new_table = em_table->state; 374 19 375 20 rcu_read_lock(); 376 21 table = em_perf_state_from_pd(pd); 377 22 for (i = 0; i < pd->nr_perf_states; i++) { 378 23 freq = table[i].frequency; 379 24 foo_get_power_perf_values(dev, freq, &new_table[i]); 380 25 } 381 26 rcu_read_unlock(); 382 27 383 28 /* Calculate 'cost' values for EAS */ 384 29 ret = em_dev_compute_costs(dev, table, pd->nr_perf_states); 385 30 if (ret) { 386 31 dev_warn(dev, "EM: compute costs failed %d\n", ret); 387 32 em_free_table(em_table); 388 33 return; 389 34 } 390 35 391 36 ret = em_dev_update_perf_domain(dev, em_table); 392 37 if (ret) { 393 38 dev_warn(dev, "EM: update failed %d\n", ret); 394 39 em_free_table(em_table); 395 40 return; 396 41 } 397 42 398 43 /* 399 44 * Since it's one-time-update drop the usage counter. 400 45 * The EM framework will later free the table when needed. 401 46 */ 402 47 em_table_free(em_table); 403 48 } 404 49 405 50 /* 406 51 * Function called periodically to check the temperature and 407 52 * update the EM if needed 408 53 */ 409 54 static void foo_thermal_em_update(struct foo_context *ctx) 410 55 { 411 56 struct device *dev = ctx->dev; 412 57 int cpu; 413 58 414 59 ctx->temperature = foo_get_temp(dev, ctx); 415 60 if (ctx->temperature < FOO_EM_UPDATE_TEMP_THRESHOLD) 416 61 return; 417 62 418 63 foo_get_new_em(ctx); 419 64 } 420