Lines Matching +full:in +full:- +full:kernel
1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
8 This document describes how to check if some Linux kernel problem occurs in code
9 currently supported by developers -- to then explain how to locate the change
14 commodity hardware who want to report a kernel bug to the upstream Linux
22 read and navigate this document -- especially when you want to look something
23 up in the reference section, then jump back to where you left off.
26 https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
32 over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes
33 the same commands as this section while describing them in brief fashion. The
35 in a reference section mention many alternatives, pitfalls, and additional
36 aspects, all of which might be essential in your present case.]*
38 **In case you want to check if a bug is present in code currently supported by
40 consider the newest Linux kernel you regularly use to be the 'working' kernel.
41 In the following example that's assumed to be 6.0, which is why its sources
44 **In case you face a regression**, follow the steps at least till the end of
45 *segment 2*. Then you can submit a preliminary report -- or continue with
47 full-fledged regression report. In the following example 6.0.13 is assumed to be
48 the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.0
53 # * Remove any software that depends on externally maintained kernel modules
55 # * Ensure Secure Boot permits booting self-compiled Linux kernels.
56 # * If you are not already running the 'working' kernel, reboot into it.
58 # * Ensure to have 15 Gigabyte free space in your home directory.
59 git clone -o mainline --no-checkout \
60 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
62 git remote add -t master stable \
63 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
64 git switch --detach v6.0
67 # * Ensure the former command picked the .config of the 'working' kernel.
71 ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
72 ./scripts/config -e CONFIG_LOCALVERSION_AUTO
74 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
79 cp .config ~/kernel-config-working
81 * **Segment 1**: build a kernel from the latest mainline codebase.
84 later need to be told about the problem; in case of a regression, this rules
90 git switch --discard-changes --detach mainline/master
92 b) Build, install, and boot a kernel::
94 cp ~/kernel-config-working .config
96 make -j $(nproc --all)
97 # * Make sure there is enough disk space to hold another kernel:
98 df -h /boot/ /lib/modules/
101 # job. See the step-by-step guide for further details.
103 command -v installkernel && sudo make install
104 # * Check how much space your self-built kernel actually needs, which
106 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
107 du -sh /lib/modules/$(make -s kernelrelease)/
109 # right kernel from the boot menu:
110 make -s kernelrelease | tee -a ~/kernels-built
112 # * Once booted, ensure you are running the kernel you just built by
114 tail -n 1 ~/kernels-built
115 uname -r
116 cat /proc/sys/kernel/tainted
118 c) Check if the problem occurs with this kernel as well.
120 * **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
128 git switch --discard-changes --detach v6.0
130 b) Build, install, and boot a kernel as described earlier in *segment 1,
131 section b* -- just feel free to skip the 'du' commands, as you have a rough
134 c) Ensure the feature that regressed with the 'broken' kernel actually works
141 git remote set-branches --add stable linux-6.1.y
151 c) Build, install, and boot a kernel as described earlier in *segment 1,
154 In case building or booting the kernel fails for unrelated reasons, run
155 ``git bisect skip``. In all other outcomes, check if the regressed feature
156 works with the newly built kernel. If it does, tell Git by executing
159 All three commands will make Git check out another commit; then re-execute
160 this step (e.g. build, install, boot, and test a kernel to then tell Git
169 git bisect log > ~/bisect-log
170 cp .config ~/bisection-config-culprit
175 git switch --discard-changes --detach mainline/master
176 git revert --no-edit cafec0cacaca0
177 cp ~/kernel-config-working .config
178 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
181 second command worked flawlessly, build, install, and boot one more kernel
182 kernel; just this time skip the first command copying the base .config file
191 (Segment 3 c). You can list them in build order using::
193 ls -ltr /lib/modules/*-local*
195 To then for example erase a kernel that identifies itself as
196 '6.0-rc1-local-gcafec0cacaca0', use this::
198 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
199 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
200 # * Note, on some distributions kernel-install is missing
211 git switch --discard-changes --detach mainline/master
212 git apply /tmp/foobars-proposed-fix-v1.patch
213 cp ~/kernel-config-working .config
214 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
216 Build, install, and boot a kernel as described in *segment 1, section b* --
222 Step-by-step guide on how to verify bugs and bisect regressions
229 Execute all steps till the end of *segment 1* to **verify if your kernel problem
230 is present in code supported by Linux kernel developers**. If it is, you are all
231 set to report the bug -- unless it did not happen with earlier kernel versions,
236 *segment 3* to **perform a bisection** for a full-fledged regression report
251 The steps in each segment illustrate the important aspects of the process, while
254 pitfalls, as well as problems that might occur at the particular step -- and how
257 For further details on how to report Linux kernel issues or regressions check
258 out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
260 the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6),
261 even if you face a problem with a kernel from a 'stable/longterm' series
267 as a regression check out Documentation/admin-guide/reporting-regressions.rst.
270 improve it, :ref:`please let the kernel developers know <submit_improvements>`.
275 ---------------------------------------------------------
280 machine; if you want to compile the kernel on another system, check
292 * Remove all software that depends on externally developed kernel drivers or
294 VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
301 ensure the system will permit your self-compiled kernel to boot. The
303 disable such techniques in the BIOS setup utility; alternatively, remove
305 ``mokutil --disable-validation``.
311 * Determine the kernel versions considered 'good' and 'bad' throughout this
314 * Do you follow this guide to verify if a bug is present in the code the
315 primary developers care for? Then consider the version of the newest kernel
316 you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2).
319 switching to a newer kernel version? In that case it depends on the version
323 (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a
325 mainline release your working kernel is based on to be the 'good'
327 (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed
328 that 6.0 is fine; this hypothesis will be checked in segment 2.
331 a later one (like 6.1-rc1) or a stable/longterm release based on it
333 the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'.
339 *Note, do not confuse 'good' version with 'working' kernel; the latter term
340 throughout this guide will refer to the last kernel that has been working
347 * Boot into the 'working' kernel and briefly use the apparently broken feature.
353 * Ensure to have enough free space for building Linux. 15 Gigabyte in your home
363 * Install all software required to build a Linux kernel. Often you will need:
374 them, as all further commands in this guide are meant to be executed from
386 git clone -o mainline --no-checkout \
387 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
389 git remote add -t master stable \
390 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
398 to ('linux-6.1.y' in this example)::
400 git remote set-branches --add stable linux-6.1.y
405 * Start preparing a kernel build configuration (the '.config' file).
407 Before doing so, ensure you are still running the 'working' kernel an earlier
409 identifier using ``uname -r``.
412 'good'. In the following example command this is assumed to be 6.0; note that
413 the version number in this and all later Git commands needs to be prefixed
416 git switch --discard-changes --detach v6.0
422 The kernel build scripts then will try to locate the build configuration file
423 for the running kernel and then adjust it for the needs of the kernel sources
426 Look out for a line starting with '# using defaults found in'. It should be
427 followed by a path to a file in '/boot/' that contains the release identifier
428 of your currently working kernel. If the line instead continues with something
430 the .config file for your running kernel -- in which case you have to put one
431 there manually, as explained in the reference section.
433 In case you can not find such a line, look for one containing '# configuration
443 * Disable any kernel modules apparently superfluous for your setup. This is
445 process enormously -- at least unless the .config file picked up in the
446 previous step was already tailored to your and your hardware needs, in which
456 There is a catch to this, as the 'apparently' in initial sentence of this step
459 The 'localmodconfig' target easily disables kernel modules for features only
460 used occasionally -- like modules for external peripherals not yet connected
462 few other things. That's because some tasks rely on kernel modules Linux only
466 something to keep in mind: if something is misbehaving with the kernels built
468 eliminate the risk with tricks outlined in the reference section; but when
469 building a kernel just for quick testing purposes this is usually not worth
480 ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
481 ./scripts/config -e CONFIG_LOCALVERSION_AUTO
489 In the context of this document it is often wise to enable them, as there is a
493 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
494 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
499 ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
500 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
506 * Check if you may want or need to adjust some other kernel configuration
510 additional adjustments explained in the reference section.
523 * Reprocess the .config after the latest adjustments and store it in a safe
527 cp .config ~/kernel-config-working
534 ----------------------------------------------------------------
537 supported by developers. In case you face a regression, it also checks that the
546 Then check the `front page of kernel.org <https://kernel.org/>`_: if it
548 latest version ('linux-6.1.y' in the following example)::
551 git switch --discard-changes --detach stable/linux-6.1.y
554 tag. In that case you might want to check if a successor series (say
555 linux-6.2.y) or mainline (see next point) fix the bug.
557 * In all other cases, run::
560 git switch --discard-changes --detach mainline/master
566 * Build the image and the modules of your first kernel using the config file you
569 cp ~/kernel-config-working .config
571 make -j $(nproc --all)
573 If you want your kernel packaged up as deb, rpm, or tar file, see the
581 * Install your newly built kernel.
585 df -h /boot/ /lib/modules/
587 For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how
590 Now install the kernel's modules and its image, which will be stored in
594 command -v installkernel && sudo make install
597 point: copying the kernel's image to /boot/, generating an initramfs, and
607 down: if you will build more kernels as described in segment 2 and 3, you will
608 have to perform those again after executing ``command -v installkernel [...]``.
614 * In case you plan to follow this guide further, check how much storage space
615 the kernel, its modules, and other related files like the initramfs consume::
617 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
618 du -sh /lib/modules/$(make -s kernelrelease)/
627 * Show and store the kernelrelease identifier of the kernel you just built::
629 make -s kernelrelease | tee -a ~/kernels-built
631 Remember the identifier momentarily, as it will help you pick the right kernel
634 * Reboot into your newly built kernel. To ensure your actually started the one
638 tail -n 1 ~/kernels-built
639 uname -r
643 * Check if the kernel marked itself as 'tainted'::
645 cat /proc/sys/kernel/tainted
654 * Verify if your bug occurs with the newly built kernel. If it does not, check
655 out the instructions in the reference section to ensure nothing went sideways
662 * Did you just built a stable or longterm kernel? And were you able to reproduce
670 git switch --discard-changes --detach mainline/master
672 Now use the checked out code to build and install another kernel using the
673 commands the earlier steps already described in more detail::
675 cp ~/kernel-config-working .config
677 make -j $(nproc --all)
678 # * Check if the free space suffices holding another kernel:
679 df -h /boot/ /lib/modules/
681 command -v installkernel && sudo make install
682 make -s kernelrelease | tee -a ~/kernels-built
685 Confirm you booted the kernel you intended to start and check its tainted
688 tail -n 1 ~/kernels-built
689 uname -r
690 cat /proc/sys/kernel/tainted
692 Now verify if this kernel is showing the problem. If it does, then you need
694 stable team. See Documentation/admin-guide/reporting-issues.rst for details.
698 Do you follow this guide to verify if a problem is present in the code
699 currently supported by Linux kernel developers? Then you are done at this
700 point. If you later want to remove the kernel you just built, check out
703 In case you face a regression, move on and execute at least the next segment
709 ---------------------------------------------------
711 In case of a regression, you now want to ensure the trimmed configuration file
717 * Build your own variant of the 'working' kernel and check if the feature that
724 git switch --discard-changes --detach v6.0
726 Now use the checked out code to configure, build, and install another kernel
727 using the commands the previous subsection explained in more detail::
729 cp ~/kernel-config-working .config
731 make -j $(nproc --all)
732 # * Check if the free space suffices holding another kernel:
733 df -h /boot/ /lib/modules/
735 command -v installkernel && sudo make install
736 make -s kernelrelease | tee -a ~/kernels-built
740 kernel you started is the one you just built::
742 tail -n 1 ~/kernels-built
743 uname -r
745 Now check if this kernel works as expected; if not, consult the reference
753 --------------------------------------------------------
756 to begin the bisection. This will make you build quite a few kernels -- usually
757 about 15 in case you encountered a regression when updating to a newer series
761 each kernel on commodity x86 machines.
766 'good' (6.0 in the following example command) and 'bad' (6.1.5)::
777 * Now use the code Git checked out to build, install, and boot a kernel using
780 cp ~/kernel-config-working .config
782 make -j $(nproc --all)
783 # * Check if the free space suffices holding another kernel:
784 df -h /boot/ /lib/modules/
786 command -v installkernel && sudo make install
787 make -s kernelrelease | tee -a ~/kernels-built
793 In case you skipped the 'test latest codebase' step in the guide, check its
794 description as for why the 'df [...]' and 'make -s kernelrelease [...]'
798 identifiers that might look odd or wrong to you -- which they are not, as it's
799 totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
806 * Now check if the feature that regressed works in the kernel you just built.
808 You again might want to start by making sure the kernel you booted is the one
812 tail -n 1 ~/kernels-built
813 uname -r
815 Now verify if the feature that regressed works at this kernel bisection point.
831 tested. Now build and install another kernel using the instructions from the
832 previous step; afterwards follow the instructions in this step again.
834 Repeat this again and again until you finish the bisection -- that's the case
840 alternatively, run ``git bisect log > ~/bisection-log``.
846 * Store Git's bisection log and the current .config file in a safe place before
850 git bisect log > ~/bisection-log
851 cp .config ~/bisection-config-culprit
864 succeeds, it is worth building another kernel, as it validates the result of
865 a bisection, which can easily deroute; it furthermore will let kernel
871 6.0.13 and 6.0.15) that does not happen in mainline? Then check out the
875 git switch --discard-changes --detach linux-6.0.y
877 * In all other cases check out latest mainline::
880 git switch --discard-changes --detach mainline/master
883 happens in mainline, there is one more thing to do: look up the mainline
884 commit-id. To do so, use a command like ``git show abcdcafecabcd`` to
887 'Upstream commit cafec0cacaca0'; use that commit-id in the next command
892 git revert --no-edit cafec0cacaca0
896 overwriting another kernel::
898 cp ~/kernel-config-working .config
899 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
901 Build a kernel using the familiar command sequence, just without copying the
905 make -j $(nproc --all)
906 # * Check if the free space suffices holding another kernel:
907 df -h /boot/ /lib/modules/
909 command -v installkernel && sudo make install
910 make -s kernelrelease | tee -a ~/kernels-built
914 with that kernel: if everything went well, it should not show the regression.
921 -----------------------------------------------------------
930 identifier. This guide stores them in '~/kernels-built', but the following
933 ls -ltr /lib/modules/*-local*
935 You in most situations want to remove the oldest kernels built during the
938 'good') might become handy to verify something later -- thus better keep them
941 To remove the modules of a kernel with the kernelrelease identifier
942 '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its
945 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
949 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
951 On quite a few distributions this will delete all other kernel files installed
952 while also removing the kernel's entry from the boot menu. But on some
953 distributions kernel-install does not exist or leaves boot-loader entries or
954 kernel image and related files behind; in that case remove them as described
955 in the reference section.
968 for a while, but safely remove all others with a '-local' in the release
971 * Did the bisection end on a merge-commit or seems questionable for other
975 * In other cases it likely is a good idea to keep the following kernels around
985 --------------------------------------------------
988 test reverts, debug patches, proposed fixes, or other versions. In that case
993 * In case you want to test mainline, fetch its latest changes before checking
997 git switch --discard-changes --detach mainline/master
999 * In case you want to test a stable or longterm kernel, first add the branch
1000 holding the series you are interested in (6.2 in the example), unless you
1003 git remote set-branches --add stable linux-6.2.y
1009 git switch --discard-changes --detach stable/linux-6.2.y
1011 * Copy your kernel build configuration over::
1013 cp ~/kernel-config-working .config
1017 * In case you just want to test the latest codebase, head to the next step,
1020 * In case you want to test if a revert fixes an issue, revert one or multiple
1023 git revert --no-edit cafec0cacaca0
1025 Now give that kernel a special tag to facilitates its identification and
1026 prevent accidentally overwriting another kernel::
1028 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
1030 * In case you want to test a patch, store the patch in a file like
1031 '/tmp/foobars-proposed-fix-v1.patch' and apply it like this::
1033 git apply /tmp/foobars-proposed-fix-v1.patch
1035 In case of multiple patches, repeat this step with the others.
1037 Now give that kernel a special tag to facilitates its identification and
1038 prevent accidentally overwriting another kernel::
1040 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
1042 * Build a kernel using the familiar commands, just without copying the kernel
1046 make -j $(nproc --all)
1047 # * Check if the free space suffices holding another kernel:
1048 df -h /boot/ /lib/modules/
1050 command -v installkernel && sudo make install
1051 make -s kernelrelease | tee -a ~/kernels-built
1054 * Now verify you booted the newly built kernel and check it.
1061 ----------
1063 You have reached the end of the step-by-step guide.
1071 CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is
1072 vital to improve this text further, which is in everybody's interest, as it
1073 will enable more people to master the task described here -- and hopefully also
1077 Reference section for the step-by-step guide
1080 This section holds additional information for almost all the items in the above
1081 step-by-step guide.
1084 ------------------------------------------
1086 *The steps in this section lay the groundwork for all further tests.*
1089 The steps in all later sections of this guide depend on those described here.
1091 [:ref:`back to step-by-step guide <introprep_bissbs>`].
1102 -- especially if you fiddle with crucial parts like the kernel of an operating
1103 system. That's what you are about to do in this process. Hence, better prepare
1106 [:ref:`back to step-by-step guide <backup_bissbs>`]
1110 Remove anything related to externally maintained kernel modules
1113 *Remove all software that depends on externally developed kernel drivers or
1116 Externally developed kernel modules can easily cause trouble during a bisection.
1119 kernel developers will not care about reports about regressions occurring with
1121 considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst
1122 explains in more detail.
1124 [:ref:`back to step-by-step guide <vanilla_bissbs>`]
1132 ensure the system will permit your self-compiled kernel to boot later.*
1136 they reject booting self-compiled kernels by default.
1138 You ideally deal with this by making your platform trust your self-built kernels
1141 its purpose; 'Documentation/admin-guide/module-signing.rst' and various web
1142 sides already explain everything needed in more detail.
1145 Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
1151 initiate this process by running ``mokutil --disable-validation``; this will
1152 tell you to create a one-time password, which is safe to write down. Now
1153 restart; right after your BIOS performed all self-tests the bootloader Shim will
1157 randomly chosen characters from the one-time password specified earlier. Once
1161 [:ref:`back to step-by-step guide <secureboot_bissbs>`]
1165 Boot the last kernel that was working
1168 *Boot into the last working kernel and briefly recheck if the feature that
1174 [:ref:`back to step-by-step guide <bootworking_bissbs>`]
1192 [:ref:`back to step-by-step guide <diskspace_bissbs>`]
1199 *Determine the kernel versions considered 'good' and 'bad' throughout this
1204 series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case
1209 that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a
1213 6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
1214 will be built and tested in the segment '2' of this guide; Git would force you
1217 [:ref:`back to step-by-step guide <rangecheck_bissbs>`]
1224 *Install all software required to build a Linux kernel.*
1227 The kernel is pretty stand-alone, but besides tools like the compiler you will
1229 depends on your Linux distribution and the configuration of the kernel you are
1237 sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
1238 pahole perl zlib ncurses qt6-base
1242 sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
1243 libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
1244 libncurses-dev qt6-base-dev g++
1255 kernel-install-tools libelf-devel make modutils openssl openssl-devel \
1256 perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel
1260 which you will only need in case you later might want to adjust the kernel build
1265 for tasks not covered in this guide -- for example when building utilities from
1266 the kernel's tools/ directory.
1268 [:ref:`back to step-by-step guide <buildrequires_bissbs>`]
1278 The step-by-step guide outlines how to download the Linux sources using a full
1280 that -- but there are two alternatives ways to retrieve the sources that might
1298 wget -c \
1299 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
1300 git clone --no-checkout clone.bundle ~/linux/
1304 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
1306 git remote add -t master stable \
1307 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1309 In case the 'wget' command fails, just re-execute it, it will pick up where
1312 [:ref:`back to step-by-step guide <sources_bissbs>`]
1322 git clone -o mainline --no-checkout --depth 1 -b master \
1323 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
1325 git remote add -t master stable \
1326 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1329 release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would
1330 be the first predecessor and 5.18 the second -- hence deepen the history up to
1333 git fetch --shallow-exclude=v5.18 mainline
1336 branches as explained in the step-by-step guide.
1344 releases (like 6.0-rc2 or 5.19-rc3).
1346 * This document in most places uses ``git fetch`` with ``--shallow-exclude=``
1348 tag). You alternatively can use the parameter ``--shallow-since=`` to specify
1349 an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
1352 the release of the mainline release your 'good' kernel is based on.
1355 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
1356 In that case run ``git repack -d`` and try again.
1358 [:ref:`back to step-by-step guide <sources_bissbs>`]
1363 Start defining the build configuration for your kernel
1366 *Start preparing a kernel build configuration (the '.config' file).*
1369 *Note, this is the first of multiple steps in this guide that create or modify
1370 build artifacts. The commands used in this guide store them right in the source
1371 tree to keep things simple. In case you prefer storing the build artifacts
1372 separately, create a directory like '~/linux-builddir/' and add the parameter
1373 ``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
1374 have to point other commands there as well -- among them the ``./scripts/config
1375 [...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
1382 that's what you intend (see next step), but in all other cases you want to
1383 delete it. This for example is important in case you followed this guide
1388 kernel and will use defaults, as briefly outlined in the guide. In that case
1390 it in the right place (e.g. '~/linux/.config') if it does. On distributions
1396 needs of the kernel about to be built.
1401 will be asked how to proceed; in case you are unsure what to answer, simply hit
1408 kernel (say 6.1) on an older mainline release -- especially if it is much older
1409 (say 5.15). That's one of the reasons why the previous step in the guide told
1410 you to boot the kernel where everything works. If you manually add a .config
1411 file you thus want to ensure it's from the working kernel and not from a one
1414 In case you want to build kernels for another machine, locate its kernel build
1415 configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
1419 [:ref:`back to step-by-step guide <oldconfig_bissbs>`]
1423 Trim the build configuration for your kernel
1426 *Disable any kernel modules apparently superfluous for your setup.*
1429 As explained briefly in the step-by-step guide already: with localmodconfig it
1430 can easily happen that your self-built kernels will lack modules for tasks you
1432 when a task requires kernel modules which are only autoloaded when you execute
1434 kernel the modules will not have been loaded -- and from localmodonfig's point
1439 additional kernel modules: start a VM, establish VPN connections, loop-mount a
1443 is hard to think of everything that might be needed -- even kernel developers
1446 Do not let that risk bother you, especially when compiling a kernel only for
1449 run the commands again to compile and install a kernel that has everything you
1452 But if you plan to build and use self-built kernels regularly, you might want to
1454 a few weeks. You can automate this with `modprobed-db
1455 <https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
1456 point localmodconfig to the list of modules modprobed-db noticed being used::
1460 That parameter also allows you to build trimmed kernels for another machine in
1462 run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
1464 step-by-step guide mentions::
1466 yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig
1468 [:ref:`back to step-by-step guide <localmodconfig_bissbs>`]
1480 '-local' in the name; it also helps picking the right entry in the boot menu and
1484 [:ref:`back to step-by-step guide <tagging_bissbs>`]
1493 Having debug symbols available can be important when your kernel throws a
1495 able to find the exact place where the problem occurred in the code. But
1497 quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
1499 symbols, but less than 1 when they were disabled. The resulting kernel image and
1503 In case you want a small kernel and are unlikely to decode a stack trace later,
1506 kernel.
1510 'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
1511 explains this process in more detail.
1513 [:ref:`back to step-by-step guide <debugsymbols_bissbs>`]
1520 *Check if you may want or need to adjust some other kernel configuration
1524 kernel configuration options.
1541 ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
1544 option point to it, as `the Debian handbook explains in more detail
1545 <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
1546 -- or generate your own, as explained in
1547 Documentation/admin-guide/module-signing.rst.
1549 [:ref:`back to step-by-step guide <configmods_bissbs>`]
1560 to enable or disable certain features using a text-based user interface; to use
1566 [:ref:`back to step-by-step guide <configmods_bissbs>`]
1573 *Reprocess the .config after the latest changes and store it in a safe place.*
1578 kernel. That's because going back and forth between different versions can alter
1579 .config files in odd ways; those occasionally cause side effects that could
1580 confuse testing or in some cases render the result of your bisection
1583 [:ref:`back to step-by-step guide <saveconfig_bissbs>`]
1588 -----------------------------------------------------
1594 point, especially if you did that already with a kernel prepared by your
1600 most likely is some problem in my setup' and 'this change needs to be skipped
1601 during the bisection, as the kernel sources at that stage contain an unrelated
1604 * These steps will rule out if your problem is caused by some change in the
1605 build configuration between the 'working' and the 'broken' kernel. This for
1607 feature in the newer kernel which was disabled or not yet supported by the
1608 older kernel. That security feature might get into the way of something you
1609 do -- in which case your problem from the perspective of the Linux kernel
1611 Documentation/admin-guide/reporting-regressions.rst explains in more detail.
1614 * If the cause for your regression was already fixed in the latest mainline
1617 often caused by problems in mainline changes that were backported -- in which
1618 case the problem will have to be fixed in mainline first. Maybe it already was
1619 fixed there and the fix is already in the process of being backported.
1622 know if the issue is specific to that series or also happens in the mainline
1623 kernel, as the report needs to be sent to different people:
1628 * Regressions also happening in mainline are something the regular Linux
1630 and does not need to be involved in the report, they just should be told
1633 Your report might be ignored if you send it to the wrong party -- and even
1637 [:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1647 In case you later want to recheck if an ever newer codebase might fix the
1648 problem, remember to run that ``git fetch --shallow-exclude [...]`` command
1651 [:ref:`back to step-by-step guide <checkoutmaster_bissbs>`]
1655 Build your kernel
1658 *Build the image and the modules of your first kernel using the config file
1662 yourself. Another subsection explains how to directly package your kernel up as
1669 setup that often can be fixed quickly; other times though the problem lies in
1678 error. To make it easier to spot, this command also omits the ``-j $(nproc
1679 --all)`` used earlier to utilize every CPU core in the system for the job -- but
1680 this parallelism also results in some clutter when failures occur.
1684 for the most important and non-generic section of that line (say 4 to 8 words);
1685 avoid or remove anything that looks remotely system-specific, like your username
1687 internet search engine with that string, afterwards search Linux kernel mailing
1688 lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.
1695 In the end, most issues you run into have likely been encountered and
1697 system, but lies in the code. If you run into one of those, you might thus find
1700 Package your kernel up argument
1703 The step-by-step guide uses the default make targets (e.g. 'bzImage' and
1704 'modules' on x86) to build the image and the modules of your kernel, which later
1708 * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package
1710 * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package
1712 * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball
1716 ``make -j $(nproc --all)``, as they will pick up everything already built.
1719 step-by-step guide's instructions on installing and removing your kernel;
1725 distribution's kernel packages.
1727 [:ref:`back to step-by-step guide <build_bissbs>`]
1731 Put the kernel in place
1734 *Install the kernel you just built.* [:ref:`... <install_bissbs>`]
1736 What you need to do after executing the command in the step-by-step guide
1740 If installkernel is found, the kernel's build system will delegate the actual
1741 installation of your kernel image to this executable, which then performs some
1744 * On almost all Linux distributions installkernel will store your kernel's
1745 image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will
1746 put a 'System.map-<kernelrelease_id>' alongside it.
1750 '/boot/initramfs-<kernelrelease_id>.img' or
1751 '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file
1756 * On some distributions installkernel will then add an entry for your kernel
1761 Consult the distribution's documentation for details. If in doubt, install the
1762 kernel manually::
1764 sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
1765 sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)
1768 process. Afterwards add your kernel to your bootloader configuration and reboot.
1770 [:ref:`back to step-by-step guide <install_bissbs>`]
1774 Storage requirements per kernel
1777 *Check how much storage space the kernel, its modules, and other related files
1780 The kernels built during a bisection consume quite a bit of space in /boot/ and
1782 fill up volumes during a bisection -- and due to that even kernels which used to
1784 space each installed kernel typically requires.
1786 Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
1787 the guide will match all files needed to boot your kernel -- but neither the
1789 need to look in different places.
1791 [:ref:`back to step-by-step guide <storagespace_bissbs>`]
1795 Check if your newly built kernel considers itself 'tainted'
1798 *Check if the kernel marked itself as 'tainted'.*
1802 follow-up errors that look totally unrelated. That is why developers might
1803 ignore or react scantly to reports from tainted kernels -- unless of course the
1804 kernel set the flag right when the reported bug occurred.
1806 That's why you want check why a kernel is tainted as explained in
1807 Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own
1810 [:ref:`back to step-by-step guide <tainted_bissbs>`]
1814 Check the kernel built from a recent mainline codebase
1817 *Verify if your bug occurs with the newly built kernel.*
1821 the kernel you built from the latest codebase. These are the most frequent:
1825 * What you suspected to be a regression was caused by a change in the build
1826 configuration the provider of your kernel carried out.
1828 * Your problem might be a race condition that does not show up with your kernel;
1832 * In case you encountered the regression with a stable/longterm kernel it might
1833 be a problem that is specific to that series; the next step in this guide will
1836 [:ref:`back to step-by-step guide <recheckbroken_bissbs>`]
1840 Check the kernel built from the latest stable/longterm codebase
1844 reproduce it with the kernel you just built using the latest mainline sources?
1848 If this kernel does not show the regression either, there most likely is no need
1851 [:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]
1856 ------------------------------------------------
1865 That is in your own interest, as trimming the configuration is not foolproof --
1871 Many readers of this guide normally run kernels that are patched, use add-on
1872 modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
1873 it's possible that the thing that regressed might never have worked in vanilla
1874 builds of the 'good' version in the first place.
1878 ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
1881 [:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]
1885 Build your own version of the 'good' kernel
1888 *Build your own variant of the working kernel and check if the feature that
1891 In case the feature that broke with newer kernels does not work with your first
1892 self-built kernel, find and resolve the cause before moving on. There are a
1900 one from the last working kernel and skip trimming it down; manually disabling
1901 some features in the .config might work as well to reduce the build time.
1903 * Maybe it's not a kernel regression and something that is caused by some fluke,
1907 * Maybe it was a feature added to your distributor's kernel which vanilla Linux
1911 to build another kernel from the latest codebase, as your earlier tests with
1915 [:ref:`back to step-by-step guide <recheckworking_bissbs>`]
1918 -------------------------------------------
1923 The steps in this segment perform and validate the bisection.
1925 [:ref:`back to step-by-step guide <introbisect_bissbs>`].
1936 check out a commit round about half-way between the 'good' and the 'bad' changes
1939 [:ref:`back to step-by-step guide <bisectstart_bissbs>`]
1943 Build a kernel from the bisection point
1946 *Build, install, and boot a kernel from the code Git checked out using the
1951 * Occasionally building the kernel will fail or it might not boot due some
1952 problem in the code at the bisection point. In that case run this command::
1960 because the Linux kernel subsystems prepare their changes for a new mainline
1962 base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then
1967 [:ref:`back to step-by-step guide <bisectbuild_bissbs>`]
1974 *Check if the feature that regressed works in the kernel you just built.*
1981 [:ref:`back to step-by-step guide <bisecttest_bissbs>`]
1988 *Store Git's bisection log and the current .config file in a safe place.*
1991 As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
1992 render the end result of a bisection useless. In that case you'd normally have
1994 allow someone to point out where a bisection likely went sideways -- and then
2001 [:ref:`back to step-by-step guide <bisectlog_bissbs>`]
2013 the bisection result up. So give it a try, you are in the flow already, building
2014 one more kernel shouldn't be a big deal at this point.
2016 The step-by-step guide covers everything relevant already except one slightly
2018 a stable/longterm series, but Git failed to revert the commit in mainline? Then
2019 try to revert the culprit in the affected stable/longterm series -- and if that
2020 succeeds, test that kernel version instead.
2022 [:ref:`back to step-by-step guide <revert_bissbs>`]
2025 ---------------------------------------------------
2030 The steps in this section describe clean-up procedures.
2032 [:ref:`back to step-by-step guide <introclosure_bissbs>`].
2043 parts are only stored in two places and clearly identifiable. You thus do not
2044 need to worry to mess up your machine when you install a kernel manually (and
2048 One of the two places is a directory in /lib/modules/, which holds the modules
2049 for each installed kernel. This directory is named after the kernel's release
2051 simply remove its modules directory in /lib/modules/.
2054 during installation of a kernel. All of them usually contain the release name in
2057 some distributions the ``kernel-install remove...`` command mentioned in the
2058 step-by-step guide will delete all of these files for you while also removing
2059 the menu entry for the kernel from your bootloader configuration. On others you
2061 interactively remove the three main files of a kernel with the release name
2062 '6.0-rc1-local-gcafec0cacaca0'::
2064 rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0
2066 Afterwards check for other files in /boot/ that have
2067 '6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
2068 Now remove the boot entry for the kernel from your bootloader's configuration;
2072 for kernels manually: you might accidentally remove files of a 6.0.13 kernel
2075 [:ref:`back to step-by-step guide <makeroom_bissbs>`]
2086 When you are really short of storage space removing the kernels as described in
2087 the step-by-step guide might not free as much space as you would like. In that
2088 case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
2090 (~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
2094 is a decent chance developers will ask you to build another kernel to
2095 perform additional tests -- like testing a debug patch or a proposed fix.
2096 Details on how to perform those can be found in the section :ref:`Optional
2100 ~/kernel-config-working file around for a few weeks.
2102 [:ref:`back to step-by-step guide <finishingtouch_bissbs>`]
2107 ----------------------------------------
2113 All the commands used in this section should be pretty straight forward, so
2114 there is not much to add except one thing: when setting a kernel tag as
2115 instructed, ensure it is not much longer than the one used in the example, as
2118 [:ref:`back to step-by-step guide <introoptional_bissbs>`].
2127 ------------------------------------
2129 To compile kernels on another system, slightly alter the step-by-step guide's
2135 * After executing ':ref:`Boot into the working kernel and briefly use the
2137 modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the
2138 build configuration for the running kernel (see ':ref:`Start defining the
2139 build configuration for your kernel <oldconfig_bisref>`' for hints on where
2140 to find it) and store it as '~/test-machine-config-working'. Transfer both
2146 * When you reach ':ref:`Start preparing a kernel build configuration[...]
2149 test machine's 'working' kernel::
2151 cp ~/test-machine-config-working ~/linux/.config
2153 * During the next step to ':ref:`disable any apparently superfluous kernel
2156 yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig
2159 install, and reboot into a kernel every time they come up. Instead build
2162 cp ~/kernel-config-working .config
2164 make -j $(nproc --all) targz-pkg
2166 This will generate a gzipped tar file whose name is printed in the last
2167 line shown; for example, a kernel with the kernelrelease identifier
2168 '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will
2169 be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.
2174 kernel. Then extract the file you transferred::
2176 sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /
2178 Afterwards :ref:`generate the initramfs and add the kernel to your boot
2182 … sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423
2184 Now reboot and ensure you started the intended kernel.
2187 cross-compilers and add the appropriate parameters to every invocation of make
2188 (e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).
2191 ---------------------------
2193 * The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
2194 `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
2195 in the Git documentation.
2196 * `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
2197 from kernel developer Nathan Chancellor.
2202 end-of-content
2206 he'll fix it. You are free to do the same in a mostly informal way if you
2207 want to contribute changes to the text -- but for copyright reasons please CC
2208 linux-doc@vger.kernel.org and 'sign-off' your contribution as
2209 Documentation/process/submitting-patches.rst explains in the section 'Sign
2210 your work - the Developer's Certificate of Origin'.
2212 This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
2213 of the file. If you want to distribute this text under CC-BY-4.0 only,
2214 please use 'The Linux kernel development community' for author attribution
2216 …https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide…
2219 Note: Only the content of this RST file as found in the Linux kernel sources
2220 is available under CC-BY-4.0, as versions of this text that were processed
2221 (for example by the kernel's build system) might contain content taken from