1.. _embargoed_hardware_issues: 2 3Embargoed hardware issues 4========================= 5 6Scope 7----- 8 9Hardware issues which result in security problems are a different category 10of security bugs than pure software bugs which only affect the Linux 11kernel. 12 13Hardware issues like Meltdown, Spectre, L1TF etc. must be treated 14differently because they usually affect all Operating Systems ("OS") and 15therefore need coordination across different OS vendors, distributions, 16silicon vendors, hardware integrators, and other parties. For some of the 17issues, software mitigations can depend on microcode or firmware updates, 18which need further coordination. 19 20.. _Contact: 21 22Contact 23------- 24 25The Linux kernel hardware security team is separate from the regular Linux 26kernel security team. 27 28The team only handles developing fixes for embargoed hardware security 29issues. Reports of pure software security bugs in the Linux kernel are not 30handled by this team and the reporter will be guided to contact the regular 31Linux kernel security team (:ref:`Documentation/admin-guide/ 32<securitybugs>`) instead. 33 34The team can be contacted by email at <hardware-security@kernel.org>. This 35is a private list of security officers who will help you coordinate a fix 36according to our documented process. 37 38The list is encrypted and email to the list can be sent by either PGP or 39S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME 40certificate. The list's PGP key and S/MIME certificate are available from 41the following URLs: 42 43 - PGP: https://www.kernel.org/static/files/hardware-security.asc 44 - S/MIME: https://www.kernel.org/static/files/hardware-security.crt 45 46While hardware security issues are often handled by the affected silicon 47vendor, we welcome contact from researchers or individuals who have 48identified a potential hardware flaw. 49 50Hardware security officers 51^^^^^^^^^^^^^^^^^^^^^^^^^^ 52 53The current team of hardware security officers: 54 55 - Linus Torvalds (Linux Foundation Fellow) 56 - Greg Kroah-Hartman (Linux Foundation Fellow) 57 - Thomas Gleixner (Linux Foundation Fellow) 58 59Operation of mailing-lists 60^^^^^^^^^^^^^^^^^^^^^^^^^^ 61 62The encrypted mailing-lists which are used in our process are hosted on 63Linux Foundation's IT infrastructure. By providing this service, members 64of Linux Foundation's IT operations personnel technically have the 65ability to access the embargoed information, but are obliged to 66confidentiality by their employment contract. Linux Foundation IT 67personnel are also responsible for operating and managing the rest of 68kernel.org's infrastructure. 69 70The Linux Foundation's current director of IT Project infrastructure is 71Konstantin Ryabitsev. 72 73 74Non-disclosure agreements 75------------------------- 76 77The Linux kernel hardware security team is not a formal body and therefore 78unable to enter into any non-disclosure agreements. The kernel community 79is aware of the sensitive nature of such issues and offers a Memorandum of 80Understanding instead. 81 82 83Memorandum of Understanding 84--------------------------- 85 86The Linux kernel community has a deep understanding of the requirement to 87keep hardware security issues under embargo for coordination between 88different OS vendors, distributors, silicon vendors, and other parties. 89 90The Linux kernel community has successfully handled hardware security 91issues in the past and has the necessary mechanisms in place to allow 92community compliant development under embargo restrictions. 93 94The Linux kernel community has a dedicated hardware security team for 95initial contact, which oversees the process of handling such issues under 96embargo rules. 97 98The hardware security team identifies the developers (domain experts) who 99will form the initial response team for a particular issue. The initial 100response team can bring in further developers (domain experts) to address 101the issue in the best technical way. 102 103All involved developers pledge to adhere to the embargo rules and to keep 104the received information confidential. Violation of the pledge will lead to 105immediate exclusion from the current issue and removal from all related 106mailing lists. In addition, the hardware security team will also exclude 107the offender from future issues. The impact of this consequence is a highly 108effective deterrent in our community. In case a violation happens the 109hardware security team will inform the involved parties immediately. If you 110or anyone else becomes aware of a potential violation, please report it 111immediately to the Hardware security officers. 112 113 114Process 115^^^^^^^ 116 117Due to the globally distributed nature of Linux kernel development, 118face-to-face meetings are almost impossible to address hardware security 119issues. Phone conferences are hard to coordinate due to time zones and 120other factors and should be only used when absolutely necessary. Encrypted 121email has been proven to be the most effective and secure communication 122method for these types of issues. 123 124Start of Disclosure 125""""""""""""""""""" 126 127Disclosure starts by emailing the Linux kernel hardware security team per 128the Contact section above. This initial contact should contain a 129description of the problem and a list of any known affected silicon. If 130your organization builds or distributes the affected hardware, we encourage 131you to also consider what other hardware could be affected. The disclosing 132party is responsible for contacting the affected silicon vendors in a 133timely manner. 134 135The hardware security team will provide an incident-specific encrypted 136mailing list which will be used for initial discussion with the reporter, 137further disclosure, and coordination of fixes. 138 139The hardware security team will provide the disclosing party a list of 140developers (domain experts) who should be informed initially about the 141issue after confirming with the developers that they will adhere to this 142Memorandum of Understanding and the documented process. These developers 143form the initial response team and will be responsible for handling the 144issue after initial contact. The hardware security team is supporting the 145response team, but is not necessarily involved in the mitigation 146development process. 147 148While individual developers might be covered by a non-disclosure agreement 149via their employer, they cannot enter individual non-disclosure agreements 150in their role as Linux kernel developers. They will, however, agree to 151adhere to this documented process and the Memorandum of Understanding. 152 153The disclosing party should provide a list of contacts for all other 154entities who have already been, or should be, informed about the issue. 155This serves several purposes: 156 157 - The list of disclosed entities allows communication across the 158 industry, e.g. other OS vendors, HW vendors, etc. 159 160 - The disclosed entities can be contacted to name experts who should 161 participate in the mitigation development. 162 163 - If an expert who is required to handle an issue is employed by a listed 164 entity or member of an listed entity, then the response teams can 165 request the disclosure of that expert from that entity. This ensures 166 that the expert is also part of the entity's response team. 167 168Disclosure 169"""""""""" 170 171The disclosing party provides detailed information to the initial response 172team via the specific encrypted mailing-list. 173 174From our experience, the technical documentation of these issues is usually 175a sufficient starting point, and further technical clarification is best 176done via email. 177 178Mitigation development 179"""""""""""""""""""""" 180 181The initial response team sets up an encrypted mailing-list or repurposes 182an existing one if appropriate. 183 184Using a mailing list is close to the normal Linux development process and 185has been successfully used to develop mitigations for various hardware 186security issues in the past. 187 188The mailing list operates in the same way as normal Linux development. 189Patches are posted, discussed, and reviewed and if agreed upon, applied to 190a non-public git repository which is only accessible to the participating 191developers via a secure connection. The repository contains the main 192development branch against the mainline kernel and backport branches for 193stable kernel versions as necessary. 194 195The initial response team will identify further experts from the Linux 196kernel developer community as needed. Any involved party can suggest 197further experts to be included, each of which will be subject to the same 198requirements outlined above. 199 200Bringing in experts can happen at any time in the development process and 201needs to be handled in a timely manner. 202 203If an expert is employed by or a member of an entity on the disclosure list 204provided by the disclosing party, then participation will be requested from 205the relevant entity. 206 207If not, then the disclosing party will be informed about the experts' 208participation. The experts are covered by the Memorandum of Understanding 209and the disclosing party is requested to acknowledge their participation. 210In the case where the disclosing party has a compelling reason to object, 211any objection must to be raised within five working days and resolved with 212the incident team immediately. If the disclosing party does not react 213within five working days this is taken as silent acknowledgment. 214 215After the incident team acknowledges or resolves an objection, the expert 216is disclosed and brought into the development process. 217 218List participants may not communicate about the issue outside of the 219private mailing list. List participants may not use any shared resources 220(e.g. employer build farms, CI systems, etc) when working on patches. 221 222Early access 223"""""""""""" 224 225The patches discussed and developed on the list can neither be distributed 226to any individual who is not a member of the response team nor to any other 227organization. 228 229To allow the affected silicon vendors to work with their internal teams and 230industry partners on testing, validation, and logistics, the following 231exception is provided: 232 233 Designated representatives of the affected silicon vendors are 234 allowed to hand over the patches at any time to the silicon 235 vendor’s response team. The representative must notify the kernel 236 response team about the handover. The affected silicon vendor must 237 have and maintain their own documented security process for any 238 patches shared with their response team that is consistent with 239 this policy. 240 241 The silicon vendor’s response team can distribute these patches to 242 their industry partners and to their internal teams under the 243 silicon vendor’s documented security process. Feedback from the 244 industry partners goes back to the silicon vendor and is 245 communicated by the silicon vendor to the kernel response team. 246 247 The handover to the silicon vendor’s response team removes any 248 responsibility or liability from the kernel response team regarding 249 premature disclosure, which happens due to the involvement of the 250 silicon vendor’s internal teams or industry partners. The silicon 251 vendor guarantees this release of liability by agreeing to this 252 process. 253 254Coordinated release 255""""""""""""""""""" 256 257The involved parties will negotiate the date and time when the embargo 258ends. At that point, the prepared mitigations are published into the 259relevant kernel trees. There is no pre-notification process: the 260mitigations are published in public and available to everyone at the same 261time. 262 263While we understand that hardware security issues need coordinated embargo 264time, the embargo time should be constrained to the minimum time that is 265required for all involved parties to develop, test, and prepare their 266mitigations. Extending embargo time artificially to meet conference talk 267dates or other non-technical reasons creates more work and burden for the 268involved developers and response teams as the patches need to be kept up to 269date in order to follow the ongoing upstream kernel development, which 270might create conflicting changes. 271 272CVE assignment 273"""""""""""""" 274 275Neither the hardware security team nor the initial response team assign 276CVEs, nor are CVEs required for the development process. If CVEs are 277provided by the disclosing party they can be used for documentation 278purposes. 279 280Process ambassadors 281------------------- 282 283For assistance with this process we have established ambassadors in various 284organizations, who can answer questions about or provide guidance on the 285reporting process and further handling. Ambassadors are not involved in the 286disclosure of a particular issue, unless requested by a response team or by 287an involved disclosed party. The current ambassadors list: 288 289 ============= ======================================================== 290 AMD Tom Lendacky <thomas.lendacky@amd.com> 291 Ampere Darren Hart <darren@os.amperecomputing.com> 292 ARM Catalin Marinas <catalin.marinas@arm.com> 293 IBM Power Michael Ellerman <ellerman@au.ibm.com> 294 IBM Z Christian Borntraeger <borntraeger@de.ibm.com> 295 Intel Tony Luck <tony.luck@intel.com> 296 Qualcomm Trilok Soni <quic_tsoni@quicinc.com> 297 RISC-V Palmer Dabbelt <palmer@dabbelt.com> 298 Samsung Javier González <javier.gonz@samsung.com> 299 300 Microsoft James Morris <jamorris@linux.microsoft.com> 301 Xen Andrew Cooper <andrew.cooper3@citrix.com> 302 303 Canonical John Johansen <john.johansen@canonical.com> 304 Debian Ben Hutchings <ben@decadent.org.uk> 305 Oracle Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> 306 Red Hat Josh Poimboeuf <jpoimboe@redhat.com> 307 SUSE Jiri Kosina <jkosina@suse.cz> 308 309 Google Kees Cook <keescook@chromium.org> 310 311 LLVM Nick Desaulniers <ndesaulniers@google.com> 312 ============= ======================================================== 313 314If you want your organization to be added to the ambassadors list, please 315contact the hardware security team. The nominated ambassador has to 316understand and support our process fully and is ideally well-connected in 317the Linux kernel community. 318 319Encrypted mailing-lists 320----------------------- 321 322We use encrypted mailing lists for communication. The operating principle 323of these lists is that email sent to the list is encrypted either with the 324list's PGP key or with the list's S/MIME certificate. The mailing list 325software decrypts the email and re-encrypts it individually for each 326subscriber with the subscriber's PGP key or S/MIME certificate. Details 327about the mailing list software and the setup that is used to ensure the 328security of the lists and protection of the data can be found here: 329https://korg.wiki.kernel.org/userdoc/remail. 330 331List keys 332^^^^^^^^^ 333 334For initial contact see the :ref:`Contact` section above. For incident 335specific mailing lists, the key and S/MIME certificate are conveyed to the 336subscribers by email sent from the specific list. 337 338Subscription to incident-specific lists 339^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 340 341Subscription to incident-specific lists is handled by the response teams. 342Disclosed parties who want to participate in the communication send a list 343of potential experts to the response team so the response team can validate 344subscription requests. 345 346Each subscriber needs to send a subscription request to the response team 347by email. The email must be signed with the subscriber's PGP key or S/MIME 348certificate. If a PGP key is used, it must be available from a public key 349server and is ideally connected to the Linux kernel's PGP web of trust. See 350also: https://www.kernel.org/signature.html. 351 352The response team verifies that the subscriber request is valid and adds 353the subscriber to the list. After subscription the subscriber will receive 354email from the mailing-list which is signed either with the list's PGP key 355or the list's S/MIME certificate. The subscriber's email client can extract 356the PGP key or the S/MIME certificate from the signature so the subscriber 357can send encrypted email to the list. 358 359