1 .. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
13 For details regarding the user-facing interface refer to the TLS
18  * Software crypto mode (``TLS_SW``) - CPU handles the cryptography.
24  * Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto
28    (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``).
29  * Full TCP NIC offload mode (``TLS_HW_RECORD``) - mode of operation where
33    abilities or QoS and packet scheduling (``ethtool`` flag ``tls-hw-record``).
35 The operation mode is selected automatically based on device configuration,
36 offload opt-in or opt-out on per-connection basis is not currently supported.
38 TX  section in Kernel TLS operation
39 --
47 Instead packets reach a device driver, the driver will mark the packets
49 and send them to the device for encryption and transmission.
52 --
54 On the receive side if the device handled decryption and authentication
60 If device decrypted all the segments of the record the decryption is skipped,
63 .. kernel-figure::  tls-offload-layers.svg
70 Device configuration
73 During driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and
80 (note that it is done twice, once for RX and once for TX direction,
82 network device is offload-capable and attempts the offload. In case offload
89 .. code-block:: c
98 to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6).
107 TX  section in Device configuration
108 --
110 After TX state is installed, the stack guarantees that the first segment
114 TX offload being fully initialized does not imply that all segments passing
121 --
129 At the minimum the device maintains the following state for each connection, in
139 Assuming segments are received in order, the device should be able to perform
141 to be possible device has to keep small amount of segment-to-segment state.
150 in order the device should be able to handle them separately and make
153 TX  section in Normal operation
154 --
159 Both the device and the driver maintain expected TCP sequence numbers
161 once the packet reaches the device.
163 a connection identifier (note that a 5-tuple lookup is insufficient to identify
165 and hands them to the device. The device identifies the packet as requiring
167 The device performs encryption and authentication of the record data.
171 --
174 and packet transformation functions) the device validates the Layer 4
175 checksum and performs a 5-tuple lookup to find any TLS connection the packet
176 may belong to (technically a 4-tuple
177 lookup is sufficient - IP addresses and TCP port numbers, as the protocol
178 is always TCP). If connection is matched device confirms if the TCP sequence
180 decryption, authentication for each record in the packet). The device leaves
182 Device indicates successful handling of TLS offload in the per-packet context
188 and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)
194 In presence of packet drops or network packet reordering, the device may lose
199 added to the device table and are in TLS_HW mode. For example,
204 TX  section in Resync handling
205 --
208 in similar ways to the receive side-retransmissions - local drops
216 the device with enough information to perform cryptographic operations.
218 segment has to be passed to the device as part of the packet context,
219 together with its TCP sequence number and TLS record number. The device
227 with the previous stream state - assuming that the out of order segment
238 and doesn't change device state. If the segment is in the future, it
239 may imply a local drop, the driver asks the stack to sync the device
244 .. code-block:: c
252 .. code-block:: c
262 --
265 In particular the device should not lose synchronization
268 .. kernel-figure::  tls-offload-reorder-good.svg
269    :alt:	reorder of non-header segment
272    Reorder of non-header segment
278 Segment 2 was dropped so 3 arrives out of order. The device knows
282 The device can, however, collect the authentication algorithm's state
285 of expected window of the device so it's passed as is without special
287 spanning segments 1, 2 and 3. The device did not get out of sync,
293 .. kernel-figure::  tls-offload-reorder-bad.svg
300 Device can only detect that segment 4 also contains a TLS header
302 the device will lose synchronization with the stream.
307 When the device gets out of sync and the stream reaches TCP sequence
309 the device starts scanning for a known header pattern. For example
312 the device continues attempting parsing headers at expected locations
317 When the header is matched the device sends a confirmation request
320 The kernel confirms the guessed location was correct and tells the device
321 the record sequence number. Meanwhile, the device had been parsing
322 and counting all records since the just-confirmed one, it adds the number
324 At this point the device is in sync and can resume decryption at next
327 In a pathological case the device may latch onto a sequence of matching
330 restart scan. Given how unlikely falsely-matching stream is, however,
337 Stack-driven resynchronization
346 The stack waits for the socket to drain and informs the device about
356 TX  section in Error handling
357 --
360 device than the selected TLS offload device. The stack will handle
364 fully acknowledged, so if skbs reach the wrong device they can be handled
367 Any device TLS offload handling error on the transmission side must result
369 due to a bug in the stack or the device, reached the device and can't
373 --
375 If the device encounters any problems with TLS offload on the receive
385 driver if device provides precise error is sufficient).
387 The Linux networking stack does not provide a way of reporting per-packet
408 --------------------
410 The number of connections device can support can be exposed via
414 -------------------------------
418 Overload of the cryptographic subsystem of the device should not have
419 significant performance impact on non-offloaded streams.
424 Following minimum set of TLS-related statistics should be reported
427  * ``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets
429  * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets
431  * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for
433  * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
435  * ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync
437  * ``rx_tls_resync_req_start`` - number of times the TLS async resync request
439  * ``rx_tls_resync_req_end`` - number of times the TLS async resync request
440     properly ended with providing the HW tracked tcp-seq.
441  * ``rx_tls_resync_req_skip`` - number of times the TLS async resync request
443  * ``rx_tls_resync_res_ok`` - number of times the TLS resync response call to
445  * ``rx_tls_resync_res_skip`` - number of times the TLS resync response call to
447  * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
449  * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device
451  * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets
452    passed to the device for encryption.
453  * ``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for
455  * ``tx_tls_ooo`` - number of TX packets which were part of a TLS stream
457  * ``tx_tls_skip_no_sync_data`` - number of TX packets which were part of
458    a TLS stream and arrived out-of-order, but skipped the HW offload routine
461  * ``tx_tls_drop_no_sync_data`` - number of TX packets which were part of
464  * ``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS
473 5-tuple matching limitations
474 ----------------------------
476 The device can only recognize received packets based on the 5-tuple
481 any intermediate software device, therefore a 5-tuple match may
482 consistently miss at the device level. In such cases the device
483 should still be able to perform TX offload (encryption) and should
487 ------------
494 ---------------
496 A device is permitted to perform packet reordering for consecutive
501 -----------------------------------------------------
504 transparently. Enabling device TLS offload should not cause any difference
508 ----------------------------
510 The device should not modify any packet headers for the purpose
513 The device should not depend on any packet headers beyond what is strictly
517 -------------
524 TLS device features
525 -------------------
527 Drivers should ignore the changes to the TLS device feature flags.
529 TLS device feature flags only control adding of new TLS connection
533 offload. Hence, TLS TX device feature flag requires TX csum offload being set.
534 Disabling the latter implies clearing the former. Disabling TX checksum offload
537 Similarly, device-offloaded TLS decryption implies doing RXCSUM. If the user
538 does not want to enable RX csum offload, TLS RX device feature is disabled