Documentation/networking/snmp_counter.rst

17 .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
30 .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
41 .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
60 .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
73 .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
81 .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
98 .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
111 .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
118 .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
168 .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169 .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170 .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172 .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173 .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174 .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175 .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176 .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177 .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178 .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
180 .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181 .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182 .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184 .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185 .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186 .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187 .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188 .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189 .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190 .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
204 .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
221 .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222 .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
230 ---------------------------------
255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
284 .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
286 It means the TCP layer sends a SYN, and come into the SYN-SENT
294 .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
297 the SYN-RCVD state.
320   retransmission but including data-in-SYN). This counter is different from
330   retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
355 would complete the 3-way handshake. As the accept queue is full, TCP
356 stack will keep the socket in the TCP half-open queue. As it is in the
360 queue, if it is full, keeps the socket in the half-open queue, at next
371 .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
377 .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
386 .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
408 The spurious retransmission timeout detected by the `F-RTO`_
411 .. _F-RTO: https://tools.ietf.org/html/rfc5682
422   - A zero window was announced from us
423   - zero window probing
425   - Out of order segments arrived.
426   - Urgent data is expected.
427   - There is no buffer space left
428   - Unexpected TCP flags/window values/header lengths are received
430   - Data is sent in both directions. The fast path only supports pure senders
433   - Unexpected TCP option.
470 .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
473 will return immediately and kernel will try to send the in-flight data
476 wait for the in-flight data are acked by the other side, the max wait
504 .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
528 .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
620 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
714 likely to re-transmit any packets, and we still receive an invalid
724 to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
770 .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
775 Packets are dropped by PAWS in Syn-Sent status.
779 Packets are dropped by PAWS in any status other than Syn-Sent.
791 .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
795 The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
798 in the Syn-Recv status. But in several scenarios, the TCP stack need
810 numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
811 or Time-Wait statuses, the skipped ACK would be counted to
819 check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
823 The ACK is skipped in Fin-Wait-2 status, the reason would be either
828 The ACK is skipped in Time-Wait status, the reason would be either
840 .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
841 .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
842 .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
849 window to zero. But the receive window might still be a no-zero
856 The TCP receive window is set to zero from a no-zero value.
860 The TCP receive window is set to no-zero value from zero.
897 .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
910 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
917 When the TCP stack receives an ACK packet in the SYN-SENT status, and
928 after the 3-way handshake, the retransmission timeout happens
929 net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
946 fastopenq->max_qlen, the TCP stack will reject the fast open request
949 TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
1031 ---------
1034   nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1038   --- 8.8.8.8 ping statistics ---
1044   nstatuser@nstat-a:~$ nstat
1074 tcp 3-way handshake
1075 -------------------
1078   nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1083   nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1087 completed the 3-way handshake.
1091   nstatuser@nstat-b:~$ nstat | grep -i tcp
1099   nstatuser@nstat-a:~$ nstat | grep -i tcp
1105 SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1108 of the 3-way handshake is a pure ACK without data, so
1111 When the client sent SYN, the client came into the SYN-SENT state, so
1116 TCP normal traffic
1117 ------------------
1120   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1125   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1126   Connection to nstat-b 9000 port [tcp/*] succeeded!
1130   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1131   Connection to nstat-b 9000 port [tcp/*] succeeded!
1136   nstatuser@nstat-a:~$ nstat
1151   nstatuser@nstat-b:~$ nstat
1164   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1165   Connection to nstat-b 9000 port [tcp/*] succeeded!
1171   nstatuser@nstat-a:~$ nstat
1187   nstatuser@nstat-b:~$ nstat
1199 Compare the first client-side nstat and the second client-side nstat,
1201 but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1202 nstat and the second server-side nstat had a difference too: the
1203 second server-side nstat had a TcpExtTCPHPHits, but the first
1204 server-side nstat didn't have it. The network traffic patterns were
1215   nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1216   Netid    Recv-Q     Send-Q            Local Address:Port             Peer Address:Port
1240 ---------------------
1260   nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1264 read it yet. We type Ctrl-C to terminate the server script. Then we
1267   nstatuser@nstat-b:~$ nstat | grep -i abort
1271 RST after we type Ctrl-C.
1274 ---------------------------------------------------
1279   sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1283   nstatuser@nstat-a:~$ cat client_orphan.py
1287   server = 'nstat-b' # server address
1305   nstatuser@nstat-b:~$ cat server_orphan.py
1333   sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1335 Type Ctrl-C on client, stop client_orphan.py.
1339   nstatuser@nstat-a:~$ nstat | grep -i abort
1344   nstatuser@nstat-a:~$ ss -s
1349   *         0         -         -
1359 the client, type Ctrl-C on client_orphan.py, the system of the client
1368 the 'ss -s' command shows the system has 10 orphan sockets, and the
1372 exactly orphan socket count by the 'ss -s' command, but when kernel
1387 iptables on the server blocked the traffic, the server wouldn't receive
1392   nstatuser@nstat-a:~$ nstat | grep -i abort
1396 ----------------------
1399   nstatuser@nstat-b:~$ cat server_linger.py
1414   nstatuser@nstat-a:~$ cat client_linger.py
1418   server = 'nstat-b' # server address
1423   s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1429   nstatuser@nstat-b:~$ python3 server_linger.py
1433   nstatuser@nstat-a:~$ python3 client_linger.py
1437   nstatuser@nstat-a:~$ nstat | grep -i abort
1441 --------------------
1462   server = 'nstat-b'
1469   nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1471 We use '-i' to come into the interactive mode, then a packet::
1483   ubuntu@nstat-b:~$ nstat
1501 -------------------------------------------
1504   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1509   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1510   Connection to nstat-b 9000 port [tcp/*] succeeded!
1519   nstatuser@nstat-b:~$ nstat -n
1523   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1533   nstatuser@nstat-b:~$ nstat
1549 -------------------------------------------------
1558   $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1559   $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1560   $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1561   $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1570   $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1574   $ nc -v 8.8.8.8 53
1588 re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1594   $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1605   $ nc -v 8.8.8.8 53
1630   $ ping -c 1 8.8.8.8
1644 --------------------------
1646 first SYN will let server create a socket, set it to Syn-Recv status,
1655   nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1656   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1660   nstatuser@nstat-a:~$ nc nstat-b 9000
1662 As the nstat-b didn't listen on port 9000, it should reply a RST, and
1668   nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1670 On nstat-b, we run nc to listen on port 9000::
1672   nstatuser@nstat-b:~$ nc -lkv 9000
1675 On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1676 RST to nstat-b::
1678   nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1680 Send 3 SYN repeatedly to nstat-b::
1682   nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1684 Check snmp counter on nstat-b::
1686   nstatuser@nstat-b:~$ nstat | grep -i skip
1692 -----------------------
1695 On nstat-b, let nc listen on port 9000::
1697   nstatuser@nstat-b:~$ nc -lkv 9000
1700 On nstat-a, run tcpdump to capture a SYN::
1702   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1703   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1705 On nstat-a, run nc as a client to connect nstat-b::
1707   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1708   Connection to nstat-b 9000 port [tcp/*] succeeded!
1713   nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1717   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1719 On nstat-b, check the snmp counter::
1721   nstatuser@nstat-b:~$ nstat | grep -i skip
1725 failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1729 ----------------------
1739 On nstat-b, open two terminals, run two nc commands to listen on both
1742   nstatuser@nstat-b:~$ nc -lkv 9000
1745   nstatuser@nstat-b:~$ nc -lkv 9001
1748 On nstat-a, run two nc clients::
1750   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1751   Connection to nstat-b 9000 port [tcp/*] succeeded!
1753   nstatuser@nstat-a:~$ nc -v nstat-b 9001
1754   Connection to nstat-b 9001 port [tcp/*] succeeded!
1756 On nstat-a, run tcpdump to capture an ACK::
1758   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1759   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1761 On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1764   nstatuser@nstat-b:~$ nc -lkv 9001
1766   Connection from nstat-a 42132 received!
1769 On nstat-a, the tcpdump should have captured the ACK. We should check
1772   nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1773   State  Recv-Q   Send-Q         Local Address:Port           Peer Address:Port
1780 …nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r…
1782 Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1784   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1786 Check TcpExtTCPACKSkippedSeq on nstat-b::
1788   nstatuser@nstat-b:~$ nstat | grep -i skip