Documentation/networking/rds.rst

1 .. SPDX-License-Identifier: GPL-2.0
14 http://oss.oracle.com/pipermail/rds-devel/2007-November/000228.html
22 cluster - so in a cluster with N processes you need N sockets, in contrast
23 to N*N if you use a connection-oriented socket transport like TCP.
25 RDS is not Infiniband-specific; it was designed to support different
29 The high-level semantics of RDS from the application's point of view are
33 	RDS uses IPv4 addresses and 16bit port numbers to identify
39 	transport has to be IP-based. In fact, RDS over IB uses a
43 	The port space is entirely independent of UDP, TCP or any other
59 	a active-active HA scenario), but only as long as the address
72 	to create RDS sockets. SOL_RDS is the socket-level to be used
87 	This is a soft limit rather than a hard limit - RDS will
90 	mark the port as "congested" and send a congestion update to
92 	processes sending to this congested port.
95 	This binds the socket to a local IP address and port, and a
105 	return with -EMSGSIZE
117 	drops below SO_SNDSIZE, the port is marked uncongested, and
141 	a destination marked congested - in this case you will loop
144 	this - by using congestion notifications, and by checking for
159 	Set or read an integer defining  the underlying
164 	This socket option may only be set exactly once on the socket,
166 	set SO_RDS_TRANSPORT on a socket for which the transport has
169 	An attempt to set SO_RDS_TRANSPORT to RDS_TRANS_NONE will
175   see rds-rdma(7) manpage (available in rds-tools)
194 	  per-packet sequence number
200 	  source port
202 	  destination port
222 	  passing RDMA-related information.
232       if another HCA is available for fail-over.
237       packets.  Ack-only packets are reduced by only allowing one to be
244       RDS's IB transport uses a credit-based mechanism to verify that
255       should return block or return -EWOULDBLOCK.
259       "back-pressure" is considered a bug.
270       is much easier to implement than some finer-grained
271       communication of per-port congestion.  The sender does a very
272       inexpensive bit test to test if the port it's about to send to
279   As mentioned above, RDS is not IB-specific. Its code is divided
303     code to handle the actual copy-to-user.
306     per-socket information
309     per-connection information
312     pointers to transport-specific functions
315     non-transport-specific statistics
328   then maintained forever -- if there are transport errors, the
329   connection will be dropped and re-established.
332   partially-sent datagrams to be retransmitted when the connection is
333   re-established.
340     - struct rds_message built from incoming data
341     - CMSGs parsed (e.g. RDMA ops)
342     - transport connection alloced and connected if not already
343     - rds_message placed on send queue
344     - send worker awoken
347     - calls rds_send_xmit() until queue is empty
350     - transmits congestion map if one is pending
351     - may set ACK_REQUIRED
352     - calls transport to send either non-RDMA or RDMA message
356     - allocs work requests from send ring
357     - adds any new send credits available to peer (h_credits)
358     - maps the rds_message's sg list
359     - piggybacks ack
360     - populates work requests
361     - post send to connection's queue pair
367     - looks at write completions
368     - unmaps recv buffer from device
369     - no errors, call rds_ib_process_recv()
370     - refill recv ring
373     - validate header checksum
374     - copy header to rds_ib_incoming struct if start of a new datagram
375     - add to ibinc's fraglist
376     - if competed datagram:
377 	 - update cong map if datagram was cong update
378 	 - call rds_recv_incoming() otherwise
379 	 - note if ack is required
382     - drop duplicate packets
383     - respond to pings
384     - find the sock associated with this datagram
385     - add to sock queue
386     - wake up sock
387     - do some congestion calculations
389     - copy data into user iovec
390     - handle CMSGs
391     - return to application
395   Mprds is multipathed-RDS, primarily intended for RDS-over-TCP
397   implementation of RDS-over-TCP is implemented by demultiplexing multiple
399   port]) over a single TCP socket between the 2 IP addresses involved. This
402   (a) upper-bounded to the single-flow bandwidth,
403   (b) suffers from head-of-line blocking for all the RDS sockets.
406   by having multiple TCP/IP flows per rds/tcp connection, i.e., multipathed
409   (e.g., of local address and RDS port number) and packets for that RDS
414   a common (to all paths) part, and a per-path struct rds_conn_path. All
416   Transports such as TCP that are multipath capable may then set up a
417   TCP socket per rds_conn_path, and this is managed by the transport via
422   transport is multipath-capable, rds_sendmsg() hashes outgoing traffic
424   local address and port that the PF_RDS socket is bound to.
436   port 0) with the ping packet having a rds extension header option  of
439   get sent from some reserved port, RDS_FLAG_PROBE_PORT (in <linux/rds.h>)
442   sent in response to a probe-ping should contain the rcvr's npaths
443   when the rcvr is mprds-capable.
445   If the rcvr is not mprds-capable, the exthdr in the ping will be
447   of the probe-ping can default to single-path mprds.