Method for binding a socket to a network device under Linux

1. Causes

The cause of the incident is that I am going to use two CDMA modems to extend the bandwidth of the point-to-point connection, and I hope to achieve load balancing between the two modems. But unfortunately, China Unicom's access equipment does not support Multilink-PPP. So, there is no way, I have to implement load balancing by myself. There are several ways to implement load balancing. One way to do this on the network is to use iproute2 to complete packet-level load balancing, which is implemented at the kernel level. But I don't want to give everything to the kernel to complete, I hope to control the traffic on each modem. So what should I do?

2. Solution

At first, the way I thought of was to create two sockets and then bind each socket to a local IP address. I thought this would This will cause data to be sent from the network device where the IP address is bound. But practice has proved that this idea is wrong. Because each time before sending a packet, the kernel looks up the routing table to decide which network interface to send the packet from. Once a suitable routing entry is found, the packet is sent out from the network interface indicated by the routing entry. Thus, there is a problem. Since the routing table is cached, each time a packet is sent, the interface that sends the packet will have a greater chance of being selected by the kernel again. In the worst case, one modem will be too busy, while the other modem will be “no one cares”. This obviously violates my original intention. The test results show that when a few hundred KB of data is sent on one modem, only dozens of Bs are still sent on the other one. It seems that this road is nowhere! Following this line of thought, a slightly confusing practice is to adjust the routing table before sending a packet. Adjusting the routing table is easy. But it is too much trouble to do so, so this idea has also been abandoned by me. I have not even tested whether this method is feasible, but in theory it works. Moreover, in the method introduced on the Internet, the load level of the routing level seems to be realized in this way, but it is just like it, I did not go into it. After several unsuccessful attempts, I moved "UNIX Network Programming" and sifted the UDP-related parts. Only found one that might have worked: you can set a socket option via setsockopt(): SO_DONTROUTE, but the effect of this option is rather vague. The vagueness is vague, I actually tried it, the result is still not, the reason is unknown. So, I had to go back to Linux itself and read it against a lot of man-made manuals. When I saw socket(7), I suddenly saw a socket option that made my eyes shine: SO_BINDTODEVICE. Literally, this option should be what I want. Subsequent test results prove that this is indeed the case. The description of the socket option in socket (7) is as follows: SO_BINDTODEVICE Bind this socket to a particular device like "eth0", as specified in the passed interface name. If the name is an empty string or the option length is zero The socket option is a variable-length null terminated interface name string with the maximum size of IFNAMSIZ. If a socket is bound to an interface, only packets received from that particular interface are processed by the socket. Note that this only works for some socket types, particularly AF_INET sockets. It is not supported for packet sockets (use normal bind(8) there). Here, I am directly copying it. However, the last bind(8) is definitely wrong, obviously it should be bind(2). Regardless of it, this is not something I am going to solve now. The central meaning of this passage is that after the socket is bound to the specified network device interface, only the packets from the device will be processed by the socket. So, what if the socket is sending packets out? Will it only be sent from this network interface? The awful thing is that it is not said here. But it doesn't matter, we will know if we try it. In the beginning, I took it for granted that it could look like this: char *dev = "ppp0"; int sock1 = socket(AF_INET, SOCK_DGRAM, 0); setsockopt(sock1, SOL_SOCKET, SO_BINDTODEVICE, dev, sizeof(dev)); Practice once again proves that I am wrong. But what can I do? The instructions in socket(7) are so embarrassing. I can't see the clues. It is justifiable. However, there is google in hand, what am I afraid of this little problem? So google, quickly found the crux of the problem: under Linux, the reference to the network device is completed by struct ifreq. The description of the structure in netdevice(7) is as follows: struct ifreq { char ifr_name[IFNAMSIZ];/* Interface name */union { struct sockaddrifr_addr; struct sockaddrifr_dstaddr; struct sockaddrifr_broadaddr; struct sockaddrifr_netmask; struct sockaddrifr_hwaddr; short ifr_flags; int Ifr_ifindex; int ifr_metric; int ifr_mtu; struct ifmapifr_map; char ifr_slave[IFNAMSIZ]; char ifr_newname[IFNAMSIZ]; char * ifr_data; }; }; Here, I only need the ifr_name member field is enough. The code is modified as follows: struct ifreq if_ppp0; struct ifreq if_ppp1; strncpy(if_ppp0.ifr_name, "ppp0", IFNAMSIZ); strncpy(if_ppp1.ifr_name, "ppp1", IFNAMSIZ); sock1 = socket(AF_INET, SOCK_DGRAM , 0); sock2 = socket(AF_INET, SOCK_DGRAM, 0); if (setsockopt(sock1, SOL_SOCKET, SO_BINDTODEVICE, (char *)&if_ppp0, sizeof(if_ppp0)) /*error handling*/} if (setsockopt(sock2 , SOL_SOCKET, SO_BINDTODEVICE, (char *)&if_ppp1, sizeof(if_ppp1)) /*error handling*/} Then, in the main part of the program, each time a packet is sent on sock1, it will also be on sock2. Send a packet, and there is no action in the program to receive data. Since all packets are equal in size, it can be expected that the amount of data sent on the two network interfaces should be comparable. The test results are powerful. This conjecture is supported: the amount of data sent on interface ppp0 is 702KB after running the program for a period of time. The amount of data sent on the ppp1 interface is 8Array5KB. Although it still differs by nearly 200KB, in any case, it has improved a lot compared to the original situation. As for why there is such a gap of 200KB, the author is also looking for reasons. It165.net 3. More Conclusions For the SO_BINDTODEVICE socket option, the authors have drawn the following conclusions after fully reading the man page: (1) For TCP sockets, UDP sockets, RAW sockets, you can use the SO_BINDTODEVICE socket option. Bind the socket to the specified network interface. After binding, all data packets of the socket are sent and received only through the specified network interface; (2) For the PACKET type socket, it cannot be bound to the specified network through SO_BINDTODEVICE. On the interface, it is bound to a specific network interface through bind(2). The socket address structure used is struct sockaddr_ll. The socket address structure is the link layer address structure, which is independent of the specific network device. For example, the address structure can be used to represent both PPP devices and ethernet devices. (3) The SO_BINDTODEVICE socket option is only available for Linux systems. If you want to write a program running on a multi-operating system platform, you can't rely on SO_BINDTODEVICE to complete the binding of the socket to the specific device. However, the author did not test the TCP socket and RAW socket. For the PACKET socket, the above conclusion is credible, because I read the source code of dhcpd and found that the PACKET socket is bound to the specified network interface through bind(2).