How GCB Works

This section explains detailed mechanism of GCB. Even if you are not interested in gory detail of GCB, you may still want to read this section because this will help you understand better how things are going underneath and what have possibly gone wrong when you have a problem with GCB. First, what components are involved in GCB and how they interact each other will be explained. Second, I will explain how the address assigned to a server socket is passed to a client(s) through a brief review of ordinary Berkeley socket API. And then how GCB provides connectivity over firewall/NAT will be explained by showing what happens in GCB when the applications call GCB socket API.

What goes where

In a GCB connection setup and data communication, three entities are involved: a client, a server, a daemon called GCB inagent. In fact, GCB inagent consists of three daemon processes. However, they are logically one entity. The following picture shows multiple clients, servers, and one GCB inagent that arranges communications between those clients and servers.

Both servers that want to accept connections from outside their private (firewalled) network and clients that want to connect to a server inside a private (firewalled) network must call GCB socket calls. This can be done by rewriting the applications or by using an interposition mechanism so that applications' regular socket calls are translated into GCB calls.

To enable inbound connections, each private (firewalled) network must have a GCB inagent that arranges connections from outside the network. The inagent must be installed on a machine that both nodes in the public and the private (firewalled) network can directly talk to. However, the inagent need not be able to initiate connections to the nodes. Therefore, the inagent is generally installed on the boundary or just outside of the network that allows outbound connections.

Review of socket API

The following picture shows a general sequence of socket calls in TCP. UDP has a slightly different APIs, but the difference is not important here because we are mainly interested in how an address is assigned to a server socket and passed to client applications.

On the assumption that you are quite familiar with these calls, I would explain few things.

• A socket is assigned an address--IP and port--through bind call. In Berkeley socket system, an application may call bind with unspecified IP--INADDR_ANY in POSIX terminology--and/or unspecified port--0 in network byte order, meaning that the operating system must choose whatever IP and/or port available. Well-known services such as http and ssh must call bind with a specific port designated for the service. Otherwise, applications should call bind with unspecified port because they don't know what ports are available in the system. On the other hand, calling bind with a specific IP is not a good practice in general because it prevents connections toward other IPs of the machine, e.g. loopback IP, from being accepted.
• getsockname asks operating system what address the socket is bound to. If bind is called with a specific IP and port, then the application doesn't need to call this function because it already knows that address. However, if bind was called with unspecified IP and/or port, then it needs to call getsockname to learn what address the socket is bound to.
• Since the client must pass server's IP and port to connect call, it needs somehow get the address that the server is listening on. If the server socket does not use a well-known address, then the server's address must be somehow passed to the client. One popular way is to pass it via another connection that has been established before--ftp data channel setup corresponds to this. In ftp, the address of the server socket for data transfer is passed via the control channel. Another way, though very unlikely, may be that your friend calls you to tell what IP and port you should use to connect to her/his server program. On the other hand, if the server uses a well-known address, then the server does not explicitly pass the address to clients. Yet, we may still say that the server implicitly passes its address to (indefinite) clients through some mechanism such as advertisement and pre-agreement.

Detailed mechanism

Since GCB provides the same syntax and semantics of Berkeley socket calls, GCB mechanism can be explained best by showing what happen in GCB layer when the application calls GCB socket calls. I use the simplest scenario of socket calls here to make the explanation not too complex. GCB, however, does support much more complex scenarios such as nonblocking connect, multiplexing socket fds and other fds by using select, concurrent server by forking and execveing a child process per a connection.

Binding, registration, and deregistration

To be able to accept connections from outside, server sockets inside private network or behind firewall must be locally bound, registered at the GCB inagent, and officially bound.

Local binding is nothing new. As in bind of Berkeley socket, an (IP, port) pair called local address is assigned to a socket. Since connections toward the socket inside a private network or behind a firewall are impossible, those connections are arranged by the GCB inagent that manages the private or firewalled network. To do this, the information of the server socket must be registered at the inagent. This process is called registration.

Official binding needs a little longer explanation. It is the process of assigning official address to the server socket behind a firewall/NAT. Official address is the address that must be known to clients. In other words, it is the address that clients pass to connect call. Because the address of the server socket is passed from the server to clients as explained in Review of socket API official address may also be defined as the address that is known to the server application. GCB uses the idea of address leasing. It creates a proxy socket in the public network per server socket behind a firewall/NAT and uses the address of the proxy as the official address of the server socket. Reasons behind this address leasing are:

1. Private IP is not routable in the public network. Therefore, we need to assign a world-routable address to private sockets.
2. Clients outside server's network must contact the GCB inagent of the server, but non-GCBnized sockets do not know the existence. Therefore, we need a proxy socket in the public network so that connections to the proxy may be forwarded to the server socket.

The following picture shows the timeline of how a server socket behind a firewall/NAT is locally bound, registered at the inagent, and officially bound. The rightmost column shows socket calls that the server application calls. Two columns in the middle show what happen in the GCB layer of the server and GCB inagent, respectively.

When the application calls socket, GCB layer creates a socket as requested and records that the socket is a GCB socket.

The application calls bind, then GCB layer binds the socket to an (IP, port) pair by calling bind with the same argument--'addr-1' in this case. At this point, we say the socket is locally bound to 'addr-2'. The important thing to note is that 'addr-1' must have unspecified IP, i.e. INADDR_ANY. When the application sees a bind call with a specific IP succeed, it knows that the socket is bound to that IP. Because that IP is already known to the application, it is the official IP. Therefore, the IP part of the official address is already determined by the application and GCB has no freedom to assign the official address. When bind is called with a specific IP, then GCB gives up and treats the socket as non-GCB socket. Note that 'addr-2' must be different from 'addr-1' since the latter may have unspecified IP.

As the next step, GCB layer makes a TCP connection to GCB inagent and sends a BIND command with 'addr-1' and 'addr-2' through this connection. This management connection is maintained open during the lifetime of the server socket and all GCB commands related to the socket are exchanged via this connection.

Upon accepting a new connection and receiving a BIND request, GCB inagent does:

• Create a proxy socket and binds it to a public address--'addr-3' in the picture. If the port of 'addr-1' is specified, the socket is bound to the port. In this way, GCB supports well-known services.
• Record necessary information about the socket, including local address, official address, the status, etc.
• Reply the result by sending BIND_OK with the official address or BIND_NAK with an error code.

listen call is handled similarly and I don't think it deserves any explanation.

When the application calls getsockname to learn what (IP, port) the socket is bound to, GCB layer returns the official address instead of the local address that the socket is actually bound to. Note that the local address is hidden from the application and only the official address is known to it.

Even though the picture does not show any timeline for close, I believe that this is the best place to explain the deregistration of the server socket from the inagent. When the application calls close for the server socket, GCB layer close the management connection then GCB inagent deletes the information of the server socket.

TCP intra network connection

This section explains how a connection from a client in the same network as the server is made. Remember that, through the process explained in the previous section,

1. the server socket gets registered at its inagent and
2. the official (IP, port) has been given (or leased) to the server socket and then passed to the client through some out-of-band mechanism as explained in Review of socket API

When the client application calls connect, GCB layer searches GCB routing table, which basically tells whether GCB mechanism should be used or not given a server's address. If GCB mechanism must be used, GCB layer binds the client socket to an (IP, port)--'addr-4' in the picture and then sends CONNECT request to server's GCB inagent with client's address, server's address, and whether it can receive a connection from outside its network. How does the GCB layer know the address of the inagent? The address is calculated from the server's address. The GCB inagent is supposed to be listening on (server socket's IP, 65432) for commands from clients.

GCB inagent replies the GCB layer ACTIVE with the local address of the server socket, meaning that the client can directly connect to the local address since it is in the same network. Then the GCB layer connects the socket to the address and returns the result to the client application.

Why the GCB layer of the server puts the accepted connection into the queue is unclear at the moment. This will be explained in TCP public-to-private connection

TCP public-to-private connection

The following picture shows the timeline of a connection setup from a client in the public network to a server behind a firewall/NAT. As intra connection case, the server socket is registered at its inagent.

GCB layer at the client side sends CONNECT request to server's GCB inagent with the same arguments as intra connection in the previous section. Note that the last argument of the request--the flag if the client can accept connections or not--must be 'true' because the client is in the public network.

GCB inagent responses PASSIVE to the GCB layer, meaning that the client can't connect to the server so it should accept the connection from the server. Then the GCB layer makes the client socket passive by calling listen and then responses that it is ready to accept a connection. Next, the inagent sends CONTACT command to the GCB layer of the server through the management connection that was setup during the registration of the server socket. Then a connection is made from the server to the client and is returned to the application as if it were made from the client to the server.

Returning the connection made in the opposite direction to the application needs a little more explanation. accept function returns a new file descriptor, while connect does not. Therefore the GCB layer at the client side must replace the original socket descriptor with the new descriptor that accept returns before it returns the result of client's connect call. On the other hand, the GCB layer at the server side must create a new socket to connect to the client and then return this new descriptor to the server application when the application calls accept.

Connections to the server socket are made either in the normal way, i.e. by clients connecting to the server's local address as intra connection case, or in the reverse way as explained in this section. Connections made in the normal way are maintained in the kernel connection queue, while those made in the reverse direction are not queued in the kernel. Furthermore, when the application calls accept, the connection accepted first both from those made in the normal direction and in the reverse direction. Therefore, the GCB layer at the server side must maintain accepted connections, no matter what direction they are made, in a single queue.

TCP legacy-to-private connection

Legacy clients may not know GCB protocol and will try to directly connect to the official address of the server socket. Remember that a GCB server socket has a proxy socket listening on the official address at the GCB inagent. This is done through the bind/registration process explained earlier.

When a connection is accepted to the proxy socket, the GCB inagent creates another proxy socket and sends the GCB layer of the server CONTACT request with the address of the second proxy--'addr-4' in the picture. Then the GCB layer makes a connection to the proxy socket and the inagent accepts the connection. Data communication is relayed by the inagent.

Actually the second half of the relay channel is made from the server to the inagent only when connections from the inagent to the server are not allowed. Placing the inagent outside of the network it manages is a representative reason for this condition true. If it can talk to the server, the inagent makes a connection for the second half of the channel.

TCP private-to-private connection

When a client and a server are in different private or firewalled networks, neither can make a connection to the other and GCB's mechanism of reversing the direction of connections cannot be used. The following picture shows the situation.

GCB uses the relay mechanism to support private(firewall)-private(firewall) communications. The following picture shows the timeline.

When the GCB inagent receives a CONNECT request from a client that cannot receive a connection, it responses ACTIVE with the official address of the server socket, i.e. the address that the proxy socket is listening on. The connection from the client to the proxy socket is treated as the same way as the connection from a legacy client. See TCP legacy-to-private connection

UDP connection

Though there is no connection setup in UDP communications, we still can define a similar concept. A UDP session is defined as a series of UDP exchanges each of which occurrs before a predefined timeout from the previous one. A UDP session whose first packet is sent by a node inside a private or firewalled network is defined as an outbound session. Inbound session is defined in a similar way.

Generally outbound UDP sessions are allowed. To enable outbound UDP sessions, the firewall/NAT creates an association between sender's and receiver's addresses when it sees the first outbound packet from the sender the receiver. Of course the association gets deleted if it has not seen a packet between the peers for a while.

To support inbound UDP sessions, GCB has to do a similar thing. It maintains a timeout-based mapping table. Each entry in the table maps the official address 'addr-1' of a peer to an address 'addr-2' meaning that packet toward 'addr-1' must be sent to 'addr-2' When a GCBnized application calls connect or sendto, GCB layer searches the mapping table for the receiver's address. If an entry is found, then the layer sends the packet to the mapped address. Otherwise, it contacts the GCB inagent of the receiver and opens a channel in a similar way as TCP cases.

Here, I will show how a UDP channel is setup from the public network to a private or firewalled network. Channels in other settings are established in similar ways. The following picture shows the timeline. Leftmost column corresponds GCB socket calls that the application in the public network issues. Rightmost column corresponds the application behind a firewall/NAT.

When the sender application calls sendto, GCB layer searches the mapping table for the address of the receiver--'addr-3' in this example. If an entry for the address is found, then it sends the data to the mapped address and returns immediately. Otherwise, it resorts to the following procedure.

After looking up the routing table as in TCP cases, the GCB layer sends the GCB inagent of the receiver INIT_COMM command with the same argument as CONNECT request in TCP cases. Then the inagent replies PASSIVE, meaning that at least one packet must be sent from the receiver socket to the sender socket. The inagent also sends TRIGGER to the GCB layer of the sender. TRIGGER command has the same argument as CONTACT in TCP cases.

Next, the GCB layer of the receiver sends DUMMY packet. The DUMMY message is sent from the receiver socket to the sender socket. Because at least a packet is sent from the receiver to the sender socket, the sender can now send data to the receiver. Upon receiving a DUMMY, the GCB layer of the sender replies DUMMY_ACK to notify that it has received DUMMY successfully. It sends the data to the receiver socket, too.

The GCB layer must be able to handle delayed DUMMY messages. Note that DUMMY messages are delivered to the socket for data transfer. Therefore, the GCB layer must examine UDP packets and queue data packets so that it may pass them to the application when it calls recvfrom.