The Kiwi Reliable Delivery Protocol (KRDP)

Top Previous Next

Background:

The Kiwi Reliable Delivery Protocol was designed to solve the problem of losing data when a TCP connection is broken due to a network failure.

KRDP uses the TCP protocol as the underlying transport. This ensures that each packet sent is sequenced and acknowledged when received. The TCP protocol on the receiving system handles the packet order and ensures that any missing packets are resent.

The problem:

TCP works well as a reliable transport when the connection can be opened and closed cleanly. During a TCP close handshake, any outstanding packets are usually received and acknowledged before the connection is closed.

However, if a break in the network occurs during message sending, the sender will continue to send packets until the TCP window size is reached. When no acknowledgement is received after a timeout period, the Winsock stack will fire a timeout event. When this happens, it is not possible to know exactly which message (or part message) was last received and acknowledged by the remote end. Any data that was sitting in the Winsock stack's buffer will be lost. Depending on the TCP window size and the speed of the data being sent, this could be hundreds of lost messages.

The solution:

KRDP works by adding another acknowledgement and sequencing layer over the top of the TCP transport. KRDP wraps each syslog message with a header which contains a unique sequence number. The KRDP sender keeps a local copy of each message it has sent. The KRDP receiver periodically acknowledges receipt of the last KRDP wrapped syslog message it has received. The KRDP sender can then remove all locally stored messages up to the last acknowledged sequence number. When the connection is broken and re-established, the receiver informs the sender which messages need to be resent.

Each KRDP sender is identified with a unique connection name. This allows the sender and receiver to re-establish the same session and sequence numbers, even if the IP address or sending port of the sender has been changed due to DHCP etc.

Unique message sequencing:

Each KRDP message is identified with a unique sequence number. The sequence starts at 1 and increments in steps of 1 up to 2147483647 (2 billion), then wraps around to 1 again. The message number 0 is used to indicate that the system does not know the last sequence number and that it has had to assume a fresh start. If this occurs, both the sender and receiver will log an error to note the lost messages.

Dealing with international characters:

Unicode allows the mapping of all international character sets into a known byte sequence. The mapping of non US-ASCII characters requires the use of more than a single byte per character. The most commonly used way of sending these multi-byte characters over TCP is to use UTF-8 encoding. The KRDP sender will encode the syslog messages as UTF-8 and the KRDP receiver will decode them back to Unicode again.

The KRDP message format:

Sender (S)

Receiver (R)

Message Types (MsgType):

00 = SenderID

01 = ReceiverResponse

02 = Sequenced message

03 = Message acknowledgement

04 = Receiver KeepAlive

99 = Error message

Message format:

KRDP AA 0000000000 Message<CR>

KRDP = Unique tag

Space (ASCII 32)

AA = Msg type (as above)

Space (ASCII 32)

0000000000 = Sequence number 0 to 2147483647

Space (ASCII 32)

Message = UTF-8 encoded message text

<CR> = Carriage return character ASCII 13 to indicate end of message stream

Sequence of events:

S connects via TCP

S sends first ID packet (MsgType 00)

R responds with ReceiverResponse message (MsgType 01)

S sends sequenced messages (MsgType 02)

Rules:

1.	If the first message R receives is not a ID message (MsgType 00), R disconnects. (Any data received is ignored).

2.	If R does not receive ID message after 60 seconds, R disconnects.

3.	After S sends the ID message, S will wait up to 60 seconds for a ReceiverResponse message. If there is no response, S will disconnect session.

4.	R sends ACK messages to S with the next expected message sequence

5.	ACK messages are sent no more frequently than once every 200ms

Message formats:

MsgType 00 (Version and SenderID)

KRDP 00 PV UniqueKey<CR>

The unique key identifies the channel and is used to synchronise the message numbers

PV = Protocol Version to use. 01 = KRDP Reliable/Acknowledged

Unique key format is free form.

An example would be: "IP=192.168.1.1, Host=myhost.com, ID=Instance1"

Or, just: "Instance1"

Since the receiver might already have an "Instance1" name from another source, the first UniqueKey would be better. Use as much information to uniquely describe the source of the messages

MsgType 01 (ReceiverResponse message)

KRDP 01 0000000000 Listener ID<CR>