The Lightstreamer network protocol is based on TCP. Lightstreamer Server is a stand-alone application (i.e. not based on a web/application server) and it owns direct control over TCP. The TCP protocol offers guaranteed delivery, since it uses backward ACK packets to notify the Server that each piece of information has reached the Client. Lightstreamer’s guaranteed delivery works as follows (we will be referring to the data streamed from the server to the client; for the backward data stream, different mechanisms are applied).
First of all, it depends on the subscription mode. Lightstreamer offers four different modes (RAW, MERGE, DISTINCT, COMMAND). Basically, in MERGE mode subsequent events can be filtered and mixed together, in order to reduce bandwidth and frequency. In DISTINCT mode, subsequent events are never mixed and they are delivered one by one to the Client. In the finance application domain, MERGE mode is typically used for stock prices; DISTINCT mode is typically used for news headlines or order execution notifications in a portfolio.
Let’s suppose that a client subscribes to a portfolio in DISTINCT mode. The maxFrequency parameter of that subscription has been set to “unfiltered”, which implies an “unlimited” ItemEvenBuffer size. A global maximum bandwidth has also been allocated for that streaming session.
The Data Adapter begins to pump updates pertaining to the items of that portfolio into the Lightstreamer Server, that in turn delivers them in real-time to the right clients. The Server guarantees the delivery of the updates in the original order. The possible scenarios for the delivery of each packet are the following:
- The packet reaches the client. A TCP acknowledge is sent back to the Server, which can keep on delivering the updates. Of course, the custom client application has to display the updates upon receipt. But if it’s not defective, this will predictably happen.
- The packet does not reach the client because the allocated maximum bandwidth (if it was allocated) is currently saturated OR there is a network congestion preventing the packets from being delivered. In these cases, the Server buffers the events until the network allows it to release them. Should the internal buffer (whose maximum physical size depends on the memory of the server box) get full (quite an impossible event for a portfolio), then one or more updates would be LOST: in this case a notification is sent to the client with high priority (on the same stream connection but in front of the buffered updates) so that the client application can choose to take some recovery action. If even the delivery of the notifications is impossible, it means there is a serious network problem and we fall into case 3.
- The packet does not reach the client because the stream connection was closed due to a network timeout or the Server crashed. In this case, instantly or after a configurable timeout for missing heartbeat packets, the client automatically and transparently aborts the current session and opens a fresh connection to the Lightstreamer Server cluster. Then, it resubscribes to all the current items (without the user having to do anything), while requesting for a “snapshot”. This means that the Server will send back the current image of the items, to guarantee that no lost events occurs during the transition between the old session and the new one. The snapshot for DISTINCT-mode items is made up of a short story of updates, with configurable depth.
DISTINCT mode was used in this example for the sake of simplicity. In reality a portfolio is easier to handle through the COMMAND mode. In COMMAND mode all the considerations above apply when delivering new rows. But it keeps the ability to merge events pertaining to the same row. This means that if you deliver an order submission confirmation to the client and after half a second you deliver to the client the execution notification of that order, if the Client establishes a new session (in case 3), as a snapshot it will receive the updated item (execution notification), not the redundant history (order submission covered by a order execution). This improves efficiency and bandwidth usage a lot. Obviously, if the client needs the full history of the Portfolio instead of “current-moment coherency”, DISTINCT mode can be used.
If explicit receipt acknowledgment is needed by the application, the client can send a message to the Server, so that the Adapter can process the acknowledgment.
