MERGE Mode
Many data sources can be defined as filterable.
Data filtering is a possibility offered by the intrinsic nature of some types of information. All information sources that over time generate updated versions of the same data, more or less frequently, can quite easily be subjected to filtering algorithms.
Examples of filterable data feeds:
- Stock prices. The price of a stock ("last price") varies continuously during the course of a stock market session. Once the market has generated a new price, the old one automatically becomes obsolete. So in the case where prices are generated with a high frequency, one could decide not to communicate each and every variation. It is obviously indispensable to maintain the consistency of the data presented on a page, despite the filtering. For example, if on the same page there is the price of a covered warrant that usually gets updated once every hour, together with the price of a stock listed on the NASDAQ, which generates 30 updates a second, it is obviously unthinkable to lose the one update of the first in order to communicate more updates of the second.
- Measurements by a probe. A probe of any sort (e.g. a physical probe to measure temperature or a software probe for network management) produces a whole series of measurements which are samples of the quantity that the probe is designed to monitor. These samples can be subsequently re-sampled to reduce their frequency even further.
Lightstreamer manages the filterable data feeds by means of the MERGE subscription mode.
The source provides updates to a persisting state of an item. The absence of a field in an itemEvent is interpreted as “unchanged data”. Any “holes” must be filled by copying the latest available value for each field. Not all the itemEvents from the Data Adapter need to be sent to the client, i.e. subsequent itemEvents can be
conflated (merged) into a single update.
Let’s imagine that each update is a small sheet divided into cells, with the current field values written in each cell. The fields are price, bid, ask, time. The sheets fall down one after another from the data source and stack on the ground.
Each update carries only the fields whose value has changed. The other cells are transparent. The first sheet carries all the field values. The second one carries only time and ask. the third one carries only price. When the second sheet stacks upon the first one, a new sheet is created, that carries the price and bid from the first sheet and the time and ask from the second. This means that if for any reason (bandwidth constraint, frequency allocation, adaptive streaming, etc.) two updates cannot be sent to the client, only the resulting merged update (“sheet”) will be sent.
When the third sheet stacks upon the first two, a resulting sheet carrying the price from the third sheet, the time and ask from the second sheet the bid from the first sheet is created.In this case the server has the possibility of sending one update only to the Client instead of three. But as mentioned before, such filtering only occurs when actually necessary, cause of several combined variables.
DISTINCT Mode
There are data sources that should not be filtered. In these cases the feed does not produce data that replaces the previous data, instead it produces data that should be displayed alongside the previous data.
Example of a
non-filterable data feed:
- News headlines. Press agencies generate fresh news items throughout the day. Typically, a real-time news visualization system shows a series of headlines in chronological order (from the latest to the oldest) which runs on receipt of each new headline. So when a fresh item of news is received, it does not eliminate the previous one; it gets listed alongside it.
Lightstreamer manages the non-filterable data feeds by means of the DISTINCT subscription mode.
In this case, the itemEvents coming from the Data Adapter are sent to the client unchanged. Lightstreamer tries to deliver each single update. Any bursts in the generation of new updates get absorbed by spreading out their dispatch to clients over time, if required, within certain limits (buffers, frequency, bandwidth, etc.).
For further information see chapter 3 of "Lightstreamer General Concepts.pdf".
