Difference between two adapters

ManKeer

I have to java adapters running on the same Lightstreamer server, the first one working with "MERGE" mode, while the other working with "COMMAND" mode, the first one pushing data directly, while the other converts the data to JSON then published to the subscribers.

The first one has two main items for subscriptions, each one has it's own DB thread and listener, the other has 7 subscriptions each one has it's own listener.

now both of them reading from the same DB tables, but unfortunately, the first one is faster than the second, client for the first one receives about 6 updates in 2-3 seconds, while the client of the second receives about 2 updates in 2-3 seconds.

when I check the logs, the data retrieved successfully without difference, the adapter logs show that the data is pushed....

Why do we have such a behavior?????

Dario Crivelli

The only scenario that I can recall is if the json data in the COMMAND item is much heavier than the field-based data in the MERGE item and the client is very slow at handling the updates.

Apart from that, we should follow the updates trip to see what happens.
We can start from the Server side with the following settings in lightstreamer_log_conf.xml
- LightstreamerLogger.subscriptions at DEBUG
- LightstreamerLogger.subscriptions.upd at DEBUG
- LightstreamerLogger.pump at DEBUG

Note that the log is verbose, but we assume that you can reproduce the issue with only the two mentioned clients connected.
On the other hand, if you can replicate the issue only in a production scenario with many clients connected, please clarify that first.

ManKeer

From client-side Is there any configurations that limit the received number of updates?

I will re-configure the logging based on what you have mentioned, and I will show the logs.

Thanks

Dario Crivelli

Yes, the client application can set a maximum frequency on the updates received on an item basis.
Also the Metadata Adapter can set such a maximum frequency on an item basis, which may further restrict the limit.
However, none of these limits is set by default.

ManKeer

Based on what I have seen in the logs, L.subscriptions.updrefers to an update from the adapter sent to LS engine, while L.pump refers to the pumped message by LS engine to the clients.

if the above is true then, I have an issue in the following log records:

22-Aug-23 13:19:19,137|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"11\":\"3.978\",\"12\":\"5817\",\"13\":\"1014770\",\"5\":\"2023-08-22 13:19:18.000\",\"6\":\"2023-08-22 13:19:19.086\",\"7\":\"94.1\",\"9\":\"440\",\"10\":\"3.6\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,148|DEBUG|L.subscriptions.upd|LMARKETS |INCOMING DATA for LMARKETS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"TBNI\":{\"11\":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7\":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\"}}","removed":null}, key=LMARKETS, command=UPDATE}
22-Aug-23 13:19:19,148|DEBUG|L.pump |PUMP POOLED THREAD 8 |Pumping event in session S8c12f8acdc0d0c2eMfccT1556739: U,42,1,||{"added":null,"changed":"{\"TBNI\":{\"11\":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7\":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\"}}","removed":null}

22-Aug-23 13:19:19,152|DEBUG|L.pump |PUMP POOLED THREAD 5 |Pumping event in session Se80d7908765fb794MfccT1557247: U,13,1,||{"added":null,"changed":"{\"TBNI\":{\"11\":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7\":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\"}}","removed":null}
U,12,4,||{"added":null,"changed":"{\"2010\":{\"6\":\"2023-08-22 13:19:18.951\"}}","removed":null}

22-Aug-23 13:19:19,294|DEBUG|L.subscriptions.upd|LDEPTH |INCOMING DATA for LDEPTH.1120,MBO on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"1120,MBO\":{\"0\":\"1120\",\"1\":\"MBO\",\"2\":\"TASI\",\"3\":\"2023-08-22 13:19:19.226\",\"4\":\"1100\\u003d20,1101\\u003d2427,1102\\u003d72.7,1103\\u003d140,1104\\u003d72.7,1105\\u003d45,1106\\u003d72.7,1107\\u003d3800,1108\\u003d72.8,1109\\u003d345,1110\\u003d72.8,1111\\u003d900,1112\\u003d72.8,1113\\u003d10,1114\\u003d72.8,1115\\u003d4000,1116\\u003d72.8,1117\\u003d7001,1118\\u003d72.8,1119\\u003d800,1120\\u003d72.8,1121\\u003d280,1122\\u003d72.8,1123\\u003d700,1124\\u003d72.8,1125\\u003d92550,1126\\u003d72.8,1127\\u003d17,1128\\u003d72.8,1129\\u003d1,1130\\u003d72.8,1131\\u003d197,1132\\u003d72.8,1133\\u003d376,1134\\u003d72.8,1135\\u003d300,1136\\u003d72.8,1137\\u003d227,1138\\u003d72.8,1139\\u003d1,1140\\u003d72.8,1141\\u003d20,1142\\u003d34,1143\\u003d72.6,1144\\u003d2000,1145\\u003d72.6,1146\\u003d1,1147\\u003d72.6,1148\\u003d2,1149\\u003d72.6,1150\\u003d495,1151\\u003d72.6,1152\\u003d1377,1153\\u003d72.6,1154\\u003d100,1155\\u003d72.6,1156\\u003d100,1157\\u003d72.6,1158\\u003d500,1159\\u003d72.6,1160\\u003d25,1161\\u003d72.6,1162\\u003d45,1163\\u003d72.6,1164\\u003d4,1165\\u003d72.6,1166\\u003d50,1167\\u003d72.6,1168\\u003d23,1169\\u003d72.6,1170\\u003d50,1171\\u003d72.6,1172\\u003d10,1173\\u003d72.6,1174\\u003d100,1175\\u003d72.6,1176\\u003d66,1177\\u003d72.6,1178\\u003d10,1179\\u003d72.6,1180\\u003d7001,1181\\u003d72.6\"}}","removed":null}, key=LDEPTH, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"4142\":{\"16\":\"724\",\"6\":\"2023-08-22 13:19:19.220\"},\"1120\":{\"16\":\"2612\",\"6\":\"2023-08-22 13:19:19.222\"},\"1010\":{\"16\":\"7278\",\"6\":\"2023-08-22 13:19:19.226\"},\"2010\":{\"14\":\"4940\",\"6\":\"2023-08-22 13:19:19.214\"},\"7040\":{\"16\":\"2497\",\"17\":\"94.2\",\"6\":\"2023-08-22 13:19:19.223\"},\"4291\":{\"16\":\"37\",\"6\":\"2023-08-22 13:19:19.229\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.1120 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"1120\":{\"16\":\"2612\",\"6\":\"2023-08-22 13:19:19.222\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.2010 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"2010\":{\"14\":\"4940\",\"6\":\"2023-08-22 13:19:19.214\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"16\":\"2497\",\"17\":\"94.2\",\"6\":\"2023-08-22 13:19:19.223\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,389|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"12\":\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,389|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"12\":\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 8 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,3,1,||{"added":null,"changed":"{\"TBNI\":{\"11\":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7\":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\"}}","removed":null}

22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S6f06ad1777945360MfccT1526937: U,25,1,||{"added":null,"changed":"{\"TBNI\":{\"11\":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7\":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\"}}","removed":null}

22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 5 |Pumping event in session Se80d7908765fb794MfccT1557247: U,12,2,||{"added":null,"changed":"{\"1120\":{\"16\":\"2612\",\"6\":\"2023-08-22 13:19:19.222\"}}","removed":null}

22-Aug-23 13:19:19,400|DEBUG|L.subscriptions.upd|LMARKETS |INCOMING DATA for LMARKETS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"TASI\":{\"11\":\"116942017\",\"3\":\"2023-08-22 13:19:19.321\",\"5\":\"11403.05\",\"6\":\"-5.4\",\"7\":\"-.047\",\"8\":\"217779\",\"10\":\"2864023605.05\"}}","removed":null}, key=LMARKETS, command=UPDATE}
22-Aug-23 13:19:19,428|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,1,1,||{"added":null,"changed":"{\"7040\":{\"12\":\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}

22-Aug-23 13:19:19,428|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,4,1,||{"added":null,"changed":"{\"1120,MBO\":{\"0\":\"1120\",\"1\":\"MBO\",\"2\":\"TASI\",\"3\":\"2023-08-22 13:19:19.226\",\"4\":\"1100\\u003d20,1101\\u003d2427,1102\\u003d72.7,1103\\u003d140,1104\\u003d72.7,1105\\u003d45,1106\\u003d72.7,1107\\u003d3800,1108\\u003d72.8,1109\\u003d345,1110\\u003d72.8,1111\\u003d900,1112\\u003d72.8,1113\\u003d10,1114\\u003d72.8,1115\\u003d4000,1116\\u003d72.8,1117\\u003d7001,1118\\u003d72.8,1119\\u003d800,1120\\u003d72.8,1121\\u003d280,1122\\u003d72.8,1123\\u003d700,1124\\u003d72.8,1125\\u003d92550,1126\\u003d72.8,1127\\u003d17,1128\\u003d72.8,1129\\u003d1,1130\\u003d72.8,1131\\u003d197,1132\\u003d72.8,1133\\u003d376,1134\\u003d72.8,1135\\u003d300,1136\\u003d72.8,1137\\u003d227,1138\\u003d72.8,1139\\u003d1,1140\\u003d72.8,1141\\u003d20,1142\\u003d34,1143\\u003d72.6,1144\\u003d2000,1145\\u003d72.6,1146\\u003d1,1147\\u003d72.6,1148\\u003d2,1149\\u003d72.6,1150\\u003d495,1151\\u003d72.6,1152\\u003d1377,1153\\u003d72.6,1154\\u003d100,1155\\u003d72.6,1156\\u003d100,1157\\u003d72.6,1158\\u003d500,1159\\u003d72.6,1160\\u003d25,1161\\u003d72.6,1162\\u003d45,1163\\u003d72.6,1164\\u003d4,1165\\u003d72.6,1166\\u003d50,1167\\u003d72.6,1168\\u003d23,1169\\u003d72.6,1170\\u003d50,1171\\u003d72.6,1172\\u003d10,1173\\u003d72.6,1174\\u003d100,1175\\u003d72.6,1176\\u003d66,1177\\u003d72.6,1178\\u003d10,1179\\u003d72.6,1180\\u003d7001,1181\\u003d72.6\"}}","removed":null}

22-Aug-23 13:19:19,450|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session Sc4736f28d89e4ee3MfccT1547256: U,30,1,||{"added":null,"changed":"{\"TASI\":{\"11\":\"116942017\",\"3\":\"2023-08-22 13:19:19.321\",\"5\":\"11403.05\",\"6\":\"-5.4\",\"7\":\"-.047\",\"8\":\"217779\",\"10\":\"2864023605.05\"}}","removed":null}
U,29,1,||{"added":null,"changed":"{\"7040\":{\"12\":\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}

The highlighted green message has been lost, and my java client didn't receive it at all, from the logs there's no pump to this message.

is that true????

ManKeer

In addition to the above, I can confirm that the adapter's code is running perfectly, and the data retrieved without any problems, however the clients for the same adapter behaves differently, java script client not receiving most of the messages, while java client is receiving around 95% of the messages.

in my last reply, I have mentioned that one message has not sent from the LS engine side [this explains why java client not receiving the messages], but why the java script client not receiving messages which are received by java client.

both clients have been configured to receive MAXFREQUENCYUPDATE = UNLIMITED.

ManKeer

How can I attach the lightstreamer.log file so you can check whatever you need.

Dario Crivelli

The log snippet includes 5 sessions, whereas I understood that two sessions were enough to exploit the issue.
This complicates the analysis. Can you please produce a log with only two sessions?
Then, assuming that it will be short, you can send it to support@lightstreamer.com
Please ensure that the log includes the Server startup and, in particular, the subscription requests by the two clients.

Note that, unless the client explicitly requests all updates, updates not sent are not lost, but just filtered out in order to always send the latest state.
This is possible also when specifying "unlimited" frequency. Only the "unfiltered" setting forces all updates.
Obviously, such filtering is expected only when there is some bottleneck or some race condition.
With these conditions, it is possible that two different clients receive a different number of updates.
Apparently, these conditions don't apply to your case; but perhaps the full log can help to clarify.

ManKeer

Hi Dario,

By two sessions, you mean that only two subscriptions [not two clients] are connected to the LS server, Am I right? please confirm...

if the above is correct, I will reproduce the logs again.

BTW, what will happen if I configuered both clients for "unfiltered" subscriptions, how this will affect the performance?

Dario Crivelli

ManKeer
BTW, what will happen if I configured both clients for "unfiltered" subscriptions, how this will affect the performance?

Preventing filtering can have an effect when the overall update flow to a client is so huge (or the client/network so slow) that it is impossible that all available updates are processed by the client.
In this case, some updates have to be filtered out and this requires that some subscription with a huge flow is not "unfiltered".
But other items with a low flow can still be subscribed "unfiltered".
As long as the update flow is manageable, there is no significant performance difference between "unlimited" and "unfiltered".
Only in rare cases of race conditions (like two subsequent updates in short sequence), "unfiltered" prevents possible filtering-out of some update.

Another case in which "unfiltered" makes the difference is when there are licensing restrictions that put an a-priori limit on the update frequency of an item.
But, in this case, this only works if the average update frequency of the item is lower than the license limit.

Note that, for MERGE mode, you can reduce filtering in almost the same way by enlarging the buffer, which by default is 1, through setRequestedBufferSize.

That said, I'm still not sure what of the above applies to your scenario.

Dario Crivelli

Sorry; if my reference to "two sessions" raises doubts, then please disregard it and send what you have available.
What's important is that the subscriptions are included in the log, so that we can better compare subscriptions and updates received.
On the other question I will answer tomorrow.

ManKeer

Hi Dario,

I have attached the logs that I took the snippet from.

you will receive an email from m.ankeer@gmail.com.

you can check all the sessions.

Regards

Dario Crivelli

The log shows that there is an underlying frequency limit of 3 updates per second due to the license in use.
This accounts for the suppression of some update when two or more are produced in short interval.
As discussed above, you can request the item as unfiltered, but you must be sure that the average frequency of updates of this item is steadily less than 3 per second.
Otherwise you can use setRequestedBufferSize and determine a size that is a trade-off between possible lost (i.e filtered out) updates and possible update delays.

This does not explain why an item in COMMAND mode yields fewer updates than an equivalent item in MERGE mode.
We can investigate this case, but please provide some precise references.
If you saw this happening in the run of the provided log, please specify two item names for the COMMAND and MERGE subscriptions with different behaviors.

ManKeer

Thanks Dario,

But I need the answers for the following questions:

1 - Is there any way to increase the (max frequency update)/item/second to more than 3?
2 - What is the max message size that can be sent?
3 - for the purpose of BW, which is better: sending small message to equivalent item [ # items = 400]... OR.... combining all messages in one big message and send it to common item [common item for 400 items].

4 - for BW calculation purpose: each client has one connection (one session), it has subscribed for 100 items, each item has 3 updates per second as you mentioned, if each message size is 50 bytes, then BW required is:

100 * 3 * 50 * 8/1000 = 120 kbits/second, Am I right???

if we combine those 100 items in 1 item, the max message size is 5000 bytes, then new BW is:
1 * 3 * 5000 * 8/1000 = 120 kbits/second, same as above?

so if the above is true, I need to know what is the effect of large message size?

Thanks for your kind help and usual support.

Dario Crivelli

In a general scenario, separate items perform better, because it may happen that only part of the 100 items get an update, while others don't.
In this case, only the updates for items that really change are sent.
On the other hand, for a single big item, a single combined update is sent every time any of the 100 values included is different.
You could reduce this problem by using fields, so that you can use a single item with 100 different fields.
In this case, in the combined update, the values that don't change are not included.
But the set of fields is fixed, so this assumes that the needed set doesn't change in time.

So, only if you confirm that all 100 items change values every time, is your count correct, but for some encoding overhead to be added.
With different items the overhead is quite small (about a dozen bytes per update, hence to be added to the 50 figure).
With a single item, the Lightstreamer overhead is small, but the overhead needed to pack the 100 values in a single one has also to be considered.
More overhead is due the generation of websocket messages and TCP packets.
With many updates, Lightstreamer can pack multiple updates in a single message, but it tries to forge small messages, hence with 100 items it may use more messages and add some overhead, whereas with one big item it is forced to use a single message.
However, small messages give some benefits, for instance when the communication channel is narrow.

To resume, assuming that the bandwidth is the same (i.e. all values change every time), single large message should not have a negative impact.
The differences are at implementation level (like memory fragmentation) and should not be significant in normal situations.
In fact, Lightstreamer doesn't enforce a maximum message size.
Obviously, really big sizes may incur in resource issues, but a safety size is difficult to estimate.

About update frequency, you can experiment with unlimited frequency by configuring a DEMO license
(you should set DEMO in <license_type> and <contract_id> in lightstreamer_edition_conf.xml).
This limits the number of sessions to 20.
If you opt for upgrading your license, you can contact info@lightstreamer.com

ManKeer

Now, my data source push messages based on the lightstreamer rate.

however, my java client receives messages but the javascript client doesn't receive, and sometimes vice - versa.

How come this happened, based on my understanding, if the message received by one client, it should be received by the other. since both subscribed to the same item.

Would you please explain?

I have sent you the logs from business email m.ankeer@aljaziracapital

Dario Crivelli

For the general question posed, I just confirm that, when filtering is possible, it is done on a client-by-client basis.
This, in case of particular race conditions, accounts for different identical subscriptions receiving data in different ways.

Now I see that the case shown is not the typical race condition that can happen due to the underlying frequency limit.
In this specific case, the Server had enough time available to send the event, but it didn't.

My only explanation is that the connection could not work at full speed.
If previous writes on the connection could not be flushed in short time, then the subsequent write might have been blocked and the next event could have been kept long enough to be covered by the following one.
I see that the affected session was reopened a few minutes earlier, after a failed attempt to recover from a previous session, which must have had connectivity issues.
So, perhaps, the communication is disturbed in some way.

As an experiment, you could try enlarging the send buffer used by the Server, which, by default, is small, because, in normal usage scenarios, filtering is encouraged.
For instance, you can set
<sendbuf>5000</sendbuf>
and see if there is any improvement.

Unfortunately, with COMMAND mode, you cannot leverage the client's setRequestedBufferSize setting, as the buffer in COMMAND mode is always a single update per each key.

BTW, can you confirm that the Metadata Adapter doesn't introduce any bandwidth limit?
This is possible, if your adapter configuration was inspired by our demo examples.
You may send us adapters.xml for a check.

ManKeer

Dario,

I want to know if the string compression can affect the BW, i.e., if we compress the sent message, do we decrease the required BW, or it doesn't affect the BW anymore?

Dario Crivelli

If I understand correctly, you would like to try this in order to have the update packets flow faster and prevent race conditions that would lead to conflation.
BTW, did you try the experiment with the <sendbuf> setting?

I confirm that compressing the values may have an effect and, in fact, in the case of your data, according to your logs, it seems quite redundant.
But perhaps, instead of compressing it, you may try to remove the redundancy at the origin.
In fact, I see many \\u003d parts in your update values. This is a unicode representation of the = symbol.
It is not clear to me how they are quoted, but perhaps they can be quoted in a shorter way.

Lightstreamer itself uses several techinques to reduce the bandwidth needs.
If you leverage different fields to separate the various parts of your updates, then you can take advantage of the delta delivery feature, which, in many cases, reduces the size of the updates.
On the other hand, if you use the json representation and cannot leverage fields, you could take advantage of the support for sending deltas in the jsonpatch format.
For big fields of generic type, there is another kind of reduced delta, based on the "diff-match-patch" algorithm.
These jsonpatch and diff-match-patch diffs require Server version 7.3 or higher and also compliant client SDKs.
Note that they can be more or less effective depending on the form of the various updates.
For this reason, they are not applied by default, but by configuration. We can expand if needed.

Only if deltas cannot be used, there is the possibility to apply a zlib compression to the whole data in transit.
At the moment, this is not available for websockets, but only for http streaming, which, hence, should be enforced.