client refresh data content incomplete

paulwiersema

Overview: using js client in command mode with snapshot=true.

Our clients don't have insight into all the items that could be published to them, so we are using a global item to subscribe to ("FLIGHTS") with a unique "key" assigned to each flight so that the client can properly associate data rows (answer the question "do I have this row already?"). The data adapter has a REST endpoint to gather all the flights which meet the publication criteria and return them. Data updates are pushed using "ADD" because the producer doesn't really know if a particular key has met the conditions for publication any time prior, it just knows it needs to be published now.

The phenomenon we are observing is that although initial state is reflected properly at startup and new (and subsequent) events process through the system as expected, if a client refreshes their browser, not all data is republished to that client.

So, a couple of questions come to mind:
1 - is there a more appropriate mode (merge?) that we should be using?
2 - are there any tools we can use to see what is in the LS kernel cache to identify where our processing logic is flawed?

Thanks for your time.
Paul

Giuseppe Corti

Hi Paul,

First of all sorry for the late reply.

I can confirm you that the COMMAND mode seems to me the best choice for a scenario like yours. In fact it was specifically designed to manage dynamic lists or tables.
Unfortunately, however, there is no a tool that allows you to inspect the state of an Item maintained by the Lightstreamer kernel.
But, if you can reproduce the issue in a relatively isolated test environment, with a few client sessions, you could leverage these log settings

<logger name="LightstreamerLogger.pump" level="DEBUG"/>
<logger name="LightstreamerLogger.subscriptions" level="DEBUG"/>
<logger name="LightstreamerLogger.subscriptions.upd" level="DEBUG"/>

to try to understand what goes wrong, matching information about the updates received by the Data Adapter and those actually sent to each clients.
Indeed, if some rows are missing when the client asks for the snapshot the possible causes could be or your adapter has sent a DELETE command or a duplicate key case. But we can verify exactly with a log after having modified the settings as indicated above.

Regards,
Giuseppe

paulwiersema

Giuseppe,

Thanks for the reply. My current working theory is that the new data is not being initially tagged to the kernel as a snapshot from the adapter. I am working on a fix to err on the side of marking an update as a snapshot, as opposed to missing the snapshot indication altogether. If this doesn't solve the issue, I'll try to increase logging, but we have a shared DEV environment, so that might prove somewhat tricky.

Regards,
Paul

paulwiersema

Followup. That appears to have been the issue. If the kernel doesn't have a snapshot for an item, then the response to the request for a snapshot from the client won't include that item. That seems to be a reasonable approach and I should have closed that loop-hole earlier. Thanks again for the response.

Regards,
Paul

Giuseppe Corti

Hi Paul,

I am happy that you think you've found the problem.

But please let me stress out, that the Lightstreamer server when an Item is no longer subscribed by any client (after a configurable timeout) discard it. This means that the snapshot maintained until then is lost.
When subsequently a new client re-subscribes that Item the Lightstreamer Kernel asks to the Data Adapter to provide a snapshot (if any) to pass to the client.
In this phase the snapshot flag becomes important because, as long as updates marked as snapshot arrive, the Lightstreamer kernel continues to build its internal state for the Item, but if it receives an EOS (End Of Snapshot) or an update with the flag turned off, it consider the snapshot completed or not available and informs consequently the client.

Regards,
Giuseppe

paulwiersema

Giuseppe,

Thanks for the heads-up. Unfortunately, even after the recent changes we are still seeing missing rows after a client browser refresh.

In our use case, the clients need all updates, so they subscribe to a generic item called "FLIGHTS", so there is only 1 subscription per client. The only time there will be no subscriptions to "FLIGHTS" is if all clients disconnect, which would be fine. The initial load URL returns a list of flight objects, with a unique key and snapshot=true on each flight object. After browser refresh, we never see any of these flights missing.

There are instances where a new flight needs to be pushed to the clients. The system that pushes this data does not have insight into whether or not the flight was already pushed or not, so it sets the snapshot=true to err on the side of caution (even though there may be some data missing from that row that needs to be filled in later). These "new" flights are the ones that are sometimes missing after a browser refresh.

There are other systems that send other partial updates, but these systems "know" that and so the snapshot is not set in those updates.

The question that sticks out in my mind is "Does the server see the SNAPSHOT as related to the subscribed item ("FLIGHTS") or the individual flight row (unique by key)? If it is by subscribed item, how do we ensure that any new flights are added to the "group"? if it is by key, do we need to ensure that only the first data push is marked as a snapshot? That would require a bit of re-architecture, but it could be done, if needed.

Thanks again for your wisdom and direction,
Paul

Dario Crivelli

Hello Paul,

The Data Adapter is not supposed to set the "isSnapshot" flag to notify that an update introduces a new key in a COMMAND mode item, which should appear in the snapshot of future subscriptions.
The meaning of the "isSnapshot" flag is that the data being passed is not a real-time update, but it occurred earlier, hence it is passed only as part of the current item state (i.e. the item snapshot).
As a consequence, the Data Adapter should set the "isSnapshot" flag only at the beginning, just after receiving the "subscribe" call, to send the accumulated data. After sending them, it should declare the end-of-snapshot and only send real-time updates, without the flag.
The same information will be forwarded by Lightstreamer to the client.

Then, if it happens that

a real-time update introduces a new key in a COMMAND mode item;
the item remains subscribed to the Data Adapter;
a new client subsequently subscribes to the item;

Lightstreamer will take care of ensuring that the new client will receive the key as part of the snapshot (provided that it requests the snapshot).
So, if you are in this case and don't see the key, this is unexpected; but only the detailed log, as requested by Giuseppe, can shed more light.

paulwiersema

Dario,

Thanks for the insight. I will revert that snapshot code for new flights and increase logging to see if we can figure out what's happening. Are there any requirements for setting the keys? For example, do the keys need to include the item name (FLIGHTS.key1) ? Just wanting to make sure we are doing what's expected.

Thanks,
Paul

Dario Crivelli

No, the key names are free; they are String objects and their equality is String equality. Only null keys or wrong commands are refused.
You should obey the constraint about using the ADD command the first time you use a key (or after a DELETE) and UPDATE subsequently, but this is not mandatory and the Server just warns if it finds that your command doesn't match its observations, but doesn't discard any event because of this.

paulwiersema

Guys,

I just realized that I am not able to reproduce this issue locally using version 7.0.1, but our enterprise is still on version 6.1. From the release notes, this may have been fixed in 7.0:

"Fixed a bug affecting update dispatching, which could have been triggered by the combination of a very fast unsubscription-resubscription sequence and significant delays in internal update processing, possibly due to high load. The bug could have caused one or more updates for the previous subscription to be used in place of an equal number of updates for the new subscription."

Can someone please confirm?

Thanks,
Paul

Dario Crivelli

Hi Paul,
That case happened very rarely, but it is possible.
In that case, a warning message saying "Unexpected duplicate event" was also observed.
Can you find it in the logs?

paulwiersema

Dario,

I don't recall seeing that message in the logs. The symptom appears to be that the real-time updates are occurring and being forwarded to the the clients, but those updates are not being added to the snapshot, so when the next snapshot is requested, they are missing. I am not able to duplicate this behavior in the current version, so we are pushing to get the latest version installed. Unfortunately, corporate red tape moves slowly, especially when creating new vendor relationships.

Thanks,
Paul