Problem re-subscribing

colmfield

Hi,

I have a web front-end using Lightstreamer Vivace, connecting to a .Net MetaDataAdapter and DataAdapter running as a Windows service. I am using the RobustNetworkedMetadataProvider.

Everything works well, except in one specific scenario that I hope someone can help me with:

When the Lightstreamer service (my .Net DataAdapters) fails and restarts (or is restarted manually), a whole load of the following errors appear in the LS log file:

09-Jul-10 13:55:32,435 |ERROR|rs.remote.data.RemoteDataProvider|MEPricesData Reply Receiver|Exception caught while subscribing to item 'item1'
com.lightstreamer.adapters.remote.RemotingException: Generic exception on remote side: Object reference not set to an instance of an object.
	at com.lightstreamer.adapters.remote.data.DataProviderProtocol.readSubscribe(DataProviderProtocol.java:83)
	at com.lightstreamer.adapters.remote.data.RemoteDataProvider$2.onReplyReceived(RemoteDataProvider.java:207)
	at com.lightstreamer.adapters.remote.request_reply.RequestSender.onNotifyReceived(RequestSender.java:296)
	at com.lightstreamer.adapters.remote.request_reply.NotifyReceiver.onNotifyReceived(NotifyReceiver.java:177)
	at com.lightstreamer.adapters.remote.request_reply.NotifyReceiver.run(NotifyReceiver.java:95)

The problem is none of the clients are notified of the errors, so I've no way of taking corrective action. My JS code includes the following:

engine.context.setDebugAlertsOnClientError(true);
engine.context.setRemoteAlertsOnClientError(true);

Also, if I stop the streaming, then restart it (via JS's changeStatus() method), then the first time I do it no updates are sent back to the client. If I do it a second time, the correct snapshot is sent.

On the client side I set

setSnapshotRequired(true)

I've also tried playing with close_notifications_recovery and events_recovery in the adapter.xml, but to no avail.

Any help much appreciated.

Dario Crivelli

The reported error messages represent errors originated in the Remote Data Adapter.
After failure and relaunch, the Remote Data Adapter is submitted Subscribe requests for all the items that were subscribed to the old instance.
It seems that the new instance doesn't handle the requests correctly and throws SubscriptionException upon them. The message of the SubscriptionException is then logged by the Proxy Adapter running in the Server.
By the way, the available recovery policies have no impact, as no updates are received.

I confirm that these kinds of errors are not notified to the clients, which only see no updates.
Such metainformation can only be managed at application level. If your Remote Data Adapter intercepted its own SubscriptionExceptions, it might communicate such events, for instance, through a dedicated item.
Anyway, in this case, the main issue seems to be that the Remote Data Adapter, once restarted, fails on the subscriptions.

Nevertheless, stopping and restarting streaming should enforce a new snapshot also the first time, as the snapshot is always kept by Lightstreamer Server.
Can you confirm that
- the items are requested in MERGE mode,
- the interruption is very short,
- the items have already yielded some values at the time of interruption?
Do you observe the updates in the page on a VisualTable or through the event callbacks?

colmfield

Hi Dario,

Thanks for your reply.

To answer you questions:

- the items are requested in MERGE mode,

Yes they are.

Do you observe the updates in the page on a VisualTable or through the event callbacks?

We use and OverwriteTable, so VisualTable.

- the interruption is very short,
- the items have already yielded some values at the time of interruption?

There seems to be something else going on here. I left my application running over the weekend, and it was scheduled to restart @ 2am. This happened successfully, but after the reconnect and once my data adapter started receiving data, I got a whole load of (1000s) the following errors in my log file:

...
2010-07-12 02:12:40,750 [4520] WARN  - Unexpected update for item item1a
2010-07-12 02:12:40,750 [4520] WARN  - Unexpected update for item item1b
2010-07-12 02:12:40,750 [4520] WARN  - Unexpected update for item item1c
2010-07-12 02:12:40,906 [4852] WARN  - Unexpected update for item item1d
...

From this post, I assume these are caused by data adapter's IItemEventListener's Update method not recognising the item it's supposed to be updating. So does that mean LS Server doesn't know that I'm subscribed to the items I'm sending data for? Could this be causing the RemotingExceptions too?

The odd thing here is that when I launched my web client this morning, sending the initial subscriptions to LS, I only received partial updates on the page. When I refreshed the page, it showed all the updates (i.e. the correct snapshot).

colmfield

Some more information re the two different errors...

The sequence of events seems to be:

Data\Metadata adapter service restarts
RobustNetworkedMetadataProvider fails, recovers, and requests a snapshot of data from adapter
My adapter takes ca.10sec to start up as it loads a lot of historic data, so in the meantime LS Server is trying to request data for items that my adapter knows nothing about yet, and throws "Generic Exception" errors

Then, any time after this happens, if a client subscribes to items via the webpage, the "Unexpected updates for item..." is logged and a full snapshot is not sent (in the LS log file I can see a new session being attached and controlled), but updates do come through.

If I then refresh the page a full snapshot is sent as expected.

Dario Crivelli

If your Remote Data Adapter takes some time to initialize and, during this time, it is unresponsive to subscription requests,
then it is probably better making it unavailable for that time.
You can either postpone the opening of the sockets or block the processing of the init method until the initialization has finished.

Note that the response to the Subscribe call is supposed to be fast and a timeout of exactly 10 seconds is applied by the Proxy Adapter. If the timeout expires, the Proxy discards the request and subsequent related updates are also discarded.
This can explain the "Unexpected update" errors.
The "Object reference not set" errors, on the other hand, seem to be internal to your Data Adapter implementation.

So, alternatively, you could enlarge the timeout (configured in adapters.xml) or, preferably, have Subscribeanswer immediately, regardless of the data availability. In fact, if you don't send the snapshot immediately after returning from Subscribe , this is not a problem from Lightstreamer point of view, provided that the client can accept that.

colmfield

I have found the issue.

You are correct Dario, in that the "Object reference not set" errors were internal to my Data Adapter implementation, and this was the root cause of everything else.

In my Subscribe method, I am calling a custom class's public method to do the sending of the updates. I am instantiating this class on a separate thread in my Init method.

However, in the Subscribe method this class was sometimes null, because in the ca.18ms it took to start the thread and initialise the class, the subscribe method had already been called.

Thanks for your help with this, these forums are excellent.

Colm