DataProvider slow subscribe operation

trader-123

Reading the JavaDocs for the DataProvider interface (subscribe / unsubscribe methods), it says:

"The method should perform as fast as possible. If the implementation is slow because of complex subscription activation operations, it might delay a subsequent unsubscription and resubscription of the same item. In that case, configuring a dedicated "DATA" thread pool for this Data Adapter is recommended, in order not to block operations for different Data Adapters."

Is there any example of how to setup a dedicated "DATA" thread pool? Also, even with a thread pool, you'd delay the first subscription to the item, right? Next subscriptions to the same item would be taken care of by the LS server, so no delay.

Our case is exactly that: in the subscribe method we need to perform a HTTP call. I guess that's considered "not fast" in terms of what your documentation says?

Another option would be to use a queue, and the subscribe / un-subscribe methods only push to that queue (the queue would be polled from another thread). The problem with that would be, for example: what if the HTTP call fails, or the item is not valid... Whatever reason it is, we wouldn't be able to throw a SubscriptionException.

If there any other possibility?

Thanks in advance!

Dario Crivelli

We probably don't show the DATA pool configuration in any example, but the way is easy.
Just take the template of the adapters.xml configuration file in the in-process Adapter SDK and look for "data_adapter_pool".
Here is the link for Moderato.

If subscribe() is slow, the delay will affect the originating client subscription and any other client subscription to the same item happening at the same time.
Afterwards, as long as there is at least one client subscription active, other client subscriptions to the same item are handled by Lightstreamer immediately, as you pointed out.
If all client subscriptions are closed, unsubscribe() is invoked and the next client subscription to the same item will cause subscribe() to be invoked again.
So, if subscribe() is very slow and the clients give up, it is possible that the unsubscribe() and a subsequent subscribe() are required before its termination and in that case they will be delayed. This is what the Javadoc meant.

If you perform subscribe() and unsubscribe() asynchronously, then, as you correctly notice, you cannot signal a subscription failure.
But this is not very important, as the only difference is that, if subscribe() throws a SubscriptionException, the Server will log the exception and will not invoke unsubscribe().
However, as subscribe() and unsubscribe() operations are enqueued by your Adapter, you must be careful in associating any update notification to the corresponding subscribe() invocation.
To this purpose, it is advised that you always use smartUpdate() rather than update().

I don't see alternatives, but the latter technique should be feasible.

trader-123

Dario, I'm sorry for the late reply.

I think that's clear now! I wasn't too far from the solutions. Would you suggest using a queue instead of the data_adapter_pool solution?

Is there a recommended way for our use-case, or is this new territory for LightStreamer?

Cheers!

Dario Crivelli

A subscribe operation performing a HTTP call is a case that we consider, and this is why we provide the solution of the DATA pool to tackle the case in a simple way.
Enabling the DATA pool is not very different from working asynchronously through a custom queue.

From the client point of view there is no difference, because the client never waits for the completion of the subscription operation.
Slow subscribes obviously may delay the arrival of data, but this can't be avoided in both ways.
The main problem we see is when a slow subscribe causes the postponement of the execution of other subscribes, either for the same item or for other items.

On this respect, a custom queue just allows for more control.
For instance, the size of the DATA pool is preconfigured, whereas with a custom queue you can choose at any time how many subscription operations to run concurrently.
Also, with a custom queue you still have the problem of handling possible sequences like [ subscribe() - unsubscribe() - subscribe() ] for the same item;
but perhaps you may have devised a way to run such operations in parallel.

In general, we don't suggest the use of a custom queue unless the use of the DATA pool proves to be too limited.

trader-123

Thanks for that great explanation, we'll measure and report back, in case it helps someone else!