Updating adapters configuration without tearing down the service

jonasby

Is it possible to add/remove/edit a data adapter configuration for a running instance of Lightstreamer server? This would aid our continuous deployment effort.

Thanks

Giuseppe Corti

Hi jonasby,

I confirm you that currently the hotplugging of Data Adapters is not available in Lightstreamer Server. Indeed, it would be a important feature and we are considering whether add the feature in a future version of Lightstreamer.
Something similar, with some limitations, it is still possible even now in the case of using Remote Adapter and robust Proxy Adapters.

In practice, you could deploy into Lightstreamer server a pool of n robust Proxy Adapters, listening on a ports range. Afterwards you would have some freedom to add and remove Metadata and Data Adapters.
So you might turn off a running Data Adapter and reopen a new version without the client even notice it (in any case it is possible notify the disconnection of the source data to the client if you want).

But if you provide us more details about your needs, we can be more accurate in discussing the scenario.

Regards,
Giuseppe

subaoye

Giuseppe Corti
In practice, you could deploy into Lightstreamer server a pool of n robust Proxy Adapters, listening on a ports range. Afterwards you would have some freedom to add and remove Metadata and Data Adapters.
So you might turn off a running Data Adapter and reopen a new version without the client even notice it (in any case it is possible notify the disconnection of the source data to the client if you want).

But if you provide us more details about your needs, we can be more accurate in discussing the scenario.

Regards,
Giuseppe

Hi Giuseppe,
Regarding of adding and remove the Metadata and DataAdapters on runtime. I would like to get some advices about the deploying an Adapter Sets on runtime.
In my scenario, my code is currently running in-process with LS server. I have some cases that would like to achieve
1. I have a few version of client with is matched with the Adapter Sets version on LS server. I'm thinking of asking the client connect to a LOBBY AdapterSets then the LOBBY will response to client the AdapterSets name they should connect to. Is there any better solution to achieve this?
2. In case I would like to add a new AdapterSets version or the current AdapterSets need to update the code, is there any solution to add or update them on runtime in a same LS server. Can you provide some demo for it please.

Thanks

Giuseppe Corti

Hi subaoye,

1. Yes I confirm that such a mechanism is a good idea, the client first subscribes to an Item on which it receives custom information to manage the session, it makes sense.

2. With In-process adapters the hot-plugging is something not easy to achieve; as I said in the previous post, in case of remote adapters it is possible to adopt some strategy to allow dynamic updates (addition, removal, replacement) to the Adapter Sets, but in case of In-Process adapters these are not feasible.
You could adopt some custom implementation that separates the code of the adapters in a minimal wrapper layer and the core of the implementation in a pluggable component. But this is something maybe too complicated.
Please you might also consider upgrade strategies that involve rebooting, or replacing the Lightstreamer instance.
I am aware that this causes the client to interrupt the client session to create a new one, but this can be limited to a very short amount of time, and is entirely managed by our client libraries and almost transparent to the client application.
Especially if we are talking about a cluster of Lightstreamer instances, considering a blue-green strategy can lead to updating the Lightstreamer server version itself and/or the adapters version reliably and with downtime near to zero.

Please let us know if you need any further clarifications.

Regards,
Giuseppe

subaoye

Hi Giuseppe,

Clustering is one of strategy we will implement and we have ours LB using nginx. I notice the client could able to reconnect to server a several times until it timeout when I restart the server. It also lead me to another concern in production environment. There will be some connections is in semi-disconnection state and it will stay in the sever until next restarting or timeout. So I would like to check with you, in the server API is there any functions that I could get all the current connections or session object with I notice will have an SessionID assigned by LS. Currently I'm implementing a map which is hold the SessionID as key and my User as value, a scheduler will trigger when server start for checking last ping time from client (we implement on client side periodically send a ping message to server). However to handle it will have some trouble for removing and find, imaging that I have about 50Kccu. it would give out a bad performance.

Summary I would like to clarify with you some thing here:
1. what is timeout for the connection on LS server, I found in API client could indicate it but I would like to manage it on server side too
2. how LS detect a disconnection and how you handle semi-disconnection which I call ghost users
3. is there any way that I could get all the current connection also the other information like total dataAdapter, total subscribers on each dataAdapter,

Regards.

Giuseppe Corti

Hi subaoye,

Please find below my considerations.

1.
Indeed, in case of a restart of a server all your clients will need to connect back to the new servers; dealing with very high numbers of concurrent connections means that a massive number of clients will try to connect at the same time.
And this might end up with delays in handling requests with clients giving up after having opened a connection due to a timeout expired waiting for the completion of the create session protocol.
Please could you confirm that you are referring to this kind of scenario?
There are several causes that lead to this situation such as an insufficient sizing of resources or maybe it is due to external services, such as the user authentication process, that take longer than expected since it is temporarily overloaded too.
However there are few strategies to mitigate this problem, one as you have already mentioned is to relax the timeouts. The parameters that come into play are both client and server side; the server side parameter involved here is session_timeout_millis (in lightstreamer_conf.xml).
But this could not be the best option to take in consideration; please consider that Lightstreamer server has a backpressure mechanism specifically intended to address these problems.
With version 7.1, backpressure has been deeply revised to be able to cope with extreme cases of massive disconnections and reconnections; please find here more details and in the case get back to us for further clarifications: https://blog.lightstreamer.com/2020/01/the-new-features-in-lightstreamer-71.html#fe3

2.
Lightstreamer server has various mechanisms to determine a disconnection and therefore determine the closure of the associated client session.
For example, the detection of a closure of the tcp socket, or any errors in the reading or writing phase of the tcp socket; but it is true that in cases of "half-open" sockets they can escape and persist in the server for a long time.
To avoid these situations you can set the reverse heartbeat from the client, please refer to the API documentation for further details https://sdk.lightstreamer.com/ls-web-client/8.0.3/api/ConnectionOptions.html#setReverseHeartbeatInterval
This refers to the latest version of the web library, but there are similarly in almost all libraries (except in cases of very old ones).
Please also note that it is the client that have to enable reverse heartbeat for the session and the server will adapt.

3.
I'm not sure if that's actually what you need, but certain information is available from our JMX interface: https://www.lightstreamer.com/api/ls-jmx-sdk/latest/
For example you can double check the DataAdapterMBean (https://sdk.lightstreamer.com/ls-jmx-sdk/5.5.6/api/com/lightstreamer/jmx/DataAdapterMBean.html) where you can get the session list for Item and the AdapterSetMbean (https://sdk.lightstreamer.com/ls-jmx-sdk/5.5.6/api/com/lightstreamer/jmx/AdapterSetMBean.html) with the list of deployed Data Adapter and the list of active session.
Please note that the JMX interface is an optional feature so may not be entitled by your production license.

Regards,
Giuseppe

subaoye

Hi Giuseppe,

1. Yes I did faced with that scenario but not in the restarting server case, in normal runtime it also happen and it is from client side, the client device could have vary situation that lead to this haft-open connection on server side. I will go through the backpressure and JMX lib and feedback to you

2. about the JMX lib is there any demo that I could study, like how to plugin it in?

3. I'm not clear about your last statement:

Please note that the JMX interface is an optional feature so may not be entitled by your production license.

Do you mean I could not use JMX interface in the production or some of the production license is not allow it?

Regards.

Giuseppe Corti

Hi subaoye,

Please find here (https://github.com/Lightstreamer/Lightstreamer-example-RoomBall-adapter-java/blob/master/src/main/java/com/lightstreamer/adapters/RoomBall/PollsBandwidth.java) an example of in-process adapter which retrieve information about the network band used by each client session through the JMX interface.

About licensing I meant that if you have purchased, or will have to purchase, an Lightstreamer Enterprise edition license to use in production, you must make sure that among the optional features purchased there is also "JMX Management API".
Instead for the Community edition I confirm that the JMX interface is not available.

Regards,
Giuseppe

subaoye

Hi Giuseppe,

After review the LS JMX interface it seem not reach my expectation. Currently I organize an map to handle all the current sessionID match with a SessionUser created by my side. the SessionUser have contain info about my system. what I would like to achieve is attaching the SessionUser to LS Session. the JMX interface seem more about monitoring not for manipulating the Session data.

Thankyou for your alert about the license I will take note about it.

Update: I also have 1 more question about the setReverseHeartbeatInterval, what happen if I set the interval is 1ms? Will LS have mechanism to prevent ddos?

And also is there any affecting to server if all client keep send heartbeat

Regards.

Giuseppe Corti

Indeed the reverse heartbeats involve a minimal workload for the server, however in the case of many client sessions expected and short frequency intervals this must be taken into account when evaluating the overall load that can be sustained by the system.
The frequency of reverse heartbeats should be therefore determined at design time and embedded in the client application; without allowing the possibility of configuration in the client application, or in any case limiting the configuration to a range considered reliable.

Therefore reverse heartbeats are not particularly associated with a possible ddos attack that can be executed with any other type of client request.
Typically the defense mechanisms rely on standard defensive measures that are not directly related to Lightstreamer itself but to network intermediaries placed in front of the Lightstreamer servers. In practice throttling requests from clients with a frequency above the reasonable limit
As for the Lightstreamer configuration, the same backpressure we talked about in a few posts above can be useful to mitigate this type of situation with peak requests.

Regards,
Giuseppe