lightstreamer suddenly closed frequently

Hi,

I checked the log and I can confirm that indeed at 14:36:44 the Lightstreamer server experinced a fatal error in the comunication with one of the Adapters:

26-Mar-18 14:36:42,343|ERROR|rs.remote.data.RemoteDataProvider|#3 Request Sender |Exception caught while unsubscribing from item 'tickbytick_stock125': Connection closed while waiting for an answer to request d740000016260f71118
26-Mar-18 14:36:42,343|INFO |mote.request_reply.NotifyReceiver|#3 Reply Receiver |Reply receiver '#3' stopped
26-Mar-18 14:36:44,265|FATAL|rs.remote.data.RemoteDataProvider|#3 Request Sender |Asynchronous exception caught: Unable to write on the request stream
com.lightstreamer.adapters.remote.RemotingException: Unable to write on the request stream
at com.lightstreamer.adapters.remote.request_reply.RequestSender.run(RequestSender.java:136) [ls-proxy-adapters.jar:na]
26-Mar-18 14:36:44,265|ERROR|LightstreamerLogger.subscriptions|#3 Request Sender |Failure invoked by Data Adapter:
com.lightstreamer.adapters.remote.RemotingException: Unable to write on the request stream
at com.lightstreamer.adapters.remote.request_reply.RequestSender.run(RequestSender.java:136) [ls-proxy-adapters.jar:na]
26-Mar-18 14:36:44,265|FATAL|LightstreamerLogger.init |#3 Request Sender |Failure in a Data Adapter.

And since you choose non-robust proxy adapters as a consequence the Lightstreamer server shut down.
To figure out what was the problem is quite difficult but, given the first error message, a hypothesis is that a subscription request (for the Item 'tickbytick_stock125') has caused a crash in the Adapter.

Regards,
Giuseppe

There are two flavors of Proxy Data Adapter: NetworkedDataProviderand RobustNetworkedDataProvider .

In your case it seems that you are using the first one that explicitly provides not to start listening on the server TCP port until both the request_reply_port and notify_port are connected and shut down Lightstreamer server in case of failure of one of the sockets.
Furthermore, the RobustNetworkedDataProvider contains some recovery capabilities and avoid to terminate the Lightstreamer Server process. Full details on the recovery behavior are available as inline comments within the example adapters.xml file available here or in the “Lightstreamer/DOCS-SDKs/adapter_remoting_infrastructure/doc/adapter_robust_conf_template” folder.

Please note that in case of Ligthstreamer server version 5.x, the configuration is slightly different (https://www.lightstreamer.com/repo/distros/Lightstreamer_Allegro-Presto-Vivace_5_1_2_Colosseo_20140722.zip%23/Lightstreamer/DOCS-SDKs/sdk_adapter_remoting_infrastructure/conf/sockets(robust)/adapters.xml).

So, you should consider to switch to RobustNetworkedDataProvider.

Regards,
Giuseppe

Hi,

Yes, and for Metadata you should use:
com.lightstreamer.adapters.remote.metadata.RobustNetworkedMetadataProvider

Please find here (https://www.lightstreamer.com/repo/distros/Lightstreamer_Allegro-Presto-Vivace_5_1_2_Colosseo_20140722.zip%23/Lightstreamer/DOCS-SDKs/sdk_adapter_remoting_infrastructure/conf/sockets(robust)/adapters.xml) an adapters.xml example for Robust proxy adapters in case of Lightstreamer 5.x.

As already pointed out, the RobustNetworkedDataProvider and RobustNetworkedMetadataProvider contain some recovery capabilities and avoid to terminate the Lightstreamer Server process. Full details on the recovery behavior are available as inline comments within the example adapters.xml file available in the “Lightstreamer/DOCS-SDKs/adapter_remoting_infrastructure/doc/adapter_robust_conf_template” folder.

Regards,
Giuseppe

The log shows that the Server suddenly blocks for about one minute.
See for instance your latest log between 02-Apr-18 14:34:46 and 02-Apr-18 14:35:49.
It is really a block, because the 2-second periodic LightstreamerMonitorText lines are also missing.

The most common explanation is a major garbage collection. However, the free heap is always very large; this is strange.
Please ensure that you have set the recommended GC options in the command line in LS.sh or LS.bat, that is:
-XX:MaxGCPauseMillis=1000
and, just in case, also
-XX:+UseG1GC

About the memory, you should ensure that the amount of memory allocated to the JVM is not larger than the available RAM.
The JVM should be smart enough not to use more memory than available;
but have you set a -Xmx setting explicitly in the command line in LS.sh or LS.bat?
Or does the JVM share the avaiable memory with other memory consuming processes? For instance with some of the Remote Adapters?

By the way, this issue seems not related with Remote Adapter problems.
However, when the Clients and the Server try to overcome the block, they put a lot of pressure on the Remote Data Adapters, which seem not able to cope with the amount of subscription requests and begins to answer too late.
You may evaluate increasing the
<param name="timeout">
element in the <metadata_provider> and <data_provider> blocks in adapters.xml.

Yes, let's try this.
Obviously, I assume that the 6G specified is significantly less than the available RAM. According to your logs, your memory requirements are lighter.
Then we will see if the disconnections of the Remote Adapters also disappear.

Please check in the Server log if you no longer see long pauses in which even the log from LightstreamerMonitorText is missing.
In this case, the configuration change has succeeded, at least, in simplifying the scenario.

Turning to the disconnections from the Remote Adapter, as far as Lightstreamer is concerned, they cannot be a direct consequence of the load.
I suppose that the disconnections are caused by a crash of the whole process running the Remote Adapter (which may be an indirect consequence of the load). Can you confirm that there is a crash?