Memory Leak - LightstreamerClient.disconnect() does not stop threads [Java]

KlausGroenbaek

I just started using the LightStreaming Java client to stream prices for financial instruments, a very common use case. What is different from many other usages is that my application is not an end-user client, but middleware running on a Tomcat server, doing some calculations and then giving the data to the end-user in a different format.
The LightStreamer java client does not look like it was designed for middleware usage at all:confused:. Any connection based Client need some sort of lifecycle, the LightStreamerClient has connect() and disconnect(), but they do not implement proper resource (Thread) management.
In a J2Se/J2EE application it is extremely important that any threads a client library starts can also be stopped by the client library, otherwise you will create a memory leak when the application is undeployed.
Debugging the code I can see that all threads have been configured as daemon, this will work fine if the LightStreamClient has the same lifecycle as the JVM it lives inside, but you should NEVER have daemon threads if you client may be used in a J2SE/J2EE container like Tomcat/JBoss.
This is a very common problem for immature frameworks, but i had honestly expected better from software that has been around for this long.

Gianluca Finocchiaro

Hi Klaus,

We are aware that a proper resource management is an indispensable feature for any framework which is targeted to a Java EE container. But please consider that the Lightstreamer Java SE Client API has been primarily designed having in mind the client side environment, as almost prevalent use case for the majority of our end users.

All that said, we have already noticed the issue you reported, and I confirm that we are currently fixing it in the new version of the Unified API library, in order to make it perfectly working also on a server side environment, at least for what concerns a properly binding of the LightstreamerClient lifecycle to the one of a Java EE Session.

The new version of the library should be available within a week or two. We will post back here to let you know the availability on Maven.

Thanks and Regards,
Gianluca

KlausGroenbaek

Perfect,
Looking forward to trying it out. Too bad it's not open source or I could have written the unit test for you :Smile_Ab:

/Klaus

Alessandro Alinone

Hi Klaus,

The good news is that we will open source all the Lightstreamer client libraries. It will take some time though.

Thanks,
Alessandro

Gianluca Finocchiaro

Hi Klaus.

just to inform you that we have released the final version of our Java SE Client API, with an extended disconnection strategy to allow a graceful shutdown of all involved threads.

Please update your pom.xml descriptor with the updated version (3.0.0), in order to get all dependencies resolved.

Your feedback will be really appreciated.

Thanks and Regards
Gianluca

KlausGroenbaek

Hi

Thats for the fix, the threads are now shutdown correctly when I undeploy my application.

However I have encountered another issue. It looks like onEndOfSnapshot is never fired after I call connect. I saw this issue periodically with 3.0.0-b2, where onEndOfSnapshot was only called about 80% of the time, in the remaining 20% I never got this event and in those cases I would never get any events on the stream.
I asked the service provider to help me debug it, so I have created a sample application on github, https://github.com/QwertGold/cityindexStreaming. If you have a few minutes to take a look and see if I'm doing anything out of the ordinary, it would be much appreciated.

Regards,
/Klaus

Gianluca Finocchiaro

Hi Klaus,

thank you very much for your feedback!

At first sight your project would seem ok, but when cloned and launched from my machine I can't see any real time update coming from the remote server. It may be a symptom that something has gone wrong in the subscription phase, so I ask you to check it with your service provider and then get back to us to go on with our investigation.

Regards,
Gianluca

KlausGroenbaek

Hi Gianluca

The service provider didn't document which mode I should use when subscribing to their Orders stream, I found a forum entry that said I could use DISTINCT, or RAW, and unlucky as I was I picked DISTINCT.
Strangely enough DISTINCT worked about 80% of the time with LS5 and LS6-b2, but not the final LS6. After I got a running example from the service provider I noticed that they used RAW, and after I made that adjustment it worked fine.
I'm a little puzzled about the 20% periodic failure rate, but I'm thinking it's related to some implementation detail in the service provider, and they probably never tested it or planed it to work with DISTINCT mode.

Thanks again, for taking time to look at this.

/Klaus

Giuseppe Corti

Hi Klaus,

I confirm that you should always be aware of which mode to use in your subscriptions on the client.

Indeed, at the server side, the Metadata Adapter should always define the subscription methods permitted for each Item through a proper implementation of the isModeAllowed method.
If this is not done, or are admitted for the same Item two conflicting modes, we could have unpredictable behavior.

For example if a client subscribe an Item in MERGE mode and a subsequent client requesting the same Item in DISTINCT mode, this will be eventually refused by Lightstreamer server.
But, if we invert the temporal order of the two clients, only the DISTINCT subscriptions will work.

Regards,
Giuseppe