DotNetAdapter LOH fragmentation and latent release of subscription strings on SOH

bkretschmer

Hello,

In our environment we're experiencing both the fragmentation of the .NET LOH (Large Object Heap) and the rather slow release of subscription strings generated by the .NET Lightstreamer Adapter libraries. Memory profiling has indicated that instance(s) of the object Lightstreamer.DotNet.Server.RequestReply.NotifySender executing Run()
is holding onto Lightstreamer subscription strings presumably generated by the Lightstreamer adapter libraries. These subscription strings are observed in the format of: "1361213581995|UD3|S|VIS.USV.02....COMMAND|S|...." and are around 1K each.

A screenshot from the profiler appears below. In this particular scenario, the aforementioned Lightstreamer object type was still referencing over 450MB in subscription string objects in the Generation 2 SOH. This occurred when we had 25 clients each establish a unique subscription for ~130,000 items 30 seconds apart from each other. (client1, client2 30 seconds later, ...) The Lightstreamer.DotNet.Server.RequestReply.NotifySender object(s) were still referencing the subscription strings long after each client had received their snapshot request from the Lightstreamer server. We've also noticed that the NotifySender.Run() continues to reference ArrayList objects on the LOH. So it would appear that these large ArrayList objects are continually created in the LOH and fragmenting the LOH.

We've tried tuning the Data Adapter's app.config file by specifying different values for either gcServer enabled or gcConconcurrent enabled/disabled. These changes have not altered the rapid rise in Generation 2 SOH memory consumption.

Please advise.

Bill

=====================================
Environment Info:
- Lightstreamer Server: Presto version 4.2
- Lightstreamer DotNetAdapter_N2 DLL: v2.0.50727 / 1.7.4175.16355
- OS: Windows 2008 R2 SP1 / .NET 4.0

Dario Crivelli

With a huge amount of updates generated in a loop (such as upon a subscription for 130000 items all of which produce a snapshot) the mentioned _queue object can grow significantly, but we expect that all the contained strings are released before they are sent through the network.

Can you confirm that you could see strings hanging in the queue after the Adapter had terminated the transmission of all the updates?

Otherwise, it is possible that at very high update rates the library can never manage to empty the queue completely.
Would you notice the same issue by lowering the number of items, or would you notice a threshold effect?

About the continuous creation of large array objects, can you identify some of them?
The mentioned _queue object is never recreated, but its internal components may and this is an aspect that may leave room for optimizations.

bkretschmer

The screenshot below(Lightstreamer4.2SOHGen2SubscriptionStrings.png) shows the subscription strings identified by the ANTs memory profiler as being retained in Gen 2 SOH (small object heap) by the NotifySender object mentioned in the first posting. This memory snapshot was taken ~4 minutes after the client had received the push data from the Lightstreamer server. The second screenshot (LightstreamerLOHLargeArrays.png) shows the creation of the large array objects on the LOH. These eventually get cleaned up, but could really fragment the LOH and lead to system degradation over time. The last two screenshots show the instance retention graphs for the object arrays held by Lightstreamer in the LOH.

We are currently running threshold tests based on longer time intervals between initial subscription snapshot requests. Once we complete this, we may also have to undertake data volume threshold testing as well. The rapid SOH rowth due to the subscription string references hanging around in the Lightstreamer Adapter objects has now become a large technical risk for us. The fragmentation of the LOH is also of serious concern for the operational stability of the system over time.

Is there testing you can do on your end to independently verify these issues?

Dario Crivelli

We have not tested the .NET Remote Adapter library against a high load because, from our point of view, this is not the inner loop of the system, as

client subscriptions are expected to have a many to one relationship with subscription requests to the Data Adapter;
the update flow from the Data Adapter is also expected to be significantly lower than the outbound flow managed by the Server.

In fact, as the last resort, one always has the option to get rid of our .NET library and communicate from the back-end to the Server by directly leaning on the ARI protocol, as documented in the Adapter Remoting Infrastructure SDK.

As far as the code is concerned, we don't notice any memory leak related with strings retained, nor a creation of array objects.
A persistency issue may arise in one of the internal queues, which is cleaned up later than needed, but simply because it is cleared after the next event is produced;
so, the issue is visible only if you send many events in a loop and then stop producing events suddenly, but, in fact, this should be considered as a "false positive".

In order to try to replicate and analyze your case in depth, we suggest you interacting with us by writing us at support@lightstreamer.com

bkretschmer

Thank you. I will submit additional questions to support@lightstreamer.com referencing this forum thread.