LightStreamer thread hangs when closing application

PowerUser

Hi,

We have a serious problem with the LightStreamer .NET Client (2.1.4588.28986). When user is closing our application the LightStreamer thread hangs and the host process doesn't shut down. Is there any way to fix this problem?

Here is a stack trace:

---------------------------------------------Thread 8
Current frame: 
Child-SP RetAddr Caller, Callee
0000000023c7e7a0 000007feea48bd9e (MethodDesc 000007feea01b368 +0x15e DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeWaitHandle))
0000000023c7e7e0 000007feea4295c4 (MethodDesc 000007feea0025d8 +0x14 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
0000000023c7e8d0 000007feea4295c4 (MethodDesc 000007feea0025d8 +0x14 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
0000000023c7e910 000007fe8ebd36c7 (MethodDesc 000007fe8ec4b370 +0x137 Lightstreamer.DotNet.Client.NotificationQueue.dequeue())
0000000023c7e950 000007feea422d47 (MethodDesc 000007feea0f4048 +0x17 System.Threading.ThreadHelper.ThreadStart_Context(System.Object))
0000000023c7e980 000007feea40f8a5 (MethodDesc 000007feea001c38 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
0000000023c7eae0 000007feea40f609 (MethodDesc 000007feea001c28 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
0000000023c7eb10 000007feea40f5c7 (MethodDesc 000007feea001c18 +0x57 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
0000000023c7eb60 000007feea422d21 (MethodDesc 000007fee9ffa8c0 +0x51 System.Threading.ThreadHelper.ThreadStart())
---------------------------------------------

---------------------------------------------
Thread  11
Current frame: 
Child-SP         RetAddr          Caller, Callee
000000002570df30 000007fee9590849 (MethodDesc 000007fee93b5ea8 +0xb9 System.Net.ContextAwareResult.CaptureOrComplete(System.Threading.ExecutionContext ByRef, Boolean))
000000002570df50 000007fee9593378 (MethodDesc 000007fee92e7f20 +0x108 System.Net.Sockets.Socket.DoBeginSend(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.OverlappedAsyncResult))
000000002570dfe0 000007fee959dfe6 (MethodDesc 000007fee92e7820 +0x126 System.Net.Sockets.Socket.BeginSend(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef, System.AsyncCallback, System.Object))
000000002570e050 000007fee958096d (MethodDesc 000007fee92fd5e0 +0xdd System.Net.Sockets.NetworkStream.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object))
000000002570e300 000007feea40fc4b (MethodDesc 000007feea001cf8 +0x1db System.Threading.ExecutionContext.Capture(System.Threading.StackCrawlMark ByRef, CaptureOptions))
000000002570e390 000007feea418520 (MethodDesc 000007feea001cd8 +0x20 System.Threading.ExecutionContext.Capture())
000000002570e3f0 000007feea4295c4 (MethodDesc 000007feea0025d8 +0x14 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
000000002570e4e0 000007feea4295c4 (MethodDesc 000007feea0025d8 +0x14 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
000000002570e520 000007fe8ebd58d9 (MethodDesc 000007fe8ec800a0 +0x149 Lightstreamer.DotNet.Client.HttpProvider.DoHTTP(Boolean))
000000002570e5a0 000007fe8ebd53cf (MethodDesc 000007fe8ec80098 +0x2f Lightstreamer.DotNet.Client.HttpProvider.DoHTTP(System.Collections.IDictionary, Boolean))
000000002570e5e0 000007fe8f029ca0 (MethodDesc 000007fe8ec4d5c8 +0x1d0 Lightstreamer.DotNet.Client.PushServerTranslator.CallResync(PushServerProxyInfo, Lightstreamer.DotNet.Client.ConnectionConstraints))
000000002570e630 000007fe8f0294f5 (MethodDesc 000007fe8ec4cac0 +0xb5 Lightstreamer.DotNet.Client.PushServerProxy.ResyncSession())
000000002570e6e0 000007fe8f0290eb (MethodDesc 000007fe8ec4af60 +0x5b Lightstreamer.DotNet.Client.ServerManager.Rebind(ActivityController))
000000002570e750 000007fe8ebd825a (MethodDesc 000007fe8ec4af58 +0x18a Lightstreamer.DotNet.Client.ServerManager.WaitEvents())
000000002570e800 000007fe8ebd7f56 (MethodDesc 000007fe8ec80b10 +0x36 Lightstreamer.DotNet.Client.ServerManager+SessionActivityManager.Run())
000000002570e860 000007feea40f8a5 (MethodDesc 000007feea001c38 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002570e9c0 000007feea40f609 (MethodDesc 000007feea001c28 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002570e9f0 000007feea40f5c7 (MethodDesc 000007feea001c18 +0x57 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000002570ea40 000007feea422d21 (MethodDesc 000007fee9ffa8c0 +0x51 System.Threading.ThreadHelper.ThreadStart())
---------------------------------------------

Dario Crivelli

Hi, I confirm that the last of the two threads you reported is not a daemon thread and can prevent the process from terminating normally.
We chose this way because it is the thread that gets updates from Lightstreamer Server and dispatches them to the application;
so, in principle, this may be the only active thread a client application may rely upon.

But this thread was not supposed to hang at that point;
in fact, in that point, it is waiting for the outcome of a HttpWebRequest.BeginGetResponse invocation, which is supposed to cause the supplied callback to be eventually invoked (and the invocation would provide the needed outcome).
Do you see other live threads other than the reported Lightstreamer-related ones?

PowerUser

Hi,

Thank you for replying to me.

I've used "!eestack -ee" WinDbg-command, and I haven't seen any other live threads (dump file: https://drive.google.com/file/d/0B-LkSR9nTvvLNGEzQTQ5cFBJOUk/edit?usp=sharing [142mb]).

How do I close an application to terminate the LS-threads properly?

Dario Crivelli

I have managed to replicate the case, but only in very unusual conditions.

According to the previous thread dump (I would like to avoid delving into the memory dump in the first place) the client has just issued a rebind request and it is waiting for the answer, which is not coming.
This can be replicated by blocking the Server, but not by closing the Server (in the latter case, HttpWebRequest.BeginGetResponse would invoke its listener).
Do you notice any issues in the Server or in the network, such that a request reaches the Server but gets no answer?

As long as the client is waiting for a connection to be established, it is correct that it keeps the process alive, as the underlying session is still to be considered alive, until the reconnection timeout expires.
By the way, the reconnection timeout is the ReconnectionTimeoutMillis property of ConnectionInfo, but if a session is in smart polling, the whole PollingIdleMillis is added.
After the timeout expires, the connection attempt should be abandoned.

However, I acknowledge that there is a bug.
In fact, after the timeout expires, the underlying connection attempt is not interrupted;
hence the thread (which is still not a background thread) keeps the process alive until something happens on the network and this may last for unboundedly long.
We'll try to fix that behavior for the next release.
Sorry for the inconvenience.

For now, the application should cooperate.
When the reconnection timeout expires, the application receives an invocation of OnFailure on the IConnectionListener.
There, it should evaluate if the process must terminate and, in that case, force the termination through Environment.Exit.
On the other hand, if the process issues CloseConnection while the blocked reconnection attempt is in place, OnFailure will not be invoked and the session will close normally;
however, the case is also affected by the bug, hence, the process will still be kept alive.
So, also after invoking CloseConnection, the application should evaluate if the process must terminate and, in that case, force the termination through Environment.Exit.
Alternatively, you could handle both cases by acting in the OnClose callback, which is always invoked.

Dario Crivelli

I must correct myself (the last row only):
The alternative of leaning on OnClose is also not available, because OnClose may also not be invoked, as a consequence of the same bug.
(I have also moved the thread to the client stuff)

PowerUser

Hi,

Thank you for replying. Could you please announce the approximate date of the fix for this issue?
Looking forward for the solution, because it is very important to us.

Dario Crivelli

The fixed library will be delivered upon the next public release of LS.
No patch for LS 5.x has been planned at the moment;
so, the case should be the next alpha release of LS 6.0
(where the included .NET Client library will be still compatible with Server 5.x).
This should happen within a few months, but no date has been scheduled yet.

If you are a customer, with an active support & maintenance contract, then you can receive a library patch in short time, by requesting it to support@lightstreamer.com