engcoder
hello,
i have very urgent problem on lightstreamer that installed on my server. i have a tool on the server that subscribes for items and make some calculations. Today and Yesterday the tool gives Exception said "push server exception" then the tool stopped working but the lightstreamer still working.
i don't know what is the reason for that so i attach the log files for today and yesterday.
this is the link of the logs :
https://www.mediafire.com/?5g57bzqla0w75ni
Dario Crivelli
Hi, Please provide details on the kind of error.
Is the "push server exception" an exception thrown by your tool or is it the PushServerException that is thrown by some of our client SDKs?
In the latter case, what is the associated message?
Can you specify the exact time of one occurrence of the issue, to help us browsing in the log?
I see you are writing on the ".NET Adapter API" forum, but can you confirm that your tool is actually a client?
Do you also notice adapter side problems?
The log shows that there are delays in answers by the Remote Adapter to subscription requests.
After a timeout, the Server gives up, hence the subscriptions, even if finally successful, will never yield any data.
Can this be a source of issues for you tool?
engcoder
hi,
the time was between 1:00 PM and 1:30 PM.
i guess the error due to the client not the adapter but i am not sure becaue i can't investigate the log file.
what is the reason of the timout that made the server gives up?
Dario Crivelli
I assume that the provided timeframe is referred to the latest log.
I don't notice anything special in that period, but for a massive disconnection and reconnection at 13:02.
Is that unexpected or is that normal and, actually, the milestone that you consider as the start of the specified timeframe?
The observed subscription timeouts are caused by delays in the Remote Adapter in answering to Subscribe requests;
this is likely originated in your Remote Adapter code, and it means that the Subscribe call took more than 10 seconds to complete.
Did you notice these delays?
Anyway, the delays occur far before and far after your timeframe; so they seem to be unrelated with client issues occurred within the timeframe.
So, it becomes important for us to get answers to the other questions, that is:
Is the "push server exception" an exception thrown by your tool or is it the PushServerException that is thrown by some of our client SDKs?
In the latter case, what is the associated message?
By the way, many warnings in the log say:
Unexpected UPDATE event for key xxxxx. Event propagated as an ADD command.
Such UPDATE is set by your Data Adapter when sending an event for an item in COMMAND mode;
it is unexpected if no previous ADD for the key was issued.
So, the UPDATE is turned into an ADD.
If you just use UPDATE instead of ADD for simplicity, this may work.
Otherwise, consider that each ADD increases the item snapshot to be kept by the Server and sent to the clients, whereas the UPDATE just changes the item state.
engcoder
hello,
at 13:02 the disconnection and reconnection is unexpected. what may cause that?
the PushServerException is thrown by client SDKs.
Dario Crivelli
By focusing on the massive disconnection, now I see that it is preceded by an abnormal Garbage Collection activity.
Until then, the JVM seems to have plenty of memory and it seems to recollect the consumed memory very easily.
However, at 13:02:30, even though the free memory is huge, it probably initiates a major collection and this keeps the Server blocked for about 20 seconds.
The block in the data flow eventually causes almost all the active sessions to be interrupted and restarted
(this is the expected consequence, because of the checks in the client SDK libraries).
This kind of accidents can and should be avoided by acting on the JVM configuration, which is specified in the Server's launch script, LS.sh or LS.bat.
Please check the GC parameters used. Try specifying an explicit maximum pause requirement through the -XX:MaxGCPauseMillis setting; the reconnections are triggered by pauses longer than a few seconds, so you don't have to force a tight requirement.
Can this episode explain the problems observed in your tool?
I see that the disconnections are then recovered successfully with new sessions and new subscriptions. Which client SDK is your tool based on?
Did you notice permanent damages in your tool's behavior?
engcoder
thanks for reply.
the GC was work normal for along time. what may cause the abnormal Garbage Collection activity at this time?
Dario Crivelli
If you have changed the JAVA_OPTS in the LS.sh or LS.bat script with custom settings, it may depend on them;
otherwise, it is strange that the default behavior of the GC in Java 8 could lead to that situation.
Please start collecting GC statistics from the JVM and if the issue reoccurs we may try to find some clue.
This is also done by changing the JAVA_OPTS; you can set -XX:+PrintGCDetails for instance.
