too much time during GC process

engcoder

hello,
we have a problem during the lightstreamer garbage collection (GC). The total heap of our LS is 6 GB and when the free heap reach a bout 4 GB it takes too much time in GC process about 5:15 seconds that may lead to the lightstreamer be disconnected at this time of GC and then lead to some adapters be disconnected because of no data received.
this problem of GC was started from 2 days ago only but befor there was no problem and the GC was not take any time.

this is the LS log file of today : http://www.mediafire.com/download/3ah8gc5lu9e6oxv/Lightstreamer+-+12-1-2016.rar

engcoder

hello,
remind, i am wait for reply

Giuseppe Corti

Hi Ahmed,

Please could you confirm the Lightstreamer server version in use and if you have added some GC tuning to your Lightstreamer instance?
The garbage collection can be tuned in the launch script of Lighstreamer server, LS.bat or LS.sh depending your os.

Please note that since Lightstreamer 6 we officially adds configuration for MaxGCPauseMillis that try to set a target for the maximum GC pause time. This is a soft goal, and the JVM will make its best effort to achieve it.
This option is valid definitely for Lighstreamer 5.x too; furthermore in case of large heap we also suggest the G1 algorithm. G1 is an incremental parallel compacting GC that provides more predictable pause times compared to standard GC in particular with large heap.

Hence, if you have not yet set any specific GC tuning you might try something like this:

set JAVA_OPTS=-server -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:G1ReservePercent=20

Regards,
Giuseppe

engcoder

we are using the standard GC for java
for long time now we not do any changes to the GC thats why we attach the log file so you may something suspicious
my LS is of virsion 5.1.2
but i remind you that this problem was started from 2 days ago only but befor there was no problem and the GC was not take any time.

this is the LS.bat file portion

rem === CHECK THIS OUT
rem =======================================================================
rem JAVA_OPTS should contain any Java Virtual Machine options. Here are some tips:
rem 1) Always use the "-server" option.
rem 2) Give more RAM to the server process, especially with heavy traffic, by specifying a min and max "heap"
rem E.g.: If you have 4 GB and the box is dedicated to Lightstreamer, you might set 1 GB min heap and 3 GB max
rem heap with these options: "-Xms1G -Xmx3G"
rem 3) Choose a better "garbage collector" if you want to reduce latency. An option that often gives good results is:
rem "-XX:+UseConcMarkSweepGC". If you are using Java 7 (apart from early versions) or later, you might prefer to try:
rem "-XX:+UseG1GC". Many other tuning options are available (please see Oracle docs).
set JAVA_OPTS=-server -Xms6G

Giuseppe Corti

Hi Ahmed,

From the log is evident the behavior of memory, which always repeats the same pattern, drops to approximately 4GB and then starts a major GC to recover approximately 2 GB; this takes several seconds (from 15 to about 1 minute).
This is behavior is not acceptable.

With a heap of 6GB we definitely recommends G1, and so I confirm to try the settings suggested above (rely on the default settings also makes you subject to possible changes of behavior).

That said, it is difficult say why the problem started just two days ago. Some updates at system level (SO updates, JVM change, ...)?
Some changes in memory usage (increase number of sessions, or the same number of sessions but with a increase in the frequency of updates, or larger size of the updates)?
The changes might be very small but have triggered a threshold effect.

Regards,
Giuseppe

engcoder

this is the LS log file of today.
http://www.mediafire.com/download/ddv60nehvhb1dxc/Lightstreamer+-+13-1-2016.rar

there is no problem in LS occured today and all thing works fine.
you can compare this log file with the log of yesterday i attached in the previous post that you may find difference lead to the problem

Giuseppe Corti

Hi Ahmed,

From the new log it is clear that the GC behaves much better.
However I found a couple of cases where we have a GC pause of a few seconds (for example at 11:57:05,996 and at 14:57:11,976).
These pauses are smaller than the connection timeout of clients and therefore have been harmless.

Since the JVM have to handle with a quite large heap, you will always have a potential risk using the default GC configuration.
What triggers the issue is hard to say, it does not seem the server load since it seems that for the Jan 13 the load was greater than Jan 12, do you confirm?
But you should also consider the general state of the macchine apart Lightstreamer JVM process.

But ultimately configuring the GC with the recommended settings you should shirk the issue.

Regards,
Giuseppe