Throttling resources used

Giuseppe Corti

Hi Kevin,

Have you already verified, in the MonitorText logger or through the monitor console, if the number of "Active threads" is constantly or recurrently greater than 0 (even just 1)? This may indicate a thread in loop.
In this case, could you take a JVM thread dump? It could provide us useful information.

Thank you.
Giuseppe

kpturner

Hello

I am not sure what you mean by "MonitorText" logger by the monitor console itself seems to show "Active threads" fluctuating between 0 and 1. It has virtually nothing to do at the moment, and yet it is consuming over 75% of the CPU (and this is a huge IBMi Power Server, so that is a lot of CPU :Smile_Ab: )

This is the only server I have every seem this happening on - we have the same setup running on lots of servers (albeit only in Development mode) and they barely use any CPU at all.

kpturner

I wanted to rule out our own adaptors as an issue so I removed then from the config and restarted. The issue was still present - using loads of CPU but doing nothing :Smile_Ac:

I really need some urgent help in diagnosing the problem because it is obviously not an acceptable situation for the customer.

kpturner

Other clues perhaps:
The clients are all using SSL to connect to the LS server - and the certificate is unsigned. Perhaps irrelevant.

I also need to verify if the server exhibits the same behaviour with no clients connected to it.

kpturner

Lots of warnings and errors on the console:

Warnings:

12:10:29 TLS/SSL read error: Received fatal alert: unknown_ca on 89.133.134.179:58413. NIO READ SELECTOR 5
Errors 12:07:25 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:58298. NIO TLS/SSL HANDSHAKE SELECTOR 4 12:07:35 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:55409. NIO TLS/SSL HANDSHAKE SELECTOR 5 12:07:35 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:58300. NIO TLS/SSL HANDSHAKE SELECTOR 5

Giuseppe Corti

Hi Kevin,

Have you the chance to take some thread dumps at short intervals?

Giuseppe Corti

Furthermore, please be sure that in lightstreamer_log_conf.xml configuration file the log level for LightstreamerMonitorText is TRACE:

<logger name="LightstreamerMonitorText" level="TRACE">

and send us (support@lightstreamer.com) the server log (lightstreamer.log).

kpturner

UPDATE: I reconfigured the servers "allow_access_from" setting to something erroneous so that it had no client connections at all to handle.

It still exhibited the same issue. That at least rules out connection handling as SSL errors as a cause.

kpturner

Hello Giuseppe

I afraid I have no idea how to take thread dumps at short intervals. Can you give me a clue?

I will switch on the tracing option now.

kpturner

My lightstreamer_log_conf.xml file contains this:

<logger name="LightstreamerMonitorText" level="TRACE">              
  <appender-ref ref="LSDailyRolling" />                             
</logger>

but I don't see anything extra appearing in the log file. Am I missing something? Can I tell from the stdout when the server is running whether or not it is using this setting because it would appear not :Smile_Ac:

Giuseppe Corti

Usually the JDK provide a tool to do that. Typically, the command is something like this:

jstack [pid]

But I'm not sure if you're using an IBM JDK and which is the specific command.

Giuseppe Corti

Can you confirm that the level was previously set to ERROR? In this case, the new setting requires a restart of the server.

In any case, if it is possible, a Lightstreamer server restart is advisable. Then, if the issue reappears send us the log (lightstreamer.log).

Thank you.

kpturner

Yes I am using the IBM JDK. There may be a way to do it but I will have to check. The server runs in a batch process so I very much doubt I will be able to get what you want.

As for the TRACE - I did as asked (and I have now confirmed that the server is loading the file that I have altered) but I don't see any extra information in the log file. The only entries for MonitorText are INFO.

Alessandro Alinone

This might help in getting the thread dump on IBM JDK:
http://publib.boulder.ibm.com/infocenter/realtime/v1r0/index.jsp?topic=%2Fcom.ibm.rt.doc.10%2Fdiag%2Ftools%2Fjavadump.html

Basically, you should add "–Xdump:java" to the JAVA_OPTS variable in LS.sh.
Then, you can trigger a dump by calling "kill -QUIT pid" on the PID of the JVM (which will not be terminated).

kpturner

I have found the simplest way of generating the thread dumps is via the normal command line interface like this:
GENJVMDMP JOB(050177/LIGHTSTRM/QP0ZSPWP)

Now the difficult bit is to get them to you, because I have no FTP access to that server. This may take me a while.

In the meantime, any ideas about the MonitorText not outputting any trace information ?

Alessandro Alinone

Please, send the dump to support@lightstreamer.comvia https://www.wetransfer.com/

My collegaues will answer regarding MonitorText.

Giuseppe Corti

Hi Kevin,

Please note that in the log file the LightstreamerMonitorText lines are always labeled as INFO. The difference lies in the frequency that passes from 2 minutes(INFO) to 2 seconds(TRACE).

kpturner

OK - will send you the log for now. I have requested that they send me the thread dumps (they are very quick to complain about the problem, but not so quick in giving me what I need - a bit like me really :Smile_Ab: )

kpturner

Actually - getting the log is just as difficult for me as the dumps :Smile_Ac: I have requested that they send me these also.

kpturner

These files are way too big to add to a forum post (even when zipped).

I will email them