Throttling resources used

kpturner

My LS server is not tuned in anyway. It is running on an IBMi server with 32 processors and 1.5TB of RAM with basically a default config. I have noticed that when it starts it takes quite a long time before it is responsive, and I take this to mean that it internally detects the CPU and memory available and starts up a number of threads accordingly. If I look at the logs it generates literally hundreds of these INFO messages at startup:

< INFO> Created selector thread: NIO blah blah blah .....>

I do not have a problem at the moment, but what I would like to know is what I can do to throttle back the number of threads created at startup and, if I do, what the implications are.

Can you point me at the relevant piece of documentation to help me understand this please.

Giuseppe Corti

Hi Kevin,

You can refer to "LOAD CONFIGURATION" section of lightstreamer_conf.xml configuration file for an overview of the parameters for load management and configuration of the thread pools.
The inline comment of each parameter specifies the default value assumed and the impact of the parameter for the activity of the server.

Please note that in the Ligtstreamer server version 6.0 (lightstreamer_conf.xml) a couple of parameter were added: <selector_pool_size> and <snapshot_pool_size>.
In particular <selector_pool_size> acts on the number of distinct NIO selectors to which you refer in the post.

I confirm you that <selector_pool_size> works on Lightstreamer server 5.x too, it was considered just as private and therefore not explicitly shown in the configuration file.
Instead, the parameter <snapshot_pool_size> is not provided in 5.x server and so if you want to configure that pool you need to act otherwise; Please let me know if I should deepen on this.

kpturner

Ok thanks. I will have a look and see what further questions arise.

kpturner

Just revisiting this now as I do have an issue being reported by a customer. We can see on their system that the LS server is consuming between 50% and 90% of the servers CPU capacity and only has about 10 connections active (it basically has nothing to do).

Why would this be? I have not modified any parameters yet. Currently it has 170 JVM threads running and in the thread pool it shows 47 and most of the time they are all available.

kpturner

On our machine the LS server is doing much more work and is actually consuming virtually no CPU at all.

Giuseppe Corti

Hi Kevin,

Have you already verified, in the MonitorText logger or through the monitor console, if the number of "Active threads" is constantly or recurrently greater than 0 (even just 1)? This may indicate a thread in loop.
In this case, could you take a JVM thread dump? It could provide us useful information.

Thank you.
Giuseppe

kpturner

Hello

I am not sure what you mean by "MonitorText" logger by the monitor console itself seems to show "Active threads" fluctuating between 0 and 1. It has virtually nothing to do at the moment, and yet it is consuming over 75% of the CPU (and this is a huge IBMi Power Server, so that is a lot of CPU :Smile_Ab: )

This is the only server I have every seem this happening on - we have the same setup running on lots of servers (albeit only in Development mode) and they barely use any CPU at all.

kpturner

I wanted to rule out our own adaptors as an issue so I removed then from the config and restarted. The issue was still present - using loads of CPU but doing nothing :Smile_Ac:

I really need some urgent help in diagnosing the problem because it is obviously not an acceptable situation for the customer.

kpturner

Other clues perhaps:
The clients are all using SSL to connect to the LS server - and the certificate is unsigned. Perhaps irrelevant.

I also need to verify if the server exhibits the same behaviour with no clients connected to it.

kpturner

Lots of warnings and errors on the console:

Warnings:

12:10:29 TLS/SSL read error: Received fatal alert: unknown_ca on 89.133.134.179:58413. NIO READ SELECTOR 5
Errors 12:07:25 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:58298. NIO TLS/SSL HANDSHAKE SELECTOR 4 12:07:35 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:55409. NIO TLS/SSL HANDSHAKE SELECTOR 5 12:07:35 Handshake error on Lightstreamer HTTPS Server: task timed out on 89.133.134.179:58300. NIO TLS/SSL HANDSHAKE SELECTOR 5

Giuseppe Corti

Hi Kevin,

Have you the chance to take some thread dumps at short intervals?

Giuseppe Corti

Furthermore, please be sure that in lightstreamer_log_conf.xml configuration file the log level for LightstreamerMonitorText is TRACE:

<logger name="LightstreamerMonitorText" level="TRACE">

and send us (support@lightstreamer.com) the server log (lightstreamer.log).

kpturner

UPDATE: I reconfigured the servers "allow_access_from" setting to something erroneous so that it had no client connections at all to handle.

It still exhibited the same issue. That at least rules out connection handling as SSL errors as a cause.

kpturner

Hello Giuseppe

I afraid I have no idea how to take thread dumps at short intervals. Can you give me a clue?

I will switch on the tracing option now.

kpturner

My lightstreamer_log_conf.xml file contains this:

<logger name="LightstreamerMonitorText" level="TRACE">              
  <appender-ref ref="LSDailyRolling" />                             
</logger>

but I don't see anything extra appearing in the log file. Am I missing something? Can I tell from the stdout when the server is running whether or not it is using this setting because it would appear not :Smile_Ac:

Giuseppe Corti

Usually the JDK provide a tool to do that. Typically, the command is something like this:

jstack [pid]

But I'm not sure if you're using an IBM JDK and which is the specific command.

Giuseppe Corti

Can you confirm that the level was previously set to ERROR? In this case, the new setting requires a restart of the server.

In any case, if it is possible, a Lightstreamer server restart is advisable. Then, if the issue reappears send us the log (lightstreamer.log).

Thank you.

kpturner

Yes I am using the IBM JDK. There may be a way to do it but I will have to check. The server runs in a batch process so I very much doubt I will be able to get what you want.

As for the TRACE - I did as asked (and I have now confirmed that the server is loading the file that I have altered) but I don't see any extra information in the log file. The only entries for MonitorText are INFO.

Alessandro Alinone

This might help in getting the thread dump on IBM JDK:
http://publib.boulder.ibm.com/infocenter/realtime/v1r0/index.jsp?topic=%2Fcom.ibm.rt.doc.10%2Fdiag%2Ftools%2Fjavadump.html

Basically, you should add "–Xdump:java" to the JAVA_OPTS variable in LS.sh.
Then, you can trigger a dump by calling "kill -QUIT pid" on the PID of the JVM (which will not be terminated).

kpturner

I have found the simplest way of generating the thread dumps is via the normal command line interface like this:
GENJVMDMP JOB(050177/LIGHTSTRM/QP0ZSPWP)

Now the difficult bit is to get them to you, because I have no FTP access to that server. This may take me a while.

In the meantime, any ideas about the MonitorText not outputting any trace information ?