Adapter ports not up

whewell

Hi guys,

We are having a weird issue with adapter ports on our Lightstreamer server.

Basically we have setup a data adapter running on "request_reply_port" 9000 & "notify_port" 9001. When we first start up the server, we could see the ports being open. But after our client side code (j2ee) making a connection request, the server would stop listening to those ports.
This suddenly started happening today. And we could find no error message in lightstreamer logs.

Can you please help us?

whewell

Could it be that if more than one connection attempts are being made to the server at one time, it brings down the ports?
We might have more than one threads trying to connect to those ports at the same.

Mone

hi,

that sounds strange to me,
which adapter are you using: the "Robust" version or the simple one?

if possible may you show me your server log?

whewell

Thank you for your reply Mone.

We are using RobustNetworkedDataProvider
Here is the config

<?xml version="1.0"?>
<adapters_conf id="AUS">
    <metadata_provider><adapter_class>com.lightstreamer.adapters.metadata.LiteralBasedProvider</adapter_class>
    </metadata_provider>
    <data_provider name="AUS_ADAPTER_1"><adapter_class>com.lightstreamer.adapters.remote.data.RobustNetworkedDataProvider</adapter_class>
        <param name="request_reply_port">9000</param>
        <param name="notify_port">9001</param>
        <param name="timeout">30000</param>
        <param name="events_recovery">use_snapshot</param>
    </data_provider>
</adapters_conf>

The server log is over 2gb in size from yesterday. I will find a way to get a segment of it.

btw, I'm getting a "delivery failed" when sending an email to support@lightstreamer.com. Is that email still in use?

whewell

Hi Mone,

You can download the server log from here

The log file when unzipped is about 677MB in size. They are from yesterday between 12:00 and 13:00.

btw, we've turned on DEBUG logging briefly at around 12:15.

Thank you for your help.

Mone

Hi,

I can't download the file (I get a timeout error), btw I was expecting something a lot smaller just showing the issue.

the support@lightstreamer.com address is still active but does not support such big attachments; may you please check that the the size of the attachment was the reason for the failed delivery?

Mone

also, the correspondent snippet of the log on the adapter side may be useful too.

whewell

Hi Mone,

Sorry about the massive log file. Please see the attached a much smaller version.

I had another look in the log. I found some interesting error message right after the server restarted. This is also when another server responsible for updating data started making multiple connection attempts to LS server. And I started seeing this error message popping up.

ERROR|LightstreamerLogger.connections  |NIO ACCEPT SELECTOR        |Accept error on socket Lightstreamer HTTP Server

Then over time, it becomes more and more frequent. At one point, the message would repeat over 40 times within a second. (see the end of after_restart.log)

The other log file attached contains logs with DEBUG turned on. I was hoping this could give more insight on what was happening.

Really appreciate your time on this one. We are very puzzled...

Cheers,
William

Mone

hi,

A
are you on windows or linux? The last time I've seen a "Accept error on socket Lightstreamer HTTP Server" error was on a linux machine: an ineffective ulimit call led to the exhaustion of the number of open files (sockets are considered as open files on the system configurations). What is strange is that the exception associated with the error is not logged as I would expect. (you can check the limits on a linux machine calling ulimit -a ). Also windows probably has a limit somewhere (register?).

B

Could it be that if more than one connection attempts are being made to the server at one time, it brings down the ports?

I'm not sure about this. The proxy data adapter expects one connection (per port) only for sure, but I don't think it will close everything down if a second request arrives. Unfortunately the guy who wrote the remote adapter is out of office until the the 9th of august. I may check the sources for confirmation in case A does not explain the issue.
Btw did you implemet the ARI protocol or did you use our .NET remote adapters?

Dario Crivelli

On point B, I confirm that the Proxy Adapter on the Server opens each of the configured ports to accept a single connection, then it stops accepting connections on the port.
Multiple remote counterparts talking to the same Proxy Adapter are not supported.
In case of a Robust Proxy Adapter, when a recovery is needed, the ports are reopened.

whewell

Hi guys,

Thanks for your reply.

We had to increase the open files limit on the server to solve the problem.
What's strange is that the number of sessions could somehow reach nearly 44,000.
Seems like when a user opens multiple pages, multiple sessions are created. Is there a good way to bring this number down? Like using only one session for each ip address?

And we have a new problem.

We are now able to start up the LS server, and keep it getting updated for a day or so. But the next morning, the socket connection between the application for supplying data and LS server would mysteriously drop. And we have to manually reconnect it through jmx. This seems to be a different issue all together. And I'm not even sure if this is a LS issue...

Any help is much appreciated!

Cheers,
William

whewell

Hi guys,

We found some error logs right before the connection went down. Does this mean anything to you?

LS_sessi13-Aug-10 09:51:07,322 |ERROR|LightstreamerLogger.connections |NIO WRITE SELECTOR |Failure in HTTP connection on Lightstreamer HTTP Server
com.lightstreamer.h.nb
at com.lightstreamer.f.a.hb.b(hb.java)
at com.lightstreamer.f.a.hb.c(hb.java)
at com.lightstreamer.f.a.i.c(i.java)
at com.lightstreamer.h.hb.g(hb.java)
at com.lightstreamer.a.qc.h(qc.java)
at com.lightstreamer.h.ob.a(ob.java)
at com.lightstreamer.h.ob.a(ob.java)
at com.lightstreamer.h.eb.a(eb.java)
at com.lightstreamer.g.a.y.b(y.java)
at com.lightstreamer.g.a.j.d(j.java)
at com.lightstreamer.g.a.j.a(j.java)
at com.lightstreamer.g.a.c.run(c.java)

Thanks,
William

Dario Crivelli

If you have multiple pages on the same web application that need to communicate with Lightstreamer Server, you are recommended to use a single session and share it among all pages. See DOCS-SDKs\sdk_client_web\doc\Web Client Dev.pdf, paragraph 2.2, or analyze the code of the various demos.

Turning to the other issue:

By "the application for supplying data" do you mean a Remote Server that communicates with Lightstreamer Server through the ARI protocol?
Which is the role of jmx?

The reported exception is about a session that founds itself closed by an external agent. It is not a normal case, but it seems unrelated with problems that may be occurring in the communication between LS Server and the Remote Server.

We need more details to proceed.
Can you identify the exact moment in which the communication between LS Server and the Remote Server is broken and provide us with a log snippet including the whole period? In case it's too big, could you manage to replicate the issue with only a few active sessions?

whewell

Hi DarioCrivelli,

Thanks, we will have a look at the doco.

The other application is responsible for feeding data to LS server through the RobustNetworkedDataProvider adapters. We've exposed mbeans in jmx to connect/disconnect socket connections to LS server.

We did a tcp dump. And we managed to find an error msg right before the moment when connection was lost. Here it is:

8-Aug-10 09:08:51,615 |ERROR|.data.RobustNetworkedDataProvider|#7 Reply Receiver          |Got fatal error from the inner Proxy Adapter
com.lightstreamer.adapters.remote.RemotingException: Exception caught while reading from the reply stream: Connection reset
        at com.lightstreamer.adapters.remote.request_reply.NotifyReceiver.run(NotifyReceiver.java:71)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
        at java.io.InputStreamReader.read(InputStreamReader.java:167)
        at java.io.BufferedReader.fill(BufferedReader.java:136)
        at java.io.BufferedReader.readLine(BufferedReader.java:299)
        at java.io.LineNumberReader.readLine(LineNumberReader.java:182)
        at com.lightstreamer.adapters.remote.request_reply.NotifyReceiver.run(NotifyReceiver.java:66)
18-Aug-10 09:09:06,307 |ERROR|LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Bad request: LS_session parameter missing
18-Aug-10 09:09:06,307 |ERROR|LightstreamerLogger.connections  |SERVER POOLED THREAD 2     |Failure in request processing: Bad request: com.lightstreamer.b.c: LS_sessi
on parameter missing
18-Aug-10 09:09:21,621 |ERROR|rs.remote.data.RemoteDataProvider|Timer-10                   |Exception caught while subscribing to item 'U20'
com.lightstreamer.adapters.remote.RemotingException: Timeout while waiting for an answer to request 460000012a7e94cbeb
        at com.lightstreamer.adapters.remote.request_reply.RequestSender$1.run(RequestSender.java:200)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

Do you know why the "fatal error" occurred?

Also on our LS server unix box, we've made some changes to Sysctl keepalive options. They have been tuned down significantly to detected stale connections sooner (in sec):

net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 45

From your point of view, would this be good to the system?

Dario Crivelli

From Lightstreamer Server point of view, the connection was closed by the Remote part. Do you feel it's possible? Is the log a full snippet or has it been filtered? Was anything logged just before the first line reported?
Of course, the error is "fatal" only for the underlying Proxy Adapter. The RobustNetworkedDataProvider can overcome the issue by creating a new underlying Proxy Adapter.

About keepalives, if you can enforce them, this can only have positive effects;
they can also prevent any intermediate node from dropping the socket.
Note that the ARI protocol also defines its own keepalives, that can be sent by the Remote Server to the Proxy Adapter (with no timeout checks, though).
I suppose you are not using our .NET Remote Adapter SDK;
otherwise, consider that the Remote Server already sends keepalives every second (keepalives are not sent when data already flows on the connection).