Error 110 Received in the browser

ctranfd

Hi,

We have a situation where
1. The browser takes a long time to connect to Lightstreamer
2. After sometime, the request to Lightstreamer times out and the following error message was received - Error 110 It has been detected that the JavaScript engine of this browser is not respecting the timeouts in setTimeout method calls. The Client has been disconnected from the Server in order to avoid reconnection loops. To try again, just refresh the page.

We're unsure as to what has caused this - is it possible that the number of connections to Lightstreamer had reached a maximum number and then Lightstreamer could not accept incoming connections?

Also if the Lightstreamer application ran out of memory, could this be a cause? Is there a way to specify how to allocate more memory? (is this specified in the LS.sh java opts?)

We haven't been able to determine the exact cause and the client raised has just this with us, so any help would be much appreciated.

Many Thanks,

CT

Mone

Hi,

such error is related to the issue described here: http://www.lightstreamer.com/vb/showthread.php?t=12

Are you in such situation?

If not,
does it happen on different browsers?
Can you send me a server log snippet that covers the period when the issue was seen (and telling me the IP of the client that had the issue)?

ctranfd

Hi Mone,

Thanks for the quick reply.

This does not seem to be isolated to Firefox I'm afraid.

I've attached the log snippet to this reply.

If it helps we have also noticed that we receive the following errors in the logs
- Accept error on socket Lightstreamer HTTP Server java.io.IOException: Too many open files
- Failure in request processing: Bad request: com.lightstreamer.b.c: It was expected an integer value for LS_polling_millis
- Bad request: It was expected an integer value for LS_polling_millis
- Sync error: Can't find session from ...

Many Thanks,

CT

ctranfd

Also, if this helps, we are using the default values in lightstreamer_conf.xml.

The message that Lightstreamer is sending to the browser is approximately 2,500 characters. We consider this quite small, but do we need to adjust any of the configuration?

The version of Lightstreamer we are using is 3.5 build 1428.4 (Allegro).

Mone

- Sync error: Can't find session from ...

This is the cause for the error 110 message. It is clearly a false positive. It is already fixed in our development trunk and will be released probably before the end of the year. In the mean time you could suppress the alert re-implementing the onClientAlert and filtering the 110 code but I advise against this workaround because a sequence of sync errors, that causes the false positive, should never happen in normal situations.
A sync error means that a session is not found or is expired on the server, it could happen sometimes but if it happens several times to the same client in the same second probably means that the client is continuosly trying to work with a session that was created in another server.
Are you using a load balancer in front of Lightstreamer?

- Accept error on socket Lightstreamer HTTP Server java.io.IOException: Too many open files

Are you running Lightstreamer on an unix box right? This is a problem with the ulimit settings; check this thread

- Bad request: It was expected an integer value for LS_polling_millis

This could be a side effect of the other two problems, let's solve them and return on this if it doesn't disappear.

ctranfd

Hi Mone,

Are you using a load balancer in front of Lightstreamer?

We are using a load balancer in front of Lightstreamer. We have 2 Lightstreamer servers in this load balancer and have only now just removed the 2nd Lightstreamer. We will be testing the load tomorrow morning.

Are you running Lightstreamer on an unix box right? This is a problem with the ulimit settings;

Yes both Lightstreamers that we have are on unix. I believe we have set up the ulimit value - but I will double check with operations to find out what number was set. Typically, if a site has about 6000 hits a day, then we would double this value. Since we have 2 Lightstreamers which were both in the Load Balancer, the value would be have set 6000 for each box. Is this the correct configuration?

Also for a Windows machine, how would the ulimit value be set? Is it required? This would help recreating the problem on our development machines.

Thanks heaps,

CT

Mone

We are using a load balancer in front of Lightstreamer. We have 2 Lightstreamer servers in this load balancer and have only now just removed the 2nd Lightstreamer. We will be testing the load tomorrow morning.

The problem is not the balancer in itself, but its configuration. Please ensure that your balancer and your servers are configured in one of the way explained in the Clustering.pdf file available under LS_HOME/DOCS-SDKs/

Typically, if a site has about 6000 hits a day, then we would double this value. Since we have 2 Lightstreamers which were both in the Load Balancer, the value would be have set 6000 for each box. Is this the correct configuration?

I thinkl that your approach is good enough, btw check if the limit is applied as you expect and doesn't fail silently.

Also for a Windows machine, how would the ulimit value be set? Is it required? This would help recreating the problem on our development machines.

we never had any problem with open sockets and windows. Someone here is suggesting that windows has a huge limit somewhere in the registry that applies to all processes.

ctranfd

Hi Mone,

We are currently testing with less load at the moment and are incrementally adding more on the live sites until we find the issue.

The ulimit values that were set are:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 155648
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 155648
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Do you see any problem with any of the other values (besides open files?).

In regards to clustering, the setup is as follows

Lightstreamer load balancer: port 80
Lightstreamer 1: port 8080
Lightstreamer 2: port 8080
Application (that updates both Lightstreamers data): connects directly to both Lightstreamers on port 8080, not through the load balancer.

The reason why the Lightstreamers are running on port 8080 was enforced by the operations team for security reasons. Should this have an effect on the error that we have seen?

Thanks!

CT

Alessandro Alinone

Hi,

@ctranfd, Wrote: Do you see any problem with any of the other values (besides open files?).

Open files seems to be the only issue.

@ctranfd, Wrote: In regards to clustering ...

Referring to the "Clustering.pdf" document, what option are you using?
Also, what edition of Lightstreamer are you using?

Regards

Alessandro

ctranfd

Hi Alessandro,

I believe we using Option 1.A.1 - all requests go through the load balancer IP address, as both Lightstreamers cannot be reached via public IP. The load balancer routes the requests to the Lightstreamer boxes.

The application which updates data to both Lightstreamers connect directly to each Lightstreamer box (as the application and both Lightstreamers are internal).

The version of Lightstreamer we are using is Lightstreamer we are using is 3.5 build 1428.4 (Allegro).

Thanks,

CT

Alessandro Alinone

Hi Claire,

Thanks for your clarifications.

So this seems to be a very simple clustering scenario, where the request dispatching is fully delegated to the load balancer. Is it possible that the balancer fails to guarantee stickiness based on IP address (perhaps the client IP addresses are changeable)?

We could understand this by comparing the logs for both the servers, to see if a session originated in one server is wrongly bound to the other server on subsequent requests.

Please contact support@lightstreamer.com.

Cheers,

Alessandro