The server I'm trying to use just stalls at bind_session.txt

abcdefg

Hi there.

I am trying to connect to a Lightstreamer server but the connection stalls at the bind_session.txt step.

I can replicate this behaviour in Postman, calling create_session.txt works and produces an LS_session (SessionId), but requests to bind_session.txt never return anything and always just time out. Just to be clear, we do not get any error message or anything at all. It just never responds to the request with any data whatsoever.

In the iOS Client, this manifests as an endless cycle of connecting and disconnecting...

	CONNECTING
	CONNECTED:STREAM-SENSING
	CONNECTED:HTTP-STREAMING
	CONNECTED:HTTP-POLLING
	DISCONNECTED:WILL-RETRY
	CONNECTING
	CONNECTED:HTTP-POLLING
	DISCONNECTED:WILL-RETRY
	CONNECTING
	CONNECTED:HTTP-POLLING
	DISCONNECTED:WILL-RETRY
	CONNECTING
	CONNECTED:HTTP-POLLING
	DISCONNECTED:WILL-RETRY
	CONNECTING
	...

The same client connects to other Lightstreamer servers ok but just not to our production server (I believe it's an Allegro server, 5.1.1).

For the record, the client code is super simple:

    override func viewDidLoad() {
        super.viewDidLoad()
        
        LSLightstreamerClient.setLoggerProvider(LSConsoleLoggerProvider())


        let client = LSLightstreamerClient(serverAddress: "https://....:443/", adapterSet: ".....")
        client.addDelegate(self)
        client.connect()
    }
    
    func client(client: LSLightstreamerClient, didChangeStatus status: String) {
        NSLog(status);
    }

I have no idea what the next debugging step would be. What would cause this? How do we debug this issue?

Giuseppe Corti

Hi abcdefg,

Please let me clarify a few preliminary considerations.

- Lightsreamer Allegro edition standard does not support iOS client, please could you confirm that your production server is entitled with the specific optional feature?
- Is the Adapter Set definition identical between the test servers and the production one?

Thank you,
Giuseppe

abcdefg

Hi

1. Well I know that the server does support both iOS and Android because it's working for plenty of our live users on those platforms. But I don't know how to verify this formally. But I don't get "License not valid for this Client version" (the way I do if I try to connect with nodejs for example).

2. No, my successful connections are essentially to a local copy of Lightstreamer-example-HelloWorld-adapter-java. So the adapter set is "HELLOWORLD" for that.

I didn't mention that I do also have some testing environments to work with that match the structure of our production server and if I try to connect to those, they behave exactly the same way as production. So I take from that that whatever this issue is it's our fault somehow.

Giuseppe Corti

Hi abcdefg,

Ok, in this case could you change the logging configuration (lightstreamer_log_conf.xml) of one of your testing server with these setting

	<logger name="LightstreamerLogger.connections" level="INFO"/>
	<logger name="LightstreamerLogger.requests" level="INFO"/>
	<logger name="LightstreamerLogger.push" level="DEBUG"/>

and repeat the test case?

So, we should get some more precise information from the server log file.

Regards,
Giuseppe

abcdefg

Should be attached to this post.

Giuseppe Corti

Hi abcdefg,

Thank you for the log, indeed the case is quite strange.
It seems that some replies from the sever are missed by the client. Please could you confirm that there are no network intermediaries (proxies or something similar) in the middle that can buffer in some cases the replies?

Otherwise, please could you retry the test case after adding this settings to lightstreamer_conf.xml (under the <lightstreamer_conf> root):

<keepalive>N</keepalive>

and add also this log setting:

	<logger name="LightstreamerLogger.requests.polling" level="INFO"/>

Regards,
Giuseppe

abcdefg

Thanks for the help Giuseppe

I've made those changes and I have another log file for you but I can still see keepalive mentioned in there ...

Giuseppe Corti

Hi abcdefg,

I confirm that it seems that the <keepalive>N</keepalive> setting was not applied.
Please could you check the lightstreamer_conf.xml configuration file, the <keepalive>N</keepalive> should be just after <lightstreamer_conf>.

If all seems correct to you please, repeat the test case with this additional log setting (I warn you that this is very verbose):

<logger name="LightstreamerLogger.requests" level="INFO"/>

In addition I am wondering if you could arrange a network sniff.

Regards,
Giuseppe

abcdefg

Okay I am very confused now.

Here is the first bit of my lightstreamer_conf.xml file:

<?xml version="1.0"?>


<lightstreamer_conf>


    <keepalive>N</keepalive>
    <logger name="LightstreamerLogger.requests" level="INFO"/>


<!--
    The following elements, subelements and attributes are consulted
    by Lightstreamer Server to setup its own configuration. Elements with
    an empty or blank value are ignored and considered as not defined at all.
    Elements described as "cumulative" can be inserted multiple times in the
    same place.

I've attached another log of a failed connection. But it looks exactly the same to me. It doesn't look like these configuration changes are having any effect. I restarted the Lightstreamer server prior to the test, just to be sure.

There are no other <logger> tags inside lightstreamer_conf.xml, but there are a lot of them inside lightstreamer_log_conf.xml. One of them is exactly the same as the above command. I'm not sure if statements in this file will also take effect.

Please could you confirm that there are no network intermediaries (proxies or something similar) in the middle that can buffer in some cases the replies?

I don't think so.

In addition I am wondering if you could arrange a network sniff.

Shouldn't be a problem, I'll probably work on this tomorrow.

Giuseppe Corti

Hi abcdefg,

I am very sorry to have been inaccurate in my instructions.
Indeed the logger configuration is related to the lightstramer_log_conf.xml file and not to lightstreamer_conf.xml and also the suggested level was wrong.

In lightstramer_log_conf.xml please change the level for <logger name="LightstreamerLogger.requests" level="WARN"/> to DEBUG.

Anyway, in this log it is clear that the <keepalive>N</keepalive> setting was successful. You may notice that there are no more rows in the log file like this:

17:01:30.930 [SERVER POOLED THREAD 1] INFO LightstreamerLogger.connections - Reused HTTP connection on "Lightstreamer HTTP Server" from 10.9.8.133:59985

Unfortunately also in this case we could not exactly figure out why the replies to some bind requests are not send to the client.

Some steps by which we can proceed with the investigation are:

- Please could you priovide us with the details about the client library in use in these tests (library type and version)?

- Repeat the test case wiht the extra log settings (in lightstreamer_log_conf.xml):
<logger name="LightstreamerLogger.requests" level="DEBUG"/>
<logger name="LightstreamerMonitorText" level="TRACE">

and if possible, a network sniff.

- Repeat the test case with a Lightstreamer server 6.0.x.

Thank you,
Giuseppe

abcdefg

Thanks for the clarification, that makes sense now.

I've added those extra log settings and attached the updated server log.

And also I have a Wireshark .pcapng which shows what the messages look like from the client side. I'm not sure that my title for this thread is correct, the bind_session.txt requests look like they did receive responses? If so then that in turn would mean that my Postman setup is not representative for some reason.

I'll admit I'm still trying to make heads-or-tails of the Wireshark output.

Please could you priovide us with the details about the client library in use in these tests (library type and version)?

I am using

pod 'Lightstreamer_OS_X_Client'

which is at version 2.0.0. I'm testing in an OSX project so as to keep things simple, but I had the same results when I was using Lightstreamer_iOS_Client, and the non-unified API as well.

Giuseppe Corti

Hi abcdefg,

First, please note the OS X Client API (Unified Version) version 2.0.x is compatible with Lightstreamer Server since version 6.0.1.
Most likely this is not the reason of the issue, since also happens with other libraries, but generally using a client lib not supported by the Lightstreamer server version in use exposes you to unpredictable behavior.

That said, the new log and the client side dump offer us some clues to explain the situation.
In fact we often find errors of this type, in the IP protocol:

"Header checksum: 0x0000 [incorrect, should be 0x6ec5 (maybe caused by "IP checksum offload"?)]"

These happen in the ack messages from the client to the server.
Now I can imagine at this point a scenario like this: the server does not receive the ack for his reply to create_session, for this reason the session remains stuck, and the following bind_session request does not receive any reply.

Regards,
Giuseppe

abcdefg

It is interesting however apparently a 0x0000 checksum might just be a quirk of Wireshark when capturing the packets.

https://ask.wireshark.org/questions/13577/header-checksum-error-incorrect-should-be

Giuseppe Corti

Could you take a sniff server side?

abcdefg

No problem, I've attached.

In the server logs, it looks like the header checksums are ok in both directions, so I'm guessing that this is just a recording quirk.

Giuseppe Corti

Hi abcdef,

You are right, these new captures show a correct situation from the network point of view.

However remains a strong suspicion that the sessions remain in some way stuck.

The best way to proceed the investigation is to reproduce the case with a server 6, and if the issue persists, I will provide you with specific instructions.

Regards,
Giuseppe

abcdefg

I think I might have isolated the problem.

Our servers run Lightstreamer 5.1.1. In 5.1.2 you added the <use_enriched_content_type> property which changes the Content-Type of messages from text/plain to text/enriched. Our servers don't have this flag set and Wireshark confirms that the Content-Type is text/plain not text/enriched.

From my testing, when the server sends text/plain messages, the Flash client will respond and connect, but the iOS client won't.

I have tested this with our 5.1.1 production server, with a 5.1.2 server, and with 6.0.3 server with <use_enriched_content_type>N</use_enriched_content_type> and it seems to be completely consistent and reproducible.

Giuseppe Corti

Hi abcdefg,

This is quite strange since the recent results do not agree with my previous considerations. In the log files I have checked so far the client lib always replied (with a bind request) to the server create_server reply.

Indeed as stated in the in-line comments of the <use_enriched_content_type> parameter, it is generally useful to work around possible buffering by network intermediaries (which is not your case).

That said, if the <use_enriched_content_type>Y</use_enriched_content_type> setting helps in your case you should consider the option of a Lightstreamer server upgrade.

Regards,
Giuseppe

abcdefg

Could you just confirm for me that you can NOT reproduce this behaviour

Server: Lightstreamer 6.0.3
Server setting: <use_enriched_content_type>N</use_enriched_content_type>

Client: Lightstreamer_OS_X_Client
Client code:

    override func viewDidLoad() {
        super.viewDidLoad()
        
        LSLightstreamerClient.setLoggerProvider(LSConsoleLoggerProvider())

        let client = LSLightstreamerClient(serverAddress: "https://....:443/", adapterSet: ".....")
        client.addDelegate(self)
        client.connect()
    }
    
    func client(client: LSLightstreamerClient, didChangeStatus status: String) {
        NSLog(status);
    }

Output:

    CONNECTING
    CONNECTED:STREAM-SENSING
    CONNECTED:HTTP-STREAMING
    CONNECTED:HTTP-POLLING
    DISCONNECTED:WILL-RETRY
    CONNECTING
    CONNECTED:HTTP-POLLING
    DISCONNECTED:WILL-RETRY
    CONNECTING
    CONNECTED:HTTP-POLLING
    DISCONNECTED:WILL-RETRY
    CONNECTING
    CONNECTED:HTTP-POLLING
    DISCONNECTED:WILL-RETRY
    CONNECTING
    ...

Giuseppe Corti

Hi abcdefg,

I can confirm that I can NOT reproduce the behaviour. My test conditions:
- Lightstreamer server 6.0.3 listening on localhost:8080 (no SSL)
- client OS X version 2.0.1
- <use_enriched_content_type>N</use_enriched_content_type>

But the results of your tests continue to confuse me; especially the server-side network sniff analyzed.

It may be that between the sniff and the Lightstreamer server there is some software component such as antivirus, firewall or similar?

Regards,
Giuseppe