How to Make Streaming Mode for Hello World Socket

gani

How to make lighstreamer run in streaming mode instead polling mode, specially in show case "Hello World Socket" like http://cometdaily.com/2008/07/29/hello-world-for-sockets-with-lightstreamer/.

I use Allegro version. Data Message sent by socket application, the data contain market value that highly need realtime condition.

If I use polling, it seem data is queing FIFO (First In First Out), this is good. But if web client application, lost connection or something, for a while, for example 10 second. Server sending expire data that late for 10 second.

Is there any solution for this problem ?

thx,

Gani

Dario Crivelli

If your client application asks for streaming and you see polling in action,
this must be because something in your network infrastructure is not compatible with streaming and the Stream-sense mechanism comes into play.
By looking at the Server log, we could confirm that.

Note that Allegro version has a built-in maximum frequency limit of one update per second on each item. Is it possible that you are confusing that with polling?

When the client loses the connection and reconnects, it is not bound to sending the messages occurred during the interruption. But see this FAQ for a broad discussion.
By the way, during the interruption can you see unsubscribe requests for the various items reaching your socket-based Remote Server, followed by new subscribe requests?

gani

First Thing, this is my Log File :

28-Apr-10 16:14:54,966 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Starting new session: Sd83b6032c3f1ee5eT1454966 from 150.70.84.75:51248
28-Apr-10 16:14:54,988 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 5     |Polling (129) to session: S9b9a938d00280abeT1153369 from 10.72.10.18:54739
28-Apr-10 16:14:56,912 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (541) to session: S93b24ff088425c25T1055241 from 10.72.10.18:61455
28-Apr-10 16:15:06,343 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 3     |Polling (174) to session: Sf05c8e951c804595T1059220 from 10.72.10.18:61455
28-Apr-10 16:15:06,357 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (172) to session: S6b366f6a2f193e52T1059246 from 10.72.10.18:61455
28-Apr-10 16:15:26,334 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 10    |Polling (150) to session: S9b9a938d00280abeT1153369 from 10.72.10.18:54739
28-Apr-10 16:15:27,535 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (570) to session: S93b24ff088425c25T1055241 from 10.72.10.18:54547
28-Apr-10 16:15:36,449 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (191) to session: S6b366f6a2f193e52T1059246 from 10.72.10.18:54739
28-Apr-10 16:15:36,449 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 4     |Polling (193) to session: Sf05c8e951c804595T1059220 from 10.72.10.18:61455
28-Apr-10 16:15:56,342 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (169) to session: S9b9a938d00280abeT1153369 from 10.72.10.18:52379
28-Apr-10 16:15:57,859 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (600) to session: S93b24ff088425c25T1055241 from 10.72.10.18:54739
28-Apr-10 16:16:08,350 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (210) to session: S6b366f6a2f193e52T1059246 from 10.72.10.18:52379
28-Apr-10 16:16:08,356 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (213) to session: Sf05c8e951c804595T1059220 from 10.72.10.18:61455
28-Apr-10 16:16:28,061 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 5     |Polling (630) to session: S93b24ff088425c25T1055241 from 10.72.10.18:54739
28-Apr-10 16:16:28,084 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (189) to session: S9b9a938d00280abeT1153369 from 10.72.10.18:54547
28-Apr-10 16:16:38,350 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 10    |Polling (228) to session: S6b366f6a2f193e52T1059246 from 10.72.10.18:54739
28-Apr-10 16:16:38,357 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (232) to session: Sf05c8e951c804595T1059220 from 10.72.10.18:54547
28-Apr-10 16:16:58,342 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (208) to session: S9b9a938d00280abeT1153369 from 10.72.10.18:61455
28-Apr-10 16:17:08,473 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (250) to session: Sf05c8e951c804595T1059220 from 10.72.10.18:61455
28-Apr-10 16:17:08,479 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (246) to session: S6b366f6a2f193e52T1059246 from 10.72.10.18:61455
28-Apr-10 16:17:10,876 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 5     |Polling (653) to session: S93b24ff088425c25T1055241 from 10.72.10.18:52379

Broadcast message is sending with heavy traffic case, but webclient open in one browser. You can see server create many Polling Session just for 1 web client, and THREAD never more than 10. My First question is, Is that normal with many polling session and THREAD never more than 10 ?

Some times client stop receiving data, between Polling Session, about 4 second to 50 second. Like I say before, this problem causing lately data, about delay periode.

Below is second log, when other computer try to open web client :

28-Apr-10 16:36:18,415 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 3     |Controlling session: Sc351dc22930e2db2T3616797 from 120.160.49.103:1760
28-Apr-10 16:36:18,872 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 3     |Polling (1033) to session: S869966960fa30e9T2817245 from 10.72.10.18:52429
28-Apr-10 16:36:19,982 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Serving request: /lightstreamer/control.html --> LS_session=Sc351dc22930e2db2T3616797&LS_window=1&LS_win_phase=4&LS_op=add&LS_req_phase=41&LS_mode1=MERGE&LS_id1=COWL-W%20PUDP%20BKSL%20MRAT%20SULI%20TLKM%20PYFA%20AGRO%20DAVO%20ASRI%20SMSM%20PTBA%20AGRO-W%20KIAS%20SRSN&LS_schema1=last_price%20time%20pct_change%20change%20bid_quantity%20bid%20ask%20ask_quantity%20min%20max%20ref_price%20open_price%20volume%20value%20stock_name%20item_status&LS_data_adapter1=QUOTE_ADAPTER_STOCK&LS_snapshot1=true&LS_requested_max_frequency1=1&LS_unique=2 from 120.160.49.103:13768
28-Apr-10 16:36:19,982 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 4     |Controlling session: Sc351dc22930e2db2T3616797 from 120.160.49.103:13768
28-Apr-10 16:36:44,923 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (359) to session: Sa0463bbb1e9407d1T2801333 from 10.72.10.18:53629
28-Apr-10 16:36:45,048 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (359) to session: Sa063a5d6c2123842T2802459 from 10.72.10.18:53663
28-Apr-10 16:36:45,345 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (348) to session: S4b5eac2d612dad6bT2801307 from 10.72.10.18:60098
28-Apr-10 16:36:46,867 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (31) to session: Sc351dc22930e2db2T3616797 from 120.160.49.103:49416
28-Apr-10 16:36:48,975 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (1102) to session: S869966960fa30e9T2817245 from 10.72.10.18:60098

Another problem is, client stop receiving data/delay, when another computer try to connect, in the logs show at serving request. This is really big problem, because this case is only 2 client, how about 500 or 1000 users, it will make all client delaying.

Currently, I used Linux Debian.

There is no Unsubscribe request, I think this is because I change the setting to "<session_timeout_millis>50000</session_timeout_millis>", default is 5000.

Any suggestion for this problem ?

Thx,

Gani

Dario Crivelli

The log demonstrates that polling mode is used, but a few seconds of log before the "Starting new session" message would be needed to confirm that Stream-sense was the cause for that.

The logged thread numbers are random, as they are from a thread pool. The pool size is 10 elements; they would be enough for serving many concurrent sessions.

You say that only one or two clients are running, but the sessions are 4 or 5 and all of them are active.
May you please clarify what clients are doing? Is it possible that the same "client" is running multiple instances of your application in the same browser?
In that case, it may exhaust the browser connection pool and cause the observed blocks and delays.

So, we need to clarify this aspect first.
By considering your first log snippet,
is it correct that 4 sessions were active
or did you expect that only 1 session would be active?
In other words, did it happen that you closed any sessions on the client side and noticed that the session was still logged by the Server?

gani

Thanks For Explanation Dario..

First, I need to confirm and explanation about view things, that maybe I didn't know..

Is it stream-sense mechanism automatically happen when something happen in network infrastructure, like you said in another thread like "intermediate node (an antivirus, for instance)" ?, How about other caused (client slow connection, tight and long queing message sending, allegro version 1 update item in one second,etc) ?

How Can I knowing sure in the logg, that some session is turn to stream-sense instead normal streaming mode ?

Is stream-sense start with

Starting new session: Sd83b6032c3f1ee5eT1454966 from 150.70.84.75:51248
28-Apr-10 16:14:54,988 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 5     |Polling (129) to session

And streaming mode start with

controlling session:

Code below is logged exactly after first client do subscribe :

.... Subscribe about 600 line item....
30-Apr-10 09:57:56,724 |WARN |rs.remote.data.RemoteDataProvider|#1 Notify Receiver         |Received Remote Server message: 
'0|S|stock_name|S|BOYPAS|S|last_price|S|410|S|time|S|09:58:47|S|pct_change|S|0|S|bid_quantity|S|939|S|bid|S|405|S|ask|S|410|S|ask_quantity|S|24|S|min|S|410|S|max|S|420|S|ref_price|S|415.16|S|open_price|S|410'
30-Apr-10 09:57:56,733 |WARN |rs.remote.data.RemoteDataProvider|#1 Notify Receiver         |Received Remote Server message: '0|S|stock_name|S|BOYULE|S|last_price|S|410|S|time|S|09:58:47|S|pct_change|S|0|S|bid_quantity|S|939|S|bid|S|405|S|ask|S|410|S|ask_quantity|S|24|S|min|S|410|S|max|S|420|S|ref_price|S|415.16|S|open_price|S|410'
30-Apr-10 09:57:56,754 |WARN |rs.remote.data.RemoteDataProvider|#1 Notify Receiver         |Received Remote Server message: '0|S|stock_name|S|BOZBRA|S|last_price|S|410|S|time|S|09:58:47|S|pct_change|S|0|S|bid_quantity|S|939|S|bid|S|405|S|ask|S|410|S|ask_quantity|S|24|S|min|S|410|S|max|S|420|S|ref_price|S|415.16|S|open_price|S|410'
30-Apr-10 09:58:15,945 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (88) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 09:58:46,095 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (301) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 09:59:16,143 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (520) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 09:59:46,240 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (697) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 09:59:50,334 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 7     |Serving request: /lightstreamer/STREAMING_IN_PROGRESS?LS_phase=8902&LS_domain=most.co.id&LS_client_version=4.3&LS_adapter=PROXY_HELLOWORLD_SOCKETS& from 216.104.15.130:24197
30-Apr-10 09:59:50,335 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 3     |Starting new session: S6eec6a4485d4188T5950335from 216.104.15.130:24197
30-Apr-10 10:00:16,344 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (821) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:00:46,351 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (996) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:01:16,474 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (1228) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:01:46,509 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (1377) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:02:16,553 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 6     |Polling (1613) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:02:46,646 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (1847) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:03:16,762 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 4     |Polling (2022) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:03:46,960 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (2207) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:04:17,303 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (2378) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:04:47,324 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (2539) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:05:18,525 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (2734) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:05:48,754 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (2937) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:06:18,761 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (3148) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:06:48,774 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (3364) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:07:18,924 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 1     |Polling (3543) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:07:48,969 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 9     |Polling (3757) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:08:19,079 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 4     |Polling (3971) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:08:50,579 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 5     |Polling (4116) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:09:20,793 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 7     |Polling (4302) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:09:50,835 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (4461) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:10:21,100 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (4668) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:10:53,349 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (4877) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:11:23,499 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 8     |Polling (5073) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:11:54,518 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (5278) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411
30-Apr-10 10:12:24,549 |INFO |LightstreamerLogger.requests     |SERVER POOLED THREAD 2     |Polling (5511) to session: S600f412f44bfebd0T5745861 from 10.72.10.18:56411

As you can see, there is Serving Request to 216.104.15.130, before "Starting new session" shown up.. We don't recognize this IP, but if we trace, this IP was located in Japan, and has something todo with "trend-micro", our network using "trend micro", but our network support in our office said that, that is not a problem..

I don't know what this IP doing, maybe it request with multiple instance..

How about your analysis about this ?

At the last reply, like you said, there is 4 session active, my computer in local network, other computer connected through internet provider and this IP 216.104.15.134, 150.70.84.203 (I don't know this either), we are not publishing yet..

I don't expect this application use by one client and 1 session, on the last reply,I thought I connected to lightstreamer with 1 session (actually 4 session like you said). Can Lightstreamer do block certain IP ?

It didn't it happen when I closed any sessions on the client side, but long delay happen in the middle of Polling to Session (Looks like random). The unknown IP that I mention keep try request to server (on the logs).

Our case problem is :
- We need lightstreamer keep in streaming mode and there is no delay
- Client didn't stop suddenly, because open two page browser contain same receiving data from lighstreamer. (Make client open just 1 instance even open two browser, or make 2 instance with no problem). CMIIW If one client open two web page, in one browser, is it automatically create 2 instance... how to make sure client just using 1 instance, we cannot preventing client open 2 browser with one page, isn't

Thank You,

Gani K

Dario Crivelli

About Stream-sense:

When a streaming connection doesn't yield an answer within a few seconds, a polling connection is tried.
Intermediate nodes with a buffering behavior will cause the streaming connection to delay, but slow clients and slow connections could also have the same effect.
Note that, in the latter cases, the subsequent polling attempt may timeout as well.
On the other hand, the Allegro version limitation does not prevent streaming.

Unfortunately, the Stream-sense is not directly notified in the Server log.
With the current Web Client Library, you have to manually recognize a

create_session
Starting new session
STREAMING_IN_PROGRESS
create_session
Starting new session
Polling () to session

pattern from the same client. We can do it for you, if you have any doubt, but we need to analyze a log that spans a few seconds.

About unknown clients:

Ok, so you are not the only user of the test system (I suppose that other people in your organization are testing the connectivity).

About blocks in the data flow:

Unfortunately, I didn't understand the remarks below:

- Client didn't stop suddenly, because open two page browser contain same receiving data from lighstreamer. (Make client open just 1 instance even open two browser, or make 2 instance with no problem). CMIIW If one client open two web page, in one browser, is it automatically create 2 instance... how to make sure client just using 1 instance, we cannot preventing client open 2 browser with one page, isn't

Do you mean that, indeed, the blocks are related with cases in which two instances of your application are running within the same browser instance?

Otherwise, you should try to reproduce the issue in the simplest possible way, by starting the Server and operating on your clients.
Then you should attach a full Server log, together with your notification of the time in which you noticed that the data flow got interrupted.
It you had the opportunity to test the system and reproduce the issues while no external users access the Server, that would make the log far more readable.