R Client for lightstreamer

toolbox

Hi. I have setup an R client for lightstreamer towards IG,
to subscribe on 1 minute OHLCV (Open, High, Close, Volume) data.

Background to move into lightstreamer is that IG:s REST API
sends to many NULL in their content, resulting in that my system removes 30% OHLC lines.

I have a client lightstreaming working that looks as below.
Since this is my first time to setup a client following the TLCP,
i am not familiar with how the stream exactly should look like.
I thought that if I subscribe on 1 minutes data I would get PROBE-lines and the OHLCV data,
but I get also all the data changes in between.

My parameter settings:

LS_session=Sa9447dbaae6d3456T2643148
LS_op=add
LS_id=CHART:IX.D.OMX.IFM.IP:1MINUTE
LS_schema=UTM + BID_OPEN + BID_HIGH + BID_LOW + BID_CLOSE + CONS_END
LS_mode=MERGE
LS_table=1
LS_reqId=1
LS_subId=1
LS_group=item1
LS_data_adapterQUOTE ADAPTER

Question1) Could you please confirm if a subscription of 1 minutes data should have data in between the 1 minutes screenshots?

Question2) Above each trading-data line there is a character/number combination.
I do not find in the documentation what they mean. Could you please clarify or refer where
to find explanation.

Question3) Is it correct to assume that an empty filed between two "pipe" characters should be
interpreted as that value is unchanged thus if having a dataframe one should populate the unchanged value with previous changed value.

#### Extraction from the data stream

<< C
<< 1,1||||||1
<<

<< 7
<< PROBE
<<

<< 1D
<< 1,1|1497276480000|$|$|$|$|0
<<

<< 27
<< 1,1||1639.75|1639.75|1639.63|1639.63|
<<

<< 19
<< 1,1|||1639.88||1639.88|
<<

<< 19
<< 1,1|||1640.13||1640.13|
<<

<< 19
<< 1,1|||1640.38||1640.25|
<<

<< 12
<< 1,1|||||1640.38|
<<

<< 12
<< 1,1|||||1640.13|
<<

###############################

My final goal is to have the data as below, in a time-series format. (called xts in R).
I am aware that I might be need to decode the pipe-separated list-of-values.

[open], [high], [low], [close], [volume]
line1-1min resolution - [streamvalue - open], [streamvalue - high], [streamvalue - low], [streamvalue - close], [streamvalue - volume]
line2-1min resolution - [streamvalue - open], [streamvalue - high], [streamvalue - low], [streamvalue - close], [streamvalue - volume],
etc...etc....

/ Regards, Alonso

Alessandro.Carioni

Hi Alonso,

1) The update maximum frequency is set by the server according to the data feed frequency. So if you notice a difference from the expected frequency, you should contact the server maintainers.
Alternately you could explicitly ask the server for a lower frequency adding the parameter LS_requested_max_frequencyto your subscription requests (see the section 4.3.1 Subscription Control Connections of the document linked below).

2) Observing the excerpted log, I warn you that
a) IG server is not using the TLCP protocol but the old text protocol. The document explaining the old protocol is not listed in the Docs page anymore,but you could retrieve a copy looking inside the distributions of the servers below version 6.1 (for example here).
The complete syntax is explained in the document Network Protocol Tutorial underDOCS-SDKs/sdk_client_generic/doc .

b) the hex numbers as C, 7,1D, 27 above the protocol messages are the sizes of the HTTP 1.1 chunks and they are not part of the text protocol.

3) The “pipe format” is explained in the section 4.5.1 Update messages of the above document.

Regards
Alessandro Carioni

Alessandro Alinone

The documentation of the right version of the network protocol used by IG is linked on IG's site: https://labs.ig.com/lightstreamer-downloads

The link sends here.

toolbox

Thanks Alessandro :Smile_Ab: . It looks like I will be able to finalize the R client for lightstreamer with
some more hard work. I still need to transform the stream into a OHLCV data.frame for my
automated trading system to use that as a base for analysis.

4) Regarding avoiding tick-data and force client to show 1MINUTE or 5MINUTE
I have now tested with IG:s Streaming Companion and despite if putting the {scale} to either [1MINUTE, 5MINUTE or HOUR], I always see the tick changes. I did change the setting, in my client, of [LS_requested_max_frequency] to both 60 seconds and changed it later for 300 seconds. There is not really any change in regards to visible tick-data in the stream. Adjusting the above parameter does not remove the stream of tick-data. The parameter settings seem otherwise fine since it looks like [BID_OPEN, BID_HIGH, etc] is allocated to candlestick data, having the {scale} after the {epic}. I assume I need to contact IG for clarifications of the [LS_requested_max_frequency].

5) Regarding using the old text protocol instead of TLCP
Since I come from using IG API REST and loosing up to 30% OHLC lines, it is still a great improvement for my client solution using lightstreamer. Are there significant disadvantages to use the old protocol? Do you recommend that I request IG to upgrade to TLCP?

6) Regarding setting up a lightstreamer to use the test scenarios.
Would you recommend for me to setup the lightstreamer server to understand
more the traffic flow? Since I am using HTTPS i cannot always sniff the TCP/HTTPS stream with an dedicated tools such as wireshark.

/ Regards, Alonso

Alessandro.Carioni

Hi Alonso,

4) It is not clear to me what is the exact meaning of the scale granularity (1MINUTE, 5MINUTE, …) you mention. I interpret is as the number of updates per unit of time, but maybe you should consult IG for the right meaning. If we stick with the above interpretation, then to limit frequency of updates you can leverage LS_requested_max_frequency. But keep in mind that LS_requested_max_frequency specifies the number of updates per second. So if you want just one update per minute, you must set the parameter to 0.017 (i.e. 1/60).

5) TLCP protocol has a more uniform and compact syntax, but it is only supported by server version 6.1 or above. So IG should update the server in order that the clients may use TLCP.

6) A local server is surely useful while you are learning the technology. You can install the Stock-List Demo, which simulates stock quotes, on your local server, and then connect with your client.
The server 6.1 can talk TLCP and the old protocol, so you can experiment with both of them.

Regards
Alessandro Carioni

Alessandro Alinone

5) To directly answer your question, there are no significant disadvantages to use the old protocol.

toolbox

Thanks Alessandro.

4) The scale granularity or resolution is by IG defined to as an item, with following syntax: CHART:{epic}:{scale}, whereof {scale} is the item. See IG streaming API reference: https://labs.ig.com/streaming-api-reference

It is basically the time interval between 2 candlesticks. From a timestamp perspective it is the timewise startpoint of a candlestick, but from a value perspective it is the LastTradePrice meaning the OHLC values as a screenshot exactly before next (new timestamp arrives). This means that taken account the updates in Lightstreamer one would need to take the timestamp + all updates until next timestamp arrives. Unless IGs candlestick solution would work because then they should send only the LastTradePrice values, which currently does not work for me.

I have tested setting the [LS_requested_max_frequency] to 0.017, for the stream to go down to 1 minute updates, but without success. I thought of change the [LS_mode] and try Command, Distinct or RAW, but seems that the IG [Chart subscriptions - Consolidated (candle) data] is supposed to be set with [LS_mode] Merge.

toolbox

4) Here is an example of scale with 5 minutes resolution, see bold text. To simplify reading I have transformed UTM format to POSIXct,and left the stock values same for all the 3 lines.

1,1|2017-06-19 09:40:00 CEST|1652.05|1652.18|1649.93|1650.18|0
1,1|2017-06-19 09:45:00 CEST|1652.05|1652.18|1649.93|1650.18|0
1,1|2017-06-19 09:50:00 CEST|1652.05|1652.18|1649.93|1650.18|0

Dario Crivelli

Hi, If I understand correctly, the Consolidated candle item provides the state of the current candle.
So, upon each new contract that causes the current candle to change, the whole candle coordinates are sent (as I infer from the traffic shown on the first post).
Then, when a candle is completed (according to the time-slot implied in the item name), the updates begin to refer to the next candle, as identified by the increased timestamp field.

Do you agree on this interpretation?
If so, please clarify what is the alternative behavior that you would like to achieve.
Perhaps you would like to get candle updates only for consolidated candles, while you are not interested in temporary candle states?

toolbox

Clarifications of goals:
8) Move from REST API to Streaming API due to REST API loosing 30% datalines (removing lines with NA).
9) Being in Streaming API secure candlestick OHLC data. Basically data when each candlestick is closed and do not update more.
10) In streaming API secure stability if data with 1 minute OHLC data.
11) If not getting all candlestick lines in streaming API, use the stream to construct the candlestick.

To summarize: I am using the lightstreamer technique to significantly improve the data quality and stability. However, my tool summarize all the lightstream updates within a candlestick resolution, eg. 5 minutes and stores the closure status of the candlestick.

Dario Crivelli

Ok, so point 9) seems to match my assumption in the previous post.
But operating on the client side to reduce the updates of candlestick data to only the ones related to candlestick close seems impossible to me. For sure, you cannot achieve that by leveraging LS_requested_max_frequency.
I think that your only possibility is to:

subscribe normally and read all updates,
keep the state of the current candlestick by integrating the updates,
however, whenever an update carries a new timestamp value, save the previous state as a full candlestick and clear the state to prepare for the next candlestick (but the incoming update should itself take care of the clearing).

I cannot understand point 10), but it seems just a reinforcement of point 9).

I also can't understand point 11). So far, the problem seems that, in streaming API you get more lines than needed, not fewer.
Perhaps you mean that if you cannot satisfy point 9) you have to give up subscribing to candlesticks and subscribe to ticks only and use the ticks to build the candlesticks yourself on the client side. Guidance on this is obviously out of our scope as Lightstreamer support.

toolbox

Yes, I understand that the guidance is limited, but I think it might be valid to keep the thread open so I can potentially update it. I assume that it is of benefit for Lightstreamer as a product that the users creates clients that are not officially released nor supported to empower and grow the usage of Lightstreamer.

Grine1973

Thats right , there are no significant disadvantages to use the old protocol.

Alessandro Alinone

@toolbox: Yes, we agree that it's good to keep this thread open, as this discussion is quite interesting to several Lightstreamer users.

Thanks,
Alessandro

toolbox

With some backwards-engineering I have now a working solution. The IG Streaming API shows now exactly the same 5 minutes OHLC(+Volume) data as the IG REST API. It will probably need major tuning to be optimized. I will summarize my findings and solution and post it here. Probably in generic terms to avoid confusion.

Alessandro Alinone

That's great news, thanks!

toolbox

Below summary is only valid based on following environment:
Server:
- Lightstreamer server positioned at IG premises.
- Server version less then 6.1
Client:
- old text protocol (not TLCP)
- OS - Linux
- Programming environment - "R".

################################################
# Summary
################################################
I am working on setting up an "automated trading system" completely
in the programming environment "R". As first step I created a download
environment based on REST API. During the test phase it was obviously that
the robustness was not acceptable, missing 5%-30% downloaded OHLC lines.
The resolution was set to 5 minutes candlesticks. The 5 min resolution means that the REST API
sends out "historical data" that is not meant change dynamically.

The reason for the missing lines was that IG sends out values "NULL" in their datalines without
any explanation of how to handle those "NULL" values. Due to that my client is
programmed to delete any line that contains minimum one "NULL".

To continue with my project I initiated the task to replace REST API with Streaming API.
(same bank though). To benchmark increase in quality of downloaded lines, the aim was
to set streaming API to 5 minutes candlesticks. You should be able to subscribe on "5 min candlestick" but I encountered 2 problems:

1) The streaming API from IG sends out all ticks, despite if you subscribe on "5 min candlesticks",
you do get the timestamps per each 5 minutes though.

2) Since you need to work with the stream as source, you need to let each line,
accumulate the changes until you "freeze" the line to be historical and will never change.
This is of course not valid for the last timestamp (since it will keep changing).

...further on tried to set the clients "LS_requested_max_frequency" in order to reduce the
stream intensity but the change did not have any impact on the stream at all.

######################################################################
# The basics of building the client in "R", without technical details.
######################################################################
1) Open up a stream connection using create_session.txt, that continuously saves to file (e.g. stream1.txt)
2) Request for a subscription using control.txt
3) Poll data from stream connection using bind_session.txt
4) Script to transform the stream1.txt to 5 minutes OHLC data.
5) Compare result between Streaming API <-> REST API.
6) Develop function to steer the stream (start, stop, etc).

######################################################################
# Result and needed improvements
######################################################################
I have managed to reduce missing OHLC data lines from REST API (5% - 30%) to Streaming API (roughly <5%). These <5% includes potential bottlenecks, such as programming language, OS, network, scheduler, modeling, architecture, etc. Benchmarking between REST API and Streaming API is currently not possible since Streaming API is "self-healing". If a line is missing, next production of line will include all historical lines, resulting in total success of 100%. From a "automated trading system" -persective, if the line is not there at a specific time, the loss for that line is 100%, and next decision is to be taken next successful line. To stresstest the system i now run 1 minute resolution.

The biggest back-draws right now is:
1) If the stream stops (hangs or breaks), the system stops to have updated download data.
2) Currently the transformation function uses a screen-shot of complete accumulated data,
at 5 p.m. it screen-shots the whole day.

Alessandro Alinone

Great, thanks for the update!