R Client for lightstreamer

Alessandro.Carioni

Hi Alonso,

4) It is not clear to me what is the exact meaning of the scale granularity (1MINUTE, 5MINUTE, …) you mention. I interpret is as the number of updates per unit of time, but maybe you should consult IG for the right meaning. If we stick with the above interpretation, then to limit frequency of updates you can leverage LS_requested_max_frequency. But keep in mind that LS_requested_max_frequency specifies the number of updates per second. So if you want just one update per minute, you must set the parameter to 0.017 (i.e. 1/60).

5) TLCP protocol has a more uniform and compact syntax, but it is only supported by server version 6.1 or above. So IG should update the server in order that the clients may use TLCP.

6) A local server is surely useful while you are learning the technology. You can install the Stock-List Demo, which simulates stock quotes, on your local server, and then connect with your client.
The server 6.1 can talk TLCP and the old protocol, so you can experiment with both of them.

Regards
Alessandro Carioni

Alessandro Alinone

5) To directly answer your question, there are no significant disadvantages to use the old protocol.

toolbox

Thanks Alessandro.

4) The scale granularity or resolution is by IG defined to as an item, with following syntax: CHART:{epic}:{scale}, whereof {scale} is the item. See IG streaming API reference: https://labs.ig.com/streaming-api-reference

It is basically the time interval between 2 candlesticks. From a timestamp perspective it is the timewise startpoint of a candlestick, but from a value perspective it is the LastTradePrice meaning the OHLC values as a screenshot exactly before next (new timestamp arrives). This means that taken account the updates in Lightstreamer one would need to take the timestamp + all updates until next timestamp arrives. Unless IGs candlestick solution would work because then they should send only the LastTradePrice values, which currently does not work for me.

I have tested setting the [LS_requested_max_frequency] to 0.017, for the stream to go down to 1 minute updates, but without success. I thought of change the [LS_mode] and try Command, Distinct or RAW, but seems that the IG [Chart subscriptions - Consolidated (candle) data] is supposed to be set with [LS_mode] Merge.

toolbox

4) Here is an example of scale with 5 minutes resolution, see bold text. To simplify reading I have transformed UTM format to POSIXct,and left the stock values same for all the 3 lines.

1,1|2017-06-19 09:40:00 CEST|1652.05|1652.18|1649.93|1650.18|0
1,1|2017-06-19 09:45:00 CEST|1652.05|1652.18|1649.93|1650.18|0
1,1|2017-06-19 09:50:00 CEST|1652.05|1652.18|1649.93|1650.18|0

Dario Crivelli

Hi, If I understand correctly, the Consolidated candle item provides the state of the current candle.
So, upon each new contract that causes the current candle to change, the whole candle coordinates are sent (as I infer from the traffic shown on the first post).
Then, when a candle is completed (according to the time-slot implied in the item name), the updates begin to refer to the next candle, as identified by the increased timestamp field.

Do you agree on this interpretation?
If so, please clarify what is the alternative behavior that you would like to achieve.
Perhaps you would like to get candle updates only for consolidated candles, while you are not interested in temporary candle states?

toolbox

Clarifications of goals:
8) Move from REST API to Streaming API due to REST API loosing 30% datalines (removing lines with NA).
9) Being in Streaming API secure candlestick OHLC data. Basically data when each candlestick is closed and do not update more.
10) In streaming API secure stability if data with 1 minute OHLC data.
11) If not getting all candlestick lines in streaming API, use the stream to construct the candlestick.

To summarize: I am using the lightstreamer technique to significantly improve the data quality and stability. However, my tool summarize all the lightstream updates within a candlestick resolution, eg. 5 minutes and stores the closure status of the candlestick.

Dario Crivelli

Ok, so point 9) seems to match my assumption in the previous post.
But operating on the client side to reduce the updates of candlestick data to only the ones related to candlestick close seems impossible to me. For sure, you cannot achieve that by leveraging LS_requested_max_frequency.
I think that your only possibility is to:

subscribe normally and read all updates,
keep the state of the current candlestick by integrating the updates,
however, whenever an update carries a new timestamp value, save the previous state as a full candlestick and clear the state to prepare for the next candlestick (but the incoming update should itself take care of the clearing).

I cannot understand point 10), but it seems just a reinforcement of point 9).

I also can't understand point 11). So far, the problem seems that, in streaming API you get more lines than needed, not fewer.
Perhaps you mean that if you cannot satisfy point 9) you have to give up subscribing to candlesticks and subscribe to ticks only and use the ticks to build the candlesticks yourself on the client side. Guidance on this is obviously out of our scope as Lightstreamer support.

toolbox

Yes, I understand that the guidance is limited, but I think it might be valid to keep the thread open so I can potentially update it. I assume that it is of benefit for Lightstreamer as a product that the users creates clients that are not officially released nor supported to empower and grow the usage of Lightstreamer.

Grine1973

Thats right , there are no significant disadvantages to use the old protocol.

Alessandro Alinone

@toolbox: Yes, we agree that it's good to keep this thread open, as this discussion is quite interesting to several Lightstreamer users.

Thanks,
Alessandro

toolbox

With some backwards-engineering I have now a working solution. The IG Streaming API shows now exactly the same 5 minutes OHLC(+Volume) data as the IG REST API. It will probably need major tuning to be optimized. I will summarize my findings and solution and post it here. Probably in generic terms to avoid confusion.

Alessandro Alinone

That's great news, thanks!

toolbox

Below summary is only valid based on following environment:
Server:
- Lightstreamer server positioned at IG premises.
- Server version less then 6.1
Client:
- old text protocol (not TLCP)
- OS - Linux
- Programming environment - "R".

################################################
# Summary
################################################
I am working on setting up an "automated trading system" completely
in the programming environment "R". As first step I created a download
environment based on REST API. During the test phase it was obviously that
the robustness was not acceptable, missing 5%-30% downloaded OHLC lines.
The resolution was set to 5 minutes candlesticks. The 5 min resolution means that the REST API
sends out "historical data" that is not meant change dynamically.

The reason for the missing lines was that IG sends out values "NULL" in their datalines without
any explanation of how to handle those "NULL" values. Due to that my client is
programmed to delete any line that contains minimum one "NULL".

To continue with my project I initiated the task to replace REST API with Streaming API.
(same bank though). To benchmark increase in quality of downloaded lines, the aim was
to set streaming API to 5 minutes candlesticks. You should be able to subscribe on "5 min candlestick" but I encountered 2 problems:

1) The streaming API from IG sends out all ticks, despite if you subscribe on "5 min candlesticks",
you do get the timestamps per each 5 minutes though.

2) Since you need to work with the stream as source, you need to let each line,
accumulate the changes until you "freeze" the line to be historical and will never change.
This is of course not valid for the last timestamp (since it will keep changing).

...further on tried to set the clients "LS_requested_max_frequency" in order to reduce the
stream intensity but the change did not have any impact on the stream at all.

######################################################################
# The basics of building the client in "R", without technical details.
######################################################################
1) Open up a stream connection using create_session.txt, that continuously saves to file (e.g. stream1.txt)
2) Request for a subscription using control.txt
3) Poll data from stream connection using bind_session.txt
4) Script to transform the stream1.txt to 5 minutes OHLC data.
5) Compare result between Streaming API <-> REST API.
6) Develop function to steer the stream (start, stop, etc).

######################################################################
# Result and needed improvements
######################################################################
I have managed to reduce missing OHLC data lines from REST API (5% - 30%) to Streaming API (roughly <5%). These <5% includes potential bottlenecks, such as programming language, OS, network, scheduler, modeling, architecture, etc. Benchmarking between REST API and Streaming API is currently not possible since Streaming API is "self-healing". If a line is missing, next production of line will include all historical lines, resulting in total success of 100%. From a "automated trading system" -persective, if the line is not there at a specific time, the loss for that line is 100%, and next decision is to be taken next successful line. To stresstest the system i now run 1 minute resolution.

The biggest back-draws right now is:
1) If the stream stops (hangs or breaks), the system stops to have updated download data.
2) Currently the transformation function uses a screen-shot of complete accumulated data,
at 5 p.m. it screen-shots the whole day.

Alessandro Alinone

Great, thanks for the update!