Subscription modes for processing order book updates

heg3

Hi,
I am looking for a solution to implement a subscription for an order book which is basically a List of Orders where an Order has multiple fields that can be updated and among them one field is a List<Execution> where an Execution has also multiple fields (like amount, executionPrice, etc).
From my understanding, I would need:

a subscription with COMMAND mode for orders item (so that I can be notified when a new order is added/deleted or any field changes). This can be also handled with a 2 level push approach. But is it possible to use the MERGE mode for updates even if there is a list of orders? (assuming that a new order cannot be added and an existing one cannot be deleted)
for executions field I would need a separate subscription either with DISTINCT mode (if execution fields are immutable and one order execution cannot be deleted) or COMMAND mode (in case executions are mutable)

Is it possible to achieve the behaviour above without adding any logic that subscribes dynamically for executions once a new orders is received on the consumer side?

Also, any links to any documentation containing details about the subscription modes that are not mentioned in the General Concepts document is much appreciated.

Thank you.

Giuseppe Corti

heg3
Unfortunately, in your scenario, the two-level COMMAND mode isn't applicable. This is because the second-level item, which in your case would contain the list of executions, cannot be used with the MERGE mode, the only mode available for second-level items.

I see two potential alternatives:

Application-level management of the second level in DISTINCT mode: the first option is to handle the second level at the application level using the DISTINCT mode (which seems to be the most suitable choice). This means that whenever your application receives an update related to the item in COMMAND mode with an ADD or DELETE command, it should subscribe to or unsubscribe from the executions item for the relevant key accordingly.
Using single-level COMMAND mode with a JSON string for executions field: In this case, you would include a JSON string representing the array of executions within a field of the single-level item. To maintain optimization and prevent resending all executions whenever a new one is added, you can mark this field for JSON Patch optimization. This is something you need to configure in your data adapter before sending the first update for the item: https://lightstreamer.com/sdks/ls-adapter-inprocess/8.0.0/api/com/lightstreamer/interfaces/data/ItemEventListener.html#declareFieldDiffOrder(java.lang.String,java.util.Map)

Please let me know what you think of these alternatives and if you'd like to delve deeper into any specific aspect of either option.

Giuseppe

heg3

Hi Giuseppe,

Thanks for your reply.
I was thinking about the solution to have a String field on Order carrying all execution data in the JSON format. I was afraid of the duplicated data I have to sent to the consumer (especially, if I have many fields for an Execution and multiples Executions for an Order), but I wasn't aware about the optimization you suggested. So thanks a lot for the hint!

heg3

Hi again Giuseppe,

I was trying to understand how your second suggestion (Using single-level COMMAND mode with a JSON string for executions field) would work and if it suits my needs. I read the documentation and from what I understand, this optimisation is just between the Lightstreamer server and the client. From the adapter, I still have to send the previous executions when a new one arrives so that the Server can compute the diff, right?

In this case, I would avoid this solution since my adapter should be stateless (when a new execution is received, the old ones are no longer present). I can keep their identifiers (in order to be able to identify an update), but I cannot store the whole execution data.

Considering this, is there any support in LightStreamer for a COMMAND mode at item's field level? Or do I have it to implement it on my own? I was thinking to something like:

Order has various fields like id, amount, price, execution
on execution I send to consumer a JSON like {command=ADD/UPDATE/DELETE, execution={id=...; execPrice=...; etc}} and the client should decide what to do with the Execution related data by looking at the command value

Another approach would be to extract the command at the Order item level so that I will have values like:

ADD_ORDER, DELETE_ORDER, UPDATE_ORDER, ADD_EXECUTION, DELETE_EXECUTION, UPDATE_EXECUTION (and for the latest the client should check the JSON received on the execution field).

What are your thoughts about this?

Giuseppe Corti

heg3 Yes, my suggestion regarding a single JSON field implies that every time a new execution arrives, the adapter sends the Lightstreamer server the JSON string representing all executions – the new one added to all the existing ones. The Lightstreamer server will then apply JSON Patch optimization and send the client only the minimal necessary changes.
If, for your implementation, maintaining the complete list of executions for each order of all subscribed order books is not feasible, then indeed, this suggestion is not applicable.

No, there is no support in Lightstreamer for a COMMAND mode at field level, but I find the first alternative you proposed interesting.
Essentially, this involves the adapter itself sending a sort of JSON patch to the client that should process the information and re-aggregate the complete list of executions.
I would just confirm that even in this case, you could consider declaring the field as subject to JSON Patch optimization. I don't anticipate significant optimizations, as the fields of each execution will presumably all be different.
However, keep in mind that for each individual update, the server will understand whether it is more efficient to send the complete JSON to the client versus the patch obtained by applying the optimization.
If, on the other hand, you believe, knowing the nature of the data, that applying optimization is just a waste of computational resources, you can forgot it.

Giuseppe

heg3

Understood, thanks a lot for the clarification.