Direct push to Lightstreamer from a C daemon.

Mlobo

Hi;

I am new to Lightstreamer and I'm in my very first attempts to work with it. I just subscribed to this forum so please bear with my probably lame questions.

I managed to put the HelloWorld and MarketDepthDemo to work on my server.

In my Job, everything is in DBF/NTX (yeah! Tell me about it!). I already wrote a daemon in C that gathers data from those files. I was wondering if I could extend my daemon to push this gathered data directly to Lightstreamer.

I tried to "relate" the java code to a C equivalent but I'm an old dog and I just couldn't do it.

So if there is a way, what are the main ordered steps to accomplish this?

Thanks for any advice.

Gianluca Finocchiaro

Hi Mlobo,

to accomplish your goal you have to set up a Remote Adapter Infrastructure, following the principles depicted in this document.

Basically, you have to create one or more processes which open a TCP communication with Lightstreamer Server and start the interaction by exchanging commands as required by the ARI protocol.

It is possible to implement a Remote Adapter for any language that supports socket programming, and C (in any its implementation) obviously does. Here the full list of Remote Adapter examples in different languages and platforms.

Please get back to us for any doubt.
Gianluca

Mlobo

Thanks Gianluca;

I'll give it a shot and will post my results.

Mario Lobo

Alessandro Alinone

Hi Mario,

The "Hello World" tutorial on pushing data to Lightstreamer server via TCP sockets is this: https://github.com/Weswit/Lightstreamer-example-HelloWorld-adapter-socket

Cheers
Alessandro

Mlobo

Here i what I did.

1) I used the adapters.xml bellow:

<?xml version="1.0"?>

<adapters_conf id="PROXY_HELLOWORLD_SOCKETS">

<metadata_provider>
<adapter_class>com.lightstreamer.adapters.metadata.LiteralBasedProvider</adapter_class>
</metadata_provider>

<data_provider>
<adapter_class>com.lightstreamer.adapters.remote.data.NetworkedDataProvider</adapter_class>
<param name="request_reply_port">7001</param>
<param name="notify_port">7002</param>
<param name="timeout">36000000</param>
</data_provider>

</adapters_conf>

2) Copied everything to the proper directories

[/usr/Lightstreamer/bin/unix-like]>ls -l ../../adapters/HelloWorldSockets/
total 8
-rw-r--r-- 1 root wheel 503 Sep 14 09:10 adapters.xml
drwxr-xr-x 2 root wheel 512 Sep 14 09:12 lib

[/usr/Lightstreamer/bin/unix-like]>ls -l ../../adapters/HelloWorldSockets/lib/
total 120
-rw-r--r-- 1 root wheel 120535 Sep 14 09:12 ls-proxy-adapters.jar

When I start lightstreamer, I get this:

[/usr/Lightstreamer/bin/unix-like]>./start.sh
Java environment:
JAVA_HOME = "/usr/local/openjdk8"
JAVA_OPTS = "-server -XX:MaxGCPauseMillis=1000"

Setting file descriptor limit to 117270
Configured file descriptors, soft limit: 117270
Configured file descriptors, hard limit: 117270

Lightstreamer Server directory:
LS_HOME = "/usr/Lightstreamer"

Lightstreamer Server main configuration file:
LS_CONFIG = "/usr/Lightstreamer/conf/lightstreamer_conf.xml"

Starting Lightstreamer Server...
Please check logs for detailed information.
15.Sep.15 16:40:51,551 < INFO> Lightstreamer Server 6.0.1 build 1780
15.Sep.15 16:40:51,591 < INFO> Lightstreamer Server starting in Moderato edition.
15.Sep.15 16:40:51,753 < WARN> Only minimal JMX management support is available with the current license.
15.Sep.15 16:40:51,858 < INFO> Started RMI server for JMX on port 8888.
15.Sep.15 16:40:52,110 < INFO> Bound RMI Connector for JMX on port 8888 (communication on port 8888).
15.Sep.15 16:40:52,211 < INFO> Bound RMI Connector for Platform mbeans on port 8888 (communication on port 8888).
15.Sep.15 16:40:52,221 < INFO> SERVER pool size set by default at 10.
15.Sep.15 16:40:52,348 < INFO> Pump pool size set by default at 4.
15.Sep.15 16:40:52,466 < INFO> data_provider element without name attribute; using DEFAULT as the default name.
15.Sep.15 16:40:52,500 < INFO> data_provider element without name attribute; using DEFAULT as the default name.
15.Sep.15 16:40:52,503 < INFO> Loading Metadata Adapter for Adapter Set WELCOME
15.Sep.15 16:40:52,503 < INFO> Loading Metadata Adapter for Adapter Set MARKETDEPTH
15.Sep.15 16:40:52,504 < INFO> Loading Metadata Adapter for Adapter Set PROXY_HELLOWORLD_SOCKETS
15.Sep.15 16:40:52,521 < INFO> Finished loading Metadata Adapter for Adapter Set PROXY_HELLOWORLD_SOCKETS
15.Sep.15 16:40:52,543 < INFO> Loading Data Adapter PROXY_HELLOWORLD_SOCKETS.DEFAULT
15.Sep.15 16:40:52,563 <ERROR> Can't find data adapter class.
java.lang.ClassNotFoundException: com.lightstreamer.adapters.remote.data.NetworkedDataProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_45]
at com.lightstreamer.g.d.an.loadClass(an.java) ~[na:na]
at com.lightstreamer.init.b.a(b.java) [ls-bootstrap.jar:na]
at com.lightstreamer.g.d.f.a(f.java) [lightstreamer.jar:na]
at com.lightstreamer.g.d.f.a(f.java) [lightstreamer.jar:na]
at com.lightstreamer.g.d.ak.b(ak.java) [lightstreamer.jar:na]
at com.lightstreamer.o.b.al.run(al.java) [lightstreamer.jar:na]
15.Sep.15 16:40:52,564 <ERROR> Error in task Init adapter set PROXY_HELLOWORLD_SOCKETS: com.lightstreamer.o.b.ae: Error in task Init for PROXY_HELLOWORLD_SOCKETS.DEFAULT.
15.Sep.15 16:40:52,567 <ERROR> Error while loading the Adapters: LoadingException: Error in Adapter loading.
15.Sep.15 16:40:52,567 <ERROR> Error while loading the Adapters: Error in Adapter loading.
15.Sep.15 16:40:52,567 <FATAL> Adapters initialization failed.

What should I do?

Thanks,

Mario Lobo

Mlobo

Alessandro;

That's exactly the demo I'm trying to run.

Giuseppe Corti

Hi Mario,

It seems you are using an Adapters configuration of the old Lightstreamer server 5.x but no longer supported in the new version 6.
Please, try with an adapters.xml like this:

[SYNTAX=XML]<?xml version="1.0"?>

<adapters_conf id="PROXY_HELLOWORLD_SOCKETS">

<metadata_provider>
<adapter_class>com.lightstreamer.adapters.metadata.LiteralBasedProvider</adapter_class>
</metadata_provider>

<data_provider>
<adapter_class>PROXY_FOR_REMOTE_ADAPTER</adapter_class>
<classloader>log-enabled</classloader>
<param name="request_reply_port">7001</param>
<param name="notify_port">7002</param>
<param name="timeout">36000000</param>
</data_provider>

</adapters_conf>
[/SYNTAX]
Indeed, the issue was that the old configuration implies the use of a common class loader shared with the server, which is not supported anymore for Proxy Adapters.

Please let us know if you need any further clarifications.

Regards,
Giuseppe

Giuseppe Corti

Please note that in my previous post, inside the xml snippet, remained some fake spaces, which I was not able to remove.

Mlobo

That did it!
grazie mille, Giuseppe!

Now I can begin experimenting!

Mario Lobo

Mlobo

Giuseppe;

One last off-topic question. The plan is to use Lightstreamer on a site that won't have more than 80 simultaneous accesses at a given time. And it will stay at that.

Will the moderato edition be suited for this?

Mlobo

Another problem came up.

Lightstreamer starts fine but it no longer opens a listening port on 8080 (http).

Here is the startup:

[/usr/Lightstreamer/bin/unix-like]>./start.sh
Java environment:
JAVA_HOME = "/usr/local/openjdk8"
JAVA_OPTS = "-server -XX:MaxGCPauseMillis=1000"

Setting file descriptor limit to 117270
Configured file descriptors, soft limit: 117270
Configured file descriptors, hard limit: 117270

Lightstreamer Server directory:
LS_HOME = "/usr/Lightstreamer"

Lightstreamer Server main configuration file:
LS_CONFIG = "/usr/Lightstreamer/conf/lightstreamer_conf.xml"

Starting Lightstreamer Server...
Please check logs for detailed information.
16.Sep.15 14:26:21,776 < INFO> Lightstreamer Server 6.0.1 build 1780
16.Sep.15 14:26:21,819 < INFO> Lightstreamer Server starting in Moderato edition.
16.Sep.15 14:26:21,907 < WARN> Only minimal JMX management support is available with the current license.
16.Sep.15 14:26:21,957 < INFO> Started RMI server for JMX on port 8888.
16.Sep.15 14:26:22,114 < INFO> Bound RMI Connector for JMX on port 8888 (communication on port 8888).
16.Sep.15 14:26:22,178 < INFO> Bound RMI Connector for Platform mbeans on port 8888 (communication on port 8888).
16.Sep.15 14:26:22,191 < INFO> SERVER pool size set by default at 10.
16.Sep.15 14:26:22,308 < INFO> Pump pool size set by default at 4.
16.Sep.15 14:26:22,397 < INFO> data_provider element without name attribute; using DEFAULT as the default name.
16.Sep.15 14:26:22,400 < INFO> data_provider element without name attribute; using DEFAULT as the default name.
16.Sep.15 14:26:22,404 < INFO> Loading Metadata Adapter for Adapter Set WELCOME
16.Sep.15 14:26:22,404 < INFO> Loading Metadata Adapter for Adapter Set PROXY_HELLOWORLD_SOCKETS
16.Sep.15 14:26:22,404 < INFO> Loading Metadata Adapter for Adapter Set MARKETDEPTH
16.Sep.15 14:26:22,417 < INFO> Finished loading Metadata Adapter for Adapter Set PROXY_HELLOWORLD_SOCKETS
16.Sep.15 14:26:22,419 < INFO> Loading Data Adapter PROXY_HELLOWORLD_SOCKETS.DEFAULT
16.Sep.15 14:26:22,435 < INFO> Connecting...
16.Sep.15 14:26:22,452 < INFO> Waiting for a connection on port 7001...
16.Sep.15 14:26:22,452 < INFO> Waiting for a connection on port 7002...
16.Sep.15 14:26:22,497 <DEBUG> Rnd generator time (1): 70
16.Sep.15 14:26:22,498 <DEBUG> Rnd generator time (2):700
16.Sep.15 14:26:22,498 <DEBUG> Rnd generator time (3): 35000
16.Sep.15 14:26:22,498 <DEBUG> Rnd generator time (4): 25000
16.Sep.15 14:26:22,498 < INFO> MarketDepth Demo Metadata Adapter start.
16.Sep.15 14:26:22,498 < INFO> Finished loading Metadata Adapter for Adapter Set MARKETDEPTH
16.Sep.15 14:26:22,500 < INFO> Loading Data Adapter MARKETDEPTH.DEFAULT
16.Sep.15 14:26:22,506 < INFO> Stock SuperFast Tech. reference price: 25.69
16.Sep.15 14:26:22,509 <DEBUG> New initial buy order - 20.12 x 1102
16.Sep.15 14:26:22,509 <DEBUG> New initial buy order - 24.65 x 77022
16.Sep.15 14:26:22,510 <DEBUG> New initial buy order - 24.77 x 83395
16.Sep.15 14:26:22,510 <DEBUG> New initial buy order - 22.75 x 97786
16.Sep.15 14:26:22,511 <DEBUG> New initial buy order - 22.37 x 38615
16.Sep.15 14:26:22,511 <DEBUG> New initial sell order - 31.12 x 73383
16.Sep.15 14:26:22,511 <DEBUG> New initial sell order - 29.72 x 70509
16.Sep.15 14:26:22,511 <DEBUG> New initial sell order - 27.02 x 68650
16.Sep.15 14:26:22,512 <DEBUG> New initial sell order - 27.48 x 28340
16.Sep.15 14:26:22,512 <DEBUG> New initial sell order - 31.78 x 92038
16.Sep.15 14:26:22,512 <DEBUG> New initial sell order - 26.56 x 90676
16.Sep.15 14:26:22,512 <DEBUG> New initial sell order - 31.79 x 72579
16.Sep.15 14:26:22,512 <DEBUG> New initial sell order - 28.05 x 1043
16.Sep.15 14:26:22,512 < INFO> Stock AXY COMPANY Ltd reference price: 43.14
16.Sep.15 14:26:22,513 <DEBUG> New initial buy order - 39.92 x 70082
16.Sep.15 14:26:22,513 <DEBUG> New initial buy order - 41.55 x 37181
16.Sep.15 14:26:22,513 <DEBUG> New initial buy order - 37.69 x 21685
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 36.44 x 86768
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 40.69 x 2288
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 32.82 x 9149
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 38.48 x 78115
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 40.91 x 51964
16.Sep.15 14:26:22,514 <DEBUG> New initial buy order - 33.64 x 87564
16.Sep.15 14:26:22,515 <DEBUG> New initial buy order - 35.43 x 51929
16.Sep.15 14:26:22,515 <DEBUG> New initial sell order - 48.76 x 13466
16.Sep.15 14:26:22,515 <DEBUG> New initial sell order - 49.67 x 6001
16.Sep.15 14:26:22,515 <DEBUG> New initial sell order - 50.71 x 86734
16.Sep.15 14:26:22,515 <DEBUG> New initial sell order - 53.03 x 16726
16.Sep.15 14:26:22,515 <DEBUG> New initial sell order - 53.29 x 18806
16.Sep.15 14:26:22,516 <DEBUG> New initial sell order - 51.56 x 76892
16.Sep.15 14:26:22,516 <DEBUG> New initial sell order - 52.42 x 96534
16.Sep.15 14:26:22,517 <DEBUG> New initial sell order - 50.23 x 14519
16.Sep.15 14:26:22,517 <DEBUG> New initial sell order - 46.37 x 57472
16.Sep.15 14:26:22,517 <DEBUG> New initial sell order - 50.84 x 71511
16.Sep.15 14:26:22,517 < INFO> Stock BTTQ Industries Sa reference price: 44.43
16.Sep.15 14:26:22,518 <DEBUG> New initial buy order - 36.44 x 12935
16.Sep.15 14:26:22,518 <DEBUG> New initial buy order - 43.04 x 85248
16.Sep.15 14:26:22,518 <DEBUG> New initial buy order - 35.6 x 19424
16.Sep.15 14:26:22,518 <DEBUG> New initial buy order - 37.33 x 35533
16.Sep.15 14:26:22,518 <DEBUG> New initial sell order - 52.9 x 55175
16.Sep.15 14:26:22,518 <DEBUG> New initial sell order - 53.07 x 69726
16.Sep.15 14:26:22,518 <DEBUG> New initial sell order - 50.92 x 30783
16.Sep.15 14:26:22,519 < INFO> Stock Old&Slow Company ... reference price: 48.1
16.Sep.15 14:26:22,519 <DEBUG> New initial buy order - 44.01 x 44187
16.Sep.15 14:26:22,519 <DEBUG> New initial buy order - 45.23 x 43643
16.Sep.15 14:26:22,519 <DEBUG> New initial buy order - 38.93 x 43995
16.Sep.15 14:26:22,519 <DEBUG> New initial buy order - 45.95 x 31931
16.Sep.15 14:26:22,519 <DEBUG> New initial buy order - 43.79 x 85519
16.Sep.15 14:26:22,519 <DEBUG> New initial sell order - 54.05 x 97436
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 57.46 x 5816
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 54.81 x 64398
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 56.51 x 44199
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 53.54 x 71399
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 58.11 x 61163
16.Sep.15 14:26:22,520 <DEBUG> New initial sell order - 52.94 x 23606
16.Sep.15 14:26:22,521 < INFO> MarketDepth Demo Data Adapter start.
16.Sep.15 14:26:22,521 < INFO> Finished loading Data Adapter MARKETDEPTH.DEFAULT
16.Sep.15 14:26:22,635 < INFO> Finished loading Metadata Adapter for Adapter Set WELCOME
16.Sep.15 14:26:22,639 < INFO> Loading Data Adapter WELCOME.TELEMETRY
16.Sep.15 14:26:22,639 < INFO> Loading Data Adapter WELCOME.CHAT
16.Sep.15 14:26:22,639 < INFO> Loading Data Adapter WELCOME.STOCKS
16.Sep.15 14:26:22,639 < INFO> Loading Data Adapter WELCOME.ROOMBALL
16.Sep.15 14:26:22,643 < INFO> WebTelemetryF1 Pronto
16.Sep.15 14:26:22,644 < INFO> Finished loading Data Adapter WELCOME.TELEMETRY
16.Sep.15 14:26:22,648 < INFO> Finished loading Data Adapter WELCOME.ROOMBALL
16.Sep.15 14:26:22,649 < INFO> StockQuotesDataAdapter ready.
16.Sep.15 14:26:22,649 < INFO> Finished loading Data Adapter WELCOME.STOCKS
16.Sep.15 14:26:22,650 < INFO> ChatDataAdapter ready
16.Sep.15 14:26:22,650 < INFO> Finished loading Data Adapter WELCOME.CHAT

Netstat -an | grep LIST

tcp46 0 0 *.7001 *.* LISTEN
tcp46 0 0 *.7002 *.* LISTEN
tcp46 0 0 *.8888 *.* LISTEN

I haven't touched any of the conf files (i.e - lightstreamer_conf.xml)

I can open connections to the proxy adapter ports (telnet) but no access to index.html on /pages.

If I remove the proxy adapter, http comes up fine.

[snip]

16.Sep.15 14:31:43,294 < INFO> Lightstreamer Server on Java Virtual Machine: Oracle Corporation, OpenJDK 64-Bit Server VM, 25.45-b02, 1.8.0_45-b14 on FreeBSD
16.Sep.15 14:31:43,294 < INFO> Lightstreamer Server 6.0.1 build 1780 starting...
16.Sep.15 14:31:43,362 < INFO> Server "Lightstreamer HTTP Server" listening to *:8080 ...

What am I stilldoing wrong?

Giuseppe Corti

Hi Mario,

There exist two flavors of Proxy Data Adapter: NetworkedDataProvider and RobustNetworkedDataProvider.
In your case you are using the first one that explicitly provides not to start listening on the server TCP port until both the request_reply_port and notify_port (7001, 7002) are connected.

You can switch to RobustNetworkedDataProvider that allows the server to start even before the Remote Adapter has connected.

<adapter_class>ROBUST_PROXY_FOR_REMOTE_ADAPTER</adapter_class>

Furthermore, the RobustNetworkedDataProvider contains some recovery capabilities and avoid to terminate the Lightstreamer Server process. Full details on the recovery behavior are available as inline comments within the example adapters.xml file available in the “Lightstreamer/DOCS-SDKs/adapter_remoting_infrastructure/doc/adapter_robust_conf_template” folder.

Regards,
Giuseppe

Giuseppe Corti

Hi Mario,

You should consider that the Moderato Edition does not allow clustering. So, since you do not need more that one server, the Moderato edition handles the load exactly as "Allegro/Presto/Vivace" editions could do.
But you have to consider if your application needs additional features that are not included in the Moderato, such as an higher Items update frequency, additional client libraries other than JavaScript, direct email support, etc ... .
For a complete list of features available in the various editions please refer here.

Regards,
Giuseppe

Mlobo

Again, it worked! Thanks Giuseppe.

As for the Moderato issue, I'll have to start testing and see how it behaves under load. What could worry me would be the higher Items update frequency, because we will be using JavaScript client lib.

As you can attest, I'll post my findings or hiccups.

Best wishes,
Mario