Importing Lightstreamer with Swift Package Manager

boris@ig

As you might heard Xcode 11 now allows us to use Swift Package Manager for iOS projects. This allows us move away from CocoaPods or Carthage. I am already converted our Lightstreamer internal wrapper into Swift Package but I was unable to build it. I am getting errors when trying to import the framework:

error: no such module 'Lightstreamer_iOS_Client'
import Lightstreamer_iOS_Client

I have also tried to wrap lightstreamer into its own SPM project and I am able to build it, but even if my package depends on it, it produces same error.

Quick research showed me that Swift packages do not support binary frameworks, however I think it is worked in Xcode beta 5, now I am using Xcode beta 6 and it doesn't work any more.

Ideally we would have Swift.package file into Lightstreamer client and this would allow us to import it, the code has to be open. Thank you.

gianluca.bertani

Hello,

we are investigating the problem. If it will be possible to wrap our client library in a Swift Package, we will surely do. Anyway, consider that we will wait for Xcode 11 to be GM before finalizing such wrapper. Don't hold your breath.

gianluca.bertani

Hello again,

we investigated the matter, and I’m afraid we have bad news. As reported even at the very beginning of the official documentation, a Swift package can't contain binaries.

We could publish our source code (in fact, our client SDKs are just reference implementations of our TLCP public protocol, there’s nothing to hide), but our build is currently based on a Java to Obj-C transpilation process (discussed in this blog post) that requires multiple libraries from an external project (Google’s J2ObjC). Those libraries could not be included, since they are binaries, and they are not available in turn as a Swift package.

This rules out the possibility to wrap our iOS client SDK, as is, in a Swift package, at least for now. The day J2ObjC should release their libraries as a Swift package, or if we should ever embark on a full rewrite of the client in Swift, we will re-evaluate the opportunity. Until then, you will have to continue with more traditional ways to add our client to your projects.

Thanks anyway for bringing this matter to our attention.

boris@ig

gianluca.bertani Hello again,

we investigated the matter, and I’m afraid we have bad news. As reported even at the very beginning of the official documentation, a Swift package can't contain binaries.

Hi, thanks for your answer. I understand the limitation.

Xcode 11 supports distributing binary libraries with new XCFrameworks format. https://developer.apple.com/videos/play/wwdc2019/416/
Just out of curiosity, I have tried to run

cd lightstreameriosclientxcodebuild
-create-xcframework -framework Lightstreamer_iOS_Client.framework -output lightstreamer.xcframework

And I am getting error:
error: binaries with multiple platforms are not supported

Is this because your binary contains Java? Only Swift and "C based code" is supported.

If you decide to rewrite your client in Swift — XCFrameworks is wonderful tool for distribution with official support. It's great alternative to SPM if you don't want to go open source. I wish you all the best.

dehesa

Yes, indeed SPM doesn't yet support precompiled binaries. It seems they are on the pitching stage for this feature. For more information check:
- Pitch: https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620
- SPM support for binary distribution discussion: https://forums.swift.org/t/spm-support-for-binaries-distribution/25549
- SPM bundles & resources support discussion: https://forums.swift.org/t/swift-pm-bundles-and-resources/13981

It is quite interesting to realize you are transpiling Java to Obj-C. It seem to work well for my usage, which is mainly desktop bounded (https://github.com/dehesa/IG). I do wonder however how well does it play with URLSession including this year's changes (low data mode, etc). I had to do a MQTT library wrapping for Apple systems some years ago and using C and Posix implied not using URLSession, which work as intended on WiFi, but on Cell stop working altogether.

gianluca.bertani

Our client library uses only native components for networking: NSURLSession on HTTPS transport, and CFStream on WebSocket transport.

The mechanism we use is a simple dependency injection pattern:

The Java engine wraps all the services it needs for connecting, receiving messages, sending messages etc., behind a Java interface.
The Java desktop and Android clients implement this interface with Java classes that use Netty as networking component.
The transpilation process scraps the interface implementations, leaving just the bare interface, which then becomes an Obj-C protocol on the iOS client.
Finally, the protocol is then implemented with a Obj-C classes that provide the same services using NSURLSession or CFStream as required.

So, our client is actually well aligned with the latest features of NSURLSession. We don't circumvent them using BSD sockets or similar expedients. Our client is, and has always been, a good iOS citizen regarding networking.

dehesa

Thanks for the details, Gianluca! and congratulations on achieving a multi-platform working architecture :Smile_Ad: (I know it is not easy).

gianluca.bertani

Thank you very much. It's a team effort of course, I'll forward your words to the other people of the team.

gianluca.bertani

Hello,

no, Java is surely not the culprit. In fact, our client library does not contain or incapsulate Java code, nor a Java Virtual Machine, by any means. It contains only pure Obj-C code transpiled from Java code, the meaning of this being: Obj-C source code automatically produced by a tool to be functionally equivalent to its corresponding Java code. Plus a number of support libraries (which are still pure Obj-C).

The error you see is due to our binaries supporting multiple architectures: different iPhone/iPad ARM code segments, plus the x64 code segment for the simulator. An XCFramework could be built starting from separate code segments, i.e. multiple single-architecture frameworks instead a single multiple-architecture framework.

I will check out the added support for binaries and keep this thread updated. Thanks for the heads up.

dehesa

Hi Gianluca,

The Swift Package Manager has now implemented all the requirements needed to distribute LightStreamer through SPM. You can see the new features in the Swift Evolution status page. Specifically:
- SE-273: Package Manager conditional target dependencies.
- SE-272: Package Manager binary dependencies.
- SE-271: Package Manager resources.
- SE-278: Package Manager localized resources.

These features are already implemented and can be accessed in the Swift development snapshots. They will be released to the broad public with Swift 5.3. Supporting SPM would definitely help me out managing dependencies and updating libraries. Please consider supporting SPM as soon as you can (SPM better than XCFramework).

In that same vein, boris@ig, IG could also update its Lightstreamer requirements. It seems it has been frozen in time.

gianluca.bertani

Thanks for your information. We will surely take a second look at supporting SPM.

And sorry for the late answer, unfortunately your post was flagged for moderation (probably due to links) and it went under our radars.

dehesa

Hello again Gianluca (and Boris),

Sorry to revive this old post. After some research and with the release of Swift 5.3, I believe I found a way to deliver Lighstreamer binaries through SPM that doesn't require code changes, just a Xcode rebuild.

Let me summarize the current state of things:
1. Lightstreamer transpiles Java to ObjC and delivers four dynamic libraries in the shape of .framework files (for macOS, iOS, tvOS, and watchOS).
2. End-clients wanting to use Lightstreamer functionality just needs to embed/link the selected Lighstreamer framework file in their Xcode project.
3. More often than not, the Lighstreamer dynamic framework is just one low-level component within a major library. A good example is IG (company where Boris works). IG uses Lightstreamer as their "real-time" messaging system, but they offer much more functionality.
4. Swift programmers wanting to use IG services, need to know all dependencies and their requirements, manually download them, embed/link the dependency frameworks, and set correctly all the build settings (they need to be careful since there are different dynamic libraries per target platform: macOS x86-64, macOS arm64, iOS arm64, iOS x86 simulator, tvOS arm64, tvOS simulator, etc.).

The goal is to make life as simple as possible for the end user. Ideally, a user specify a SPM url in his/her project, and all the required dependencies (for the right platforms) are downloaded and compiled.

SPM and Swift 5.3 enable this through the new .binaryTarget. It is now possible to specify binary dependencies, by providing a URL and a checksum. The only caveat is that the binary dependency must be exposed as a .xcframework. A xcframeworkis just a standard bundle of compiled binaries. It is actually possible to create a single xcframework from all the already released Lighstreamer dynamic libraries, by using the following terminal command:

xcodebuild -create-xcframework \
-framework Lightstreamer_iOS_Client.framework \
-framework Lightstreamer_macOS_Client.framework \
-framework Lightstreamer_tvOS_Client.framework \
-output Lightstreamer.xcframework

Sadly,this command fails because the iOS and tvOS frameworks currently bundle both their arm64 code and the x86 simulator code. xcframework enforces that each framework must only target a single architecture. Therefore, for this command to work there must be a framework for each architecture: iOS arm64, iOS simulator, tvOS arm64, tvOS simulator, etc.

So Gianluca, I would kindly ask you to do one of the following things:
- Provide a framework per platform architecture (do not bundle simulator and real device code), or
- Provide a xcframework zip file.

This would enable the delivery of Swift Packages, ease the use of the Lighstreamer framework, and reduce the amount of compilation/linking errors. Personally, I need a xcframeworkfor Lightstreamer Allegro Presto Vivace 6.1.0 2017/01/23 (unified API 2.1.2), since this is what IG support. Is it possible to receive such xcframework zip file?

gianluca.bertani

Hello,

thank you for your valuable insights into the problem. I will surely take a look and let you know.

dehesa

Great. Looking forward your analysis/response. In the meantime, you can find a project currently delivering Lighstreamer binaries through SPM here. That project delivers a library (which can statically or dynamically linked) and a packaged Lighstreamer.xcframework.

Regards,

eric280

Creating an XCFramework for the Lightstreamer iOS Client

1. Create a work directory structure:
work/
os/
sim/

mkdir -p work/os work/sim

2. Change directory into work.
cd work

3. Download the Lightstreamer iOS Client.
curl -O https://www.lightstreamer.com/repo/cocoapods/ls-ios-client/4.2.1/ls-ios-client-4.2.1.zip

4. Unzip ls-ios-client-4.2.1.zip
unzip ls-ios-client-4.2.1.zip

5. Copy the Lightstreamer_iOS_Client.framework directory into os/ and sim/
cp -r Lightstreamer_iOS_Client.framework os
cp -r Lightstreamer_iOS_Client.framework sim

6. Change directory into os/Lightstreamer_iOS_Client.framework
cd os/Lightstreamer_iOS_Client.framework

7. List all of the architectures included in the Lightstreamer_iOS_Client file.
lipo -info Lightstreamer_iOS_Client

8. Remove all architectures from the Lightstreamer_iOS_Client file, except arm64
lipo -remove i386 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client
lipo -remove x86_64 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client
lipo -remove armv7 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client

9. Change directory into sim/Lightstreamer_iOS_Client.framework
cd ../../sim/Lightstreamer_iOS_Client.framework

10. List all of the architectures included in the Lightstreamer_iOS_Client file.
lipo -info Lightstreamer_iOS_Client

11. Remove all architectures from the Lightstreamer_iOS_Client file, except x86_64
lipo -remove i386 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client
lipo -remove armv7 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client
lipo -remove arm64 Lightstreamer_iOS_Client -output Lightstreamer_iOS_Client

12. Change directory into work
cd ../../

13. Create the XCFramework.
xcodebuild -create-xcframework -framework os/Lightstreamer_iOS_Client.framework -framework sim/Lightstreamer_iOS_Client.framework -output Lightstreamer_iOS_Client.xcframework

Creating a Swift Package for Lightstreamer_iOS_Client.xcframework

1. Create a directory in which to make the package.
mkdir Lightstreamer_iOS_Client

2. Change into the package directory.
cd Lightstreamer_iOS_Client

3. Initialize the package.
swift package init --type library

4. Remove unneeded files.
rm -rf Sources Tests

5. Copy the Lightstreamer_iOS_Client.xcframework directory to the package directory.
cp -r ../path/to/Lightstreamer_iOS_Client.xcframework .

6. Modify the Package.swift file to add the xcframework.
// swift-tools-version:5.3
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
name: "Lightstreamer_iOS_Client",
products: [
.library(name: "Lightstreamer_iOS_Client", targets: ["Lightstreamer_iOS_Client"])
],
dependencies: [],
targets: [
.binaryTarget(name: "Lightstreamer_iOS_Client", path: "Lightstreamer_iOS_Client.xcframework")
]
)

At this stage, the package is ready to be hosted in git.

dehesa

Thank you Eric, that worked! I wasn't aware you could remove architectures from a fat binary. I will make sure to remember this!

Gianluca, that solved my issue. I would still recommend delivering xcframeworks or SPM packages for the future (or even better provide Swift source code). If you were in the process of rebuilding Lightstreamer Allegro Presto Vivace 6.1.0 (unified API 2.1.3) as I requested, I would kindly ask you to keep doing it and add support for macOS ARM64. That architecture is not currently available in the Lightstreamer download page (only macOS x86_64).

gianluca.bertani

Hello dehesa and eric280,

using your suggestions we were able to build a distribution of the latest version (4.3) of our client library as an XCFramework. Thank you very much for sharing them with us.

Unfortunately, we encountered difficulties with CocoaPods: its support for XCFrameworks, when distributed as binary packages, seems to be buggy. The pod spec lint command fails with linking errors that should not show up. Since CocoaPods is our package manager of choice, at least for now, and most of our customers use it, we have to wait until its support for XCFrameworks improves.

dehesa

That is understandable. The way Lighstreamer is currently being delivered (as dynamic frameworks) and with Eric's suggestion, we can all cover our requirements. I would still ask, if possible, to compile the macOS dynamic framework with "Standard Architectures (64-bit Intel and ARM)". Currently it is only x86-64.

It is not urgent, since there are no macOS arm64 devices yet. But I expect them to land at the end of this year and we will need to support them. Xcode 12 currently lets you build macOS dynamic frameworks for both x86-64 and arm64 (take a look hereor here ).

Regards and thank you, Gianluca, for having taken the time to look at this.

gianluca.bertani

Hello again guys,

we finally nailed it. Find out more about support for SPM and Apple Silicon on our latest blog post: Introducing support for Swift Package Manager and Apple Silicon.

Let us know of any issue you should find.

dehesa

Congratulations on reaching such big milestone. Looking forward trying it out.