About TelemeThing (Division of Autonom LLC)

 Here we see a simulated Airsim drone flying a mission from both a mobile device (Xamarin on IOS + Android) and from a HoloLens. 

The simulation is run in HITL and works in concert with physical GPS devices. The relevant function is pretty much identical a physical drone flying around. The mobile device and the HoloLens are connected and work together. The mobile device communicates with a ROS node running on the drone, relaying telemetry and control to the HoloLens. 

The ROS node on the drone can control both PX4 and DJI. This allows for full control of the drone through ROS, or shared control along with conventional RC and ground station channels, as preferred. 

The mobile device is a 3D tile server for the HoloLens, allowing for pre-fetch and remediation of limited storage on the HoloLens, however the HoloLens will cache tiles within limits. 

 This is a mobile client talking Mavlink and ROS to a ROS node, which then controls the flight controller, which can be PX4 ( or DJI via the PX4/DJI Translator). 

In this scene we see Airsim acting as the simulator. We created this because we didn’t want to get tied to one flight platform, or one radio system, or anything proprietary. This is not an attempt to create yet another ground station, we wrote this specifically to be a really flexible interface to ROS platforms running on UAVs, after we got tired of an endless array of partial solutions and one-off solutions.   

If the platform UAV has the standard RC radio control, then that still works. You can use this platform to just control the ROS payloads and leave the standard RC for classic control, or you can use this as a full replacement of RC, perhaps using LoRa, or packet radio on an amateur band, or something from a satellite. The same goes for the ground station, if you wish you can keep using your ground station and use this just for control of the ROS payloads.  

The client in implemented as a .NET standard lib, so it runs anywhere, including Linux and Azure services. In this video is a client UI  (seen on the right) which we wrote in Xamarin, which means it runs on iOS and Android and Windows. That client is a ROS node, which connects to a ROS node running on the UAV. That client is also a Mavlink flight controller. The Mavlink signals travel between the nodes as ROS messages. The UAV ROS node can run on Windows but it’s probably best run on Ubuntu. We have ROS running on various UAV platforms from medium to large, from RPi and Up and Nvidia, including the awesome Xavier for the serious AV and CV payloads.  The UAV ROS node connects to the flight controller via a serial link. 

If the flight controller is PX4 or Ardupilot then the protocol from node to FC is Mavlink. If the flight controller is DJI then the protocol from node to FC the DJI SDK. In any case: DJI, PX4, Ardupilot; the ground station talks to the UAV only in ROS and Mavlink, this is possible because we wrote a full Mavlink/DJI two-way translator as a ROS node. There is no need to bother any other part of the system with DJI/PX4 specific handling. We did this by porting full on PX4 to ROS, and creating bridges between controls and telemetry. The solution is so complete that you can control a DJI directly from QGC.  

In this video we see the process of setting up and running a waypoint mission. That’s fairly basic. The real value of this platform is the support of ROS running AI, full autonomy and navigation, computer video, Lidar, thermal, searching, tracking, etc. Not seen in this video is invocation of a ROS payload, or the streaming video (including 3D) built in. Also not shown is the HoloLens platform, which is really good for telecasting a remote ‘situation’. We'll feature those in upcoming videos. 

A view of the alignment tool, which is used at setup time to create the translations for extrinsics which are applied at run time in real time in the fusion node. 

Here we see: 1) IR from the FLIR, 2) RGB from the Visible, and 3) the point cloud from the LIDAR. All looking in the same direction, but not aligned and not of the same FOV.   

In the LIDAR view, the temporal fusion span is low enough and the points are small enough to reveal the interesting scan pattern of the LIDAR when looking from the sensor POV. As the viewpoint is manipulated, the 3-D structure of the scene becomes apparent. Don't be misled by the blocky appearance, this view has been down sampled to only 1% of collected points.  

The final product (not shown here) is the fused cloud displayed in real time on iOS, Android, and HoloLens. Detection and segmentation are applied per application, most notably for augmented reality.  Despite running on ROS, most of this is done with lower level tool-kits or direct point manipulation in order to make it fast enough for small hardware, running on a small drone/UAV.