Skip to content

Latest commit

 

History

History
118 lines (95 loc) · 5.72 KB

Realsense.md

File metadata and controls

118 lines (95 loc) · 5.72 KB

Using a RealSense Camera

After making sure that everything works by running the EuRoC datasets, it should be possible to use the realsense driver from Monado to get any RealSense camera that has an IMU and one or more cameras to get tracked with SLAM. However, this was only tested on a D455, so if you are having problems with another device, please open an issue. Also, open an issue if you manage to make it work with other devices so that I can add it to this README.

Index

Overview of the Setup (D455)

Let's first assume you have a RealSense D455, which is the one that works with the defaults. Even if you have another RealSense device follow this section, you might at least get something working, although not at its best.

SLAM-Tracked RealSense Driver

Set these environment variables:

  • export RS_HDEV_LOG=debug: Make our realsense device logs more verbose
  • export RS_SOURCE_INDEX=0: Indicate that we want to use the first RealSense device connected as data source
  • export RS_TRACKING=2: Only try to use "host-slam". See other options here.
  • export SLAM_CONFIG=/usr/local/etc/basalt/d455.toml: Configuration file for Basalt and the D455.
  • export SLAM_SUBMIT_FROM_START=true: Send samples to Basalt from start.

RealSense-Tracked Simulated HMD

You now have a RealSense device that you can use to track another device, for example, let's track the Simulated HMD that Monado creates by default when no physical HMD is present.

You can create a "tracking override" to track the Simulated HMD device with the Intel RealSense Host-SLAM device. For this open monado-gui, then Tracking Overrides, Add One. Selecting Simulated HMF as Target Device and Intel RealSense Host-SLAM as Tracker Device and pressing Save. That will add a tracking override section to Monado's config file (~/.config/monado/config_v0.json).

And that's it! You can now start an OpenXR application with Monado and get your view tracked with your D455 camera.

Non-D455 RealSense Devices

While I was unable to test other devices because I don't have access to them, it should be possible to make them work by:

Configuring the RealSense Pipeline

These fields determine your RealSense streaming configuration, and these are their current defaults that work on a D455. You can change those fields by setting any of them in your config_v0.json inside a config_realsense_hdev field. Also note that as we already set RS_HDEV_LOG=debug, you should see the values they are currently taking at the start of Monado.

For example, let's say you have a realsense device which has two fisheye cameras that support streaming 640x360 at 30fps (a T265 I think), then a configuration like this should work:

"config_realsense_hdev": {
  "stereo": true,
  "video_format": 9, // 9 gets casted to RS2_FORMAT_Y8 (see https://git.io/Jzkfw), grayscale
  "video_width": 640, // I am assuming the T265 supports 640x360 streams at 30fps
  "video_height": 360,
  "video_fps": 30,
  "gyro_fps": 0, // 0 indicates any
  "accel_fps": 0,
  "stream_type": 4, // 4 gets casted to RS2_STREAM_FISHEYE (see https://git.io/Jzkvq)
  "stream1_index": -1, // If there were more than one possible stream with these properties select them, -1 is for auto
  "stream2_index": -1,
}

The particular values you could set here are very dependent on your camera. I recommend seeing the values that get output by running the rs-sensor-control command.

Configuring Basalt

Sending the calibration of the camera to Basalt has not been automatized yet so you need to do the following to get the best tracking results.

As you might've noticed, we set SLAM_CONFIG to /usr/local/etc/basalt/d455.toml which is this config file that I added for the D455. This file points to a calibration file and a VIO configuration file.

For the tracking to be as good as possible you should set the intrinsics/extrinsics of the device in a similar calibration file and point to it with the SLAM_CONFIG config file. You can obtain that information from the previously mentioned rs-sensor-control utility. Issues like this (T265) and this (D435) provide configuration files tried by other users. Additionally Basalt provides custom calibration tools that can work for any camera-IMU setup or tools like basalt_rs_t265_record or the euroc_recorder in Monado that can help creating an initial calibration file for RealSense devices.