Config files are detailed and robot-specific — if you don't have a reference .yaml provided by us, reach out before starting from scratch!

This guide covers the sections that change most frequently. For anything beyond what's listed here, contact us on Slack.

Cameras

Cameras are defined in the top-level cameras: section, each with an adapter plugin and encoder settings. As its a list, you can add and remove cameras however you want.

<aside> 💡

Due to USB and device access mechanics with camera devices, each camera is started in the order you specified in the cameras list with a 5 second delay between each successive camera. All hardware access is isolated so if If 1 camera fails to open, the remaining will continue on.

</aside>

Camera Adapters

ZED Cameras (stereo vision, neck mounting):

cameras:
  - name: "neck"
    adapter:
      plugin: "sentinel_vision_adapter_zed::ZEDCamera"
      config:
        serial_number: "PLACEHOLDER_SERIAL"
        resolution: "HD720"  # HD720, HD1080, HD2K, VGA, SVGA
        fps: 60
        depth_mode: "NONE"  # NONE, PERFORMANCE, QUALITY
        view_mode: "SIDE_BY_SIDE"  # Stereo output format, also supports "LEFT" and "RIGHT"

RealSense Cameras (wrist/end-effector mounting):

- name: "left_wrist"
  adapter:
    plugin: "sentinel_vision_adapter_realsense::RealSenseCamera"
    config:
      serial_number: "230322276758"
      width: 640
      height: 480
      fps: 30
      enable_color: true

Encoders

Encoders compress camera feeds for streaming to XR headsets. The GStreamerEncoder is the most flexible. Like cameras, encoders are also a list per camera, meaning you can attach an arbitrary number of encoders to each camera. For example, h264 for streaming and raw images for storage.

<aside> 💡

Note: If you want to see the video in the quest, a GStreamerEncoder h264 encoder is required.

</aside>

Hardware-accelerated H.264/5 (NVIDIA GPU required):

encoders:
  - name: "streaming"
    plugin: "sentinel_vision_encoder_gstreamer::GStreamerEncoder"
    topic: "encoded"
    config:
      encoding: "h264"
      width: 2560
      height: 720
      framerate: 60
      bitrate: 20000  # kbps
      hw_accel: "nvenc"  # NVIDIA hardware encoding
      preset: "ll"  # Low latency
      rate_control: "cbr"  # Constant bitrate
      profile: "baseline"
      gop_size: 60  # Keyframe interval
      bframes: 0  # No B-frames for low latency

Software H.264/5 (CPU-only, lower bitrate - on highend cpus this can be faster than nvenc):

encoders:
  - name: "streaming"
    plugin: "sentinel_vision_encoder_gstreamer::GStreamerEncoder"
    config:
      encoding: "h264"
      bitrate: 1200  # Lower for CPU encoding
      hw_accel: "none"
      preset: "ultrafast"  # CPU preset
      tune: "zerolatency"

Adjustable parameters: