CodeTheRobot
Course outline

ROS 2 Lyrical Luth · Lesson 84 of 86

Building a perception pipeline

Project: camera and lidar processing, detection, tracking and publishing results.

Estimated time · 120–180 min

Learning goals

What you will understand

  • Explain building a perception pipeline in your own words
  • Use it in a small robotics example
  • Verify the result instead of assuming it worked

Understand the idea first

The mental model

A perception pipeline turns raw sensor data into information the rest of the robot uses: rectified images, point clouds, detected objects with poses in a common frame, and tracks with IDs over time. Build it as separate nodes (or components in one container) connected by topics with clear message types such as vision_msgs.

Keep frames and timestamps consistent, transform detections into base_link or map with TF, and measure each stage's latency. Run heavy stages at a lower rate, drop old frames rather than queueing them, and record a test bag so the same input can verify each software change.

Guided lesson

Work through these steps

  1. 1.

    Source the ROS 2 Lyrical environment in a fresh terminal.

  2. 2.

    Read the building a perception pipeline example and identify what each line does.

  3. 3.

    Build only the affected package, source the workspace overlay, and run the example.

  4. 4.

    Inspect the result from a second terminal with the ros2 CLI or RViz2 and compare it with the expected result.

Worked example

Read it before you run it

camera driver          -> /camera/image_raw
# Raw image.
image_proc            -> /camera/image_rect
# Rectified image.
detector (YOLO)       -> /detections (vision_msgs/Detection2DArray)
# Objects in the image.
depth fusion          -> /objects_3d (vision_msgs/Detection3DArray)
# Objects with positions in metres.
tracker               -> /tracks
# IDs that stay with an object over time.
ros2 launch perception_pipeline pipeline.launch.py
# Start all nodes together.

What the example is doing

  1. 1.The ros2 commands query the running graph; they do not prove that a node's internal behavior is correct.
  2. 2.Build from the workspace root, then source install/setup.bash so the shell can discover the new package.
  3. 3.Use a second sourced terminal to inspect nodes and interfaces while the example runs.

Watch it run · Sensor view

See what this code does

The terminal types this lesson's practice solution and runs it. The scene shows the kind of robot behaviour that code drives. It plays by itself and loops.

SENSOR VIEW

Demo for Building a perception pipeline. Expected result: The detector node starts and /detections appears

Practice code

Now type it yourself

Copy the starter into your ROS 2 package, replace every TODO, then build with colcon build and run it from a sourced terminal.

# launch/pipeline.launch.py
from launch import LaunchDescription
from launch_ros.actions import Node


def generate_launch_description():
    # TODO: start the node detector_node from package perception_pipeline, remapping image to /camera/image_rect
    return LaunchDescription([detector])

Expected output

The detector node starts and /detections appears
Stuck? Show the full solution (try it yourself first)
# launch/pipeline.launch.py
from launch import LaunchDescription
from launch_ros.actions import Node


def generate_launch_description():
    detector = Node(package='perception_pipeline', executable='detector_node',
                    remappings=[('image', '/camera/image_rect')])
    return LaunchDescription([detector])

Expected result

How you know it worked

  • ROS_DISTRO is lyrical in every terminal used for the exercise.
  • The complete package from the exercise builds and the expected graph entity appears in a second terminal.

Practical exercise

Apply the lesson

Assemble a pipeline (rectify, detect, project to 3D, track) on a recorded bag, measure the latency of every stage, and show the tracked objects in RViz2 with markers.

Before you continue

  • ROS_DISTRO reports lyrical in every terminal you used
  • The example builds and runs without errors
  • You checked the result with a ros2 command, RViz2 or a test, not just by reading the code

If something goes wrong

ros2 cannot find a package or executable.

Source /opt/ros/lyrical/setup.bash, build from the workspace root, then source install/setup.bash in that terminal.

Nodes run but cannot discover each other.

Confirm both terminals use Lyrical, the same ROS_DOMAIN_ID, and compatible network/discovery settings.