ROS 2 Lyrical Luth · Lesson 48 of 86

Depth camera basics

Depth images, intrinsics, projecting pixels to 3D and registered point clouds.

Estimated time · 45–75 min

Learning goals

What you will understand

Explain depth camera basics in your own words
Use it in a small robotics example
Verify the result instead of assuming it worked

Understand the idea first

The mental model

A depth camera (stereo, structured light or time of flight) gives each pixel a distance. With the camera intrinsics fx, fy (focal lengths in pixels) and cx, cy (principal point) you can project pixel (u, v) with depth z to 3D: x = (u − cx) z / fx, y = (v − cy) z / fy. The camera_info topic supplies these numbers.

Depth is usually published in metres as 32-bit float or in millimetres as 16-bit unsigned integer; check the encoding. Depth is noisy at range and invalid (0 or NaN) on shiny or dark surfaces. Use the depth_image_proc package to make a PointCloud2 from a depth image.

Guided lesson

Work through these steps

1.
Source the ROS 2 Lyrical environment in a fresh terminal.
2.
Read the depth camera basics example and identify what each line does.
3.
Build only the affected package, source the workspace overlay, and run the example.
4.
Inspect the result from a second terminal with the ros2 CLI or RViz2 and compare it with the expected result.

Worked example

Read it before you run it

double fx = 500.0, fy = 500.0, cx = 320.0, cy = 240.0;
// Intrinsics from the camera_info message (K matrix).
double u = 420.0, v = 240.0, z = 2.0;
// A pixel and its measured depth in metres.
double x = (u - cx) * z / fx;
// Horizontal offset from the optical axis.
double y = (v - cy) * z / fy;
// Vertical offset from the optical axis.
// The point (x, y, z) is in the camera optical frame: x right, y down, z forward.
// Remember the optical-frame axis convention.

What the example is doing

1.The ros2 commands query the running graph; they do not prove that a node's internal behavior is correct.
2.Build from the workspace root, then source install/setup.bash so the shell can discover the new package.
3.Use a second sourced terminal to inspect nodes and interfaces while the example runs.

Watch it run · Sensor view

See what this code does

The terminal types this lesson's practice solution and runs it. The scene shows the kind of robot behaviour that code drives. It plays by itself and loops.

Practice code

Now type it yourself

Copy the starter into practice.cpp and replace every TODO with your own code. Build and run it: g++ -std=c++17 -Wall -Wextra -Wpedantic practice.cpp -o practice -pthread && ./practice

#include <iostream>

int main() {
  const double fx = 500.0, fy = 500.0, cx = 320.0, cy = 240.0;
  const double u = 420.0, v = 240.0, z = 2.0;
  // TODO: project the pixel to 3D: x = (u-cx)*z/fx, y = (v-cy)*z/fy
  std::cout << x << ' ' << y << ' ' << z << '\n';
  return 0;
}

Expected output

0.4 0 2

Stuck? Show the full solution (try it yourself first)

#include <iostream>

int main() {
  const double fx = 500.0, fy = 500.0, cx = 320.0, cy = 240.0;
  const double u = 420.0, v = 240.0, z = 2.0;
  const double x = (u - cx) * z / fx;
  const double y = (v - cy) * z / fy;
  std::cout << x << ' ' << y << ' ' << z << '\n';
  return 0;
}

Expected result

How you know it worked

ROS_DISTRO is lyrical in every terminal used for the exercise.
The complete package from the exercise builds and the expected graph entity appears in a second terminal.

Practical exercise

Apply the lesson

Read camera_info from a simulated depth camera, project five pixels to 3D by hand, and compare with points from depth_image_proc's point_cloud_xyz node in RViz2.

Before you continue

ROS_DISTRO reports lyrical in every terminal you used
The example builds and runs without errors
You checked the result with a ros2 command, RViz2 or a test, not just by reading the code

If something goes wrong

ros2 cannot find a package or executable.

Source /opt/ros/lyrical/setup.bash, build from the workspace root, then source install/setup.bash in that terminal.

Nodes run but cannot discover each other.

Confirm both terminals use Lyrical, the same ROS_DOMAIN_ID, and compatible network/discovery settings.