ROS 2 Lyrical Luth · Lesson 54 of 86

Object detection: YOLO and OpenCV

Running a detector on camera frames, bounding boxes, IoU and non-maximum suppression.

Estimated time · 60–90 min

Learning goals

What you will understand

Explain object detection: yolo and opencv in your own words
Use it in a small robotics example
Verify the result instead of assuming it worked

Understand the idea first

The mental model

An object detector such as YOLO takes an image and returns bounding boxes with a class and a confidence. Run it in a node that subscribes to the camera, calls the model (through OpenCV's dnn module or an inference runtime) and publishes vision_msgs/Detection2DArray.

Overlapping boxes for one object are removed by non-maximum suppression, which uses IoU (intersection over union): the area of overlap divided by the area of the union. Filter by a confidence threshold, run the network at a lower rate than the camera, and use a GPU when available.

Guided lesson

Work through these steps

1.
Source the ROS 2 Lyrical environment in a fresh terminal.
2.
Read the object detection: yolo and opencv example and identify what each line does.
3.
Build only the affected package, source the workspace overlay, and run the example.
4.
Inspect the result from a second terminal with the ros2 CLI or RViz2 and compare it with the expected result.

Worked example

Read it before you run it

cv::dnn::Net net = cv::dnn::readNetFromONNX("yolov8n.onnx");
// Load a trained model.
cv::Mat blob = cv::dnn::blobFromImage(frame, 1.0 / 255.0, cv::Size(640, 640), cv::Scalar(), true, false);
// Resize, scale and reorder the pixels into a network input.
net.setInput(blob);
// Feed the input.
cv::Mat out = net.forward();
// Run the network and get raw predictions.
std::vector<int> keep;
// Indices of boxes that survive suppression.
cv::dnn::NMSBoxes(boxes, scores, 0.5f, 0.45f, keep);
// Drop boxes below 0.5 confidence or overlapping a better box by more than 0.45 IoU.

What the example is doing

1.The ros2 commands query the running graph; they do not prove that a node's internal behavior is correct.
2.Build from the workspace root, then source install/setup.bash so the shell can discover the new package.
3.Use a second sourced terminal to inspect nodes and interfaces while the example runs.

Watch it run · Sensor view

See what this code does

The terminal types this lesson's practice solution and runs it. The scene shows the kind of robot behaviour that code drives. It plays by itself and loops.

Practice code

Now type it yourself

Copy the starter into practice.cpp and replace every TODO with your own code. Build and run it: g++ -std=c++17 -Wall -Wextra -Wpedantic practice.cpp -o practice -pthread && ./practice

#include <algorithm>
#include <iostream>

struct Box {
  double x1, y1, x2, y2;
};

double iou(const Box& a, const Box& b) {
  // TODO: intersection width/height = overlap of the two boxes (min of maxes - max of mins, at least 0); IoU = inter / (areaA + areaB - inter)
}

int main() {
  std::cout << iou({0, 0, 2, 2}, {1, 1, 3, 3}) << '\n';
  return 0;
}

Expected output

0.142857

Stuck? Show the full solution (try it yourself first)

#include <algorithm>
#include <iostream>

struct Box {
  double x1, y1, x2, y2;
};

double iou(const Box& a, const Box& b) {
  const double iw = std::max(0.0, std::min(a.x2, b.x2) - std::max(a.x1, b.x1));
  const double ih = std::max(0.0, std::min(a.y2, b.y2) - std::max(a.y1, b.y1));
  const double inter = iw * ih;
  const double area_a = (a.x2 - a.x1) * (a.y2 - a.y1);
  const double area_b = (b.x2 - b.x1) * (b.y2 - b.y1);
  return inter / (area_a + area_b - inter);
}

int main() {
  std::cout << iou({0, 0, 2, 2}, {1, 1, 3, 3}) << '\n';
  return 0;
}

Expected result

How you know it worked

ROS_DISTRO is lyrical in every terminal used for the exercise.
The complete package from the exercise builds and the expected graph entity appears in a second terminal.

Practical exercise

Apply the lesson

Run a YOLO model on frames from a simulated camera, publish a Detection2DArray, draw boxes in an annotated image topic, and measure the frame rate. Implement IoU yourself and test it on three box pairs.

Before you continue

ROS_DISTRO reports lyrical in every terminal you used
The example builds and runs without errors
You checked the result with a ros2 command, RViz2 or a test, not just by reading the code

If something goes wrong

ros2 cannot find a package or executable.

Source /opt/ros/lyrical/setup.bash, build from the workspace root, then source install/setup.bash in that terminal.

Nodes run but cannot discover each other.

Confirm both terminals use Lyrical, the same ROS_DOMAIN_ID, and compatible network/discovery settings.