Deploying a custom ONNX Model
Users are not limited to the models available or generated by easy_perception_deployment itself.
You can generate your own ONNX model seperately, using your own training framework.
As long as your ONNX model file fits the input shapes required, it can be deployed as a ROS2 inference engine all the same.
Steps
Follow the instruction below to deploy your own ONNX model. Be warned that the input shape of your model could be currently incompatible with that of easy_perception_deployment. If incompatible, please refer to the Incompatible Model Input Shape? section below if the deployment fails.
1. Double-click on easy_perception_deployment.desktop file on your Desktop.
The window labelled, easy_perception_deployment should appear.
2. Click on button labelled, Deploy.
The window labelled, Deploy should appear.
3. Click on the button labelled, ONNX Model, and select your .onnx model file.
4. Click on the button labelled, Label List, and select your .txt file.
Note
Our label list should contain the object text labels that are sorted such that they are correctly mapped to the way the model has been trained to classify. See data/label_list/coco_classes.txt for an example.
5. Click on the DropDown menu labelled, Use Case, and select the Use-Case mode you wish to use.
For more details on what Use-Case filter you can use to finetune your model inference output, please refer to the section, Use Case Configuration below.
6. Once your Use-Case filter is selected, Click on the button labelled, Run.
Your ONNX model will be read and deployed as a ROS2 package, awaiting input images to arrive on its topic.
Use Case Configuration
Is your ONNX model trained to detect too many objects? Is your only option to limit the number of detection to retrain the model?
Retraining a deep learning model is an extremely time-consuming task. That is why easy_perception_deployment offers the use of Use-Case Configuration to easily finetune the outputs of your inference engine.
There are currently 4 modes available.
UseCase Mode No. |
Description |
|---|---|
0 |
Classification |
1 |
Counting |
2 |
Color-Matching |
3 |
Localization |
4 |
Tracking |
Classification
This use-case runs default and does not alter the inference output of the input ONNX model.
Its results are outputted as EPDObjectDetection.msg.
EPDObjectDetection.msg Message Format |
|---|
std_msgs/Header header |
uint64[] class_indices |
float64[] scores |
sensor_msgs/RegionOfInterest[] bboxes |
sensor_msgs/Image[] masks |
Counting
This use-case allows users to select a subset of object labels from the original list of object labels which the input ONNX model is designed to detect.
The inference output of a P2 or P3 model is then filtered to only show the user-selected objects.
Its results are outputted as EPDObjectDetection.msg.
EPDObjectDetection.msg Message Format |
|---|
std_msgs/Header header |
uint64[] class_indices |
float64[] scores |
sensor_msgs/RegionOfInterest[] bboxes |
sensor_msgs/Image[] masks |
Color-Matching
This use-case allows users to select a template reference image in the form of a .png/.jpeg/.jpg image file.
The inference output of a P2 or P3 model is then filtered to only show detections that share similiarity with the reference image.
Its results are outputted as EPDObjectDetection.msg.
EPDObjectDetection.msg Message Format |
|---|
std_msgs/Header header |
uint64[] class_indices |
float64[] scores |
sensor_msgs/RegionOfInterest[] bboxes |
sensor_msgs/Image[] masks |
Localization
This use-case allows users to find the determine a detected object’s 3D euclidean centroid point, estimated object length, breadth, height and orientation.
The inference output of a P3 model is then filtered to localize user-trained objects.
Note
The current use-case strongly relies on Intel Realsense 3D cameras. The ROS2 driver package can be found here with its own installation instructions.
For the exact steps to set this up in ROS2 Foxy, please refer to this link.
For localization, a camera_to_plane_distance_mm ROS2 parameter has been made available to provide a quick pass-through filter that allows you to remove an underlying detection plane to obtain a representative PointCloud cluster of an object.
You can use the following commands to edit camera_to_plane_distance_mm ROS2 parameter.
source /opt/ros/foxy/setup.bash
ros2 param set /processor camera_to_plane_distance_mm <double value>
#eg. ros2 param set /processor camera_to_plane_distance_mm 450
Its results are outputted as EPDObjectLocalization.msg with each object represented as LocalizedObject.msg.
EPDObjectLocalization.msg Message Format |
|---|
std_msgs/Header header |
LocalizedObject[] objects |
float64 ppx |
float64 ppy |
float64 fx |
float64 fy |
uint32 frame_width |
uint32 frame_height |
sensor_msgs/Image depth_image |
uint32 process_time |
LocalizedObject.msg Message Format |
|---|
string name |
sensor_msgs/RegionOfInterest roi |
sensor_msgs/Image segmented_binary_mask |
geometry_msgs/Point centroid |
float64 length |
float64 breadth |
float64 height |
sensor_msgs/PointCloud2 segmented_pcl |
geometry_msgs/Vector3 axis |
Tracking
This use case allows users to find the determine a detected object’s 3D euclidean centroid point, estimated object length, breadth, height and orientation. This use case extends the Localization feature and uniquely tag each new objects that appear in front of the camera.
Note
The current use-case strongly relies on Intel Realsense 3D cameras. The ROS2 driver package can be found here with its own installation instructions.
For the exact steps to set this up in ROS2 Foxy, please refer to this link.
For localization, a camera_to_plane_distance_mm ROS2 parameter has been made available to provide a quick pass-through filter that allows you to remove an underlying detection plane to obtain a representative PointCloud cluster of an object.
You can use the following commands to edit camera_to_plane_distance_mm ROS2 parameter.
source /opt/ros/foxy/setup.bash
ros2 param set /processor camera_to_plane_distance_mm <double value>
#eg. ros2 param set /processor camera_to_plane_distance_mm 450
Its results are outputted as EPDObjectTracking.msg with each object represented as LocalizedObject.msg and tagged uniquely with corresponding integer label.
EPDObjectTracking.msg Message Format |
|---|
std_msgs/Header header |
LocalizedObject[] objects |
string[] object_ids |
float64 ppx |
float64 ppy |
float64 fx |
float64 fy |
uint32 frame_width |
uint32 frame_height |
sensor_msgs/Image depth_image |
uint32 process_time |
LocalizedObject.msg Message Format |
|---|
string name |
sensor_msgs/RegionOfInterest roi |
sensor_msgs/Image segmented_binary_mask |
geometry_msgs/Point centroid |
float64 length |
float64 breadth |
float64 height |
sensor_msgs/PointCloud2 segmented_pcl |
geometry_msgs/Vector3 axis |
Incompatible Model Input Shape?
Here at team ROS-Industrial Consortium Asia Pacific, we are continuously expanding the compatibility of different ONNX model with easy_perception_deployment over time.
So if it is extremely difficult or impossible for you to tweak your model’s input shape please let us know by posting it as a GitHub issue here at official GitHub repository.
Thank you.