Enable object detection with initialization parameters zed_error = zedCamera. The detection model parameter detection_tection_model must be set to CUSTOM_BOX_OBJECTS: ObjectDetectionRuntimeParameters will be ignored for the Custom Model, no filtering will be applied (score thresholds, NMS, etc) to the input 2D boxes. To configure the Object Detection module, use ObjectDetectionParameters at initialization. The detections are relative to the left (rectified) image at native resolution and should be rescaled accordingly if the inference was done at a lower resolution. Then using retrieveObjects the tracked 3D objects can be retrieved. WorkflowĪfter each grab call, the image can be retrieved and sent to your detector, and the bounding box detections result can be ingested into the ZED SDK for processing. Note: The inference can also directly be done in Python using a training framework such as PyTorch or TensorFlow for instance. Optional post-processing steps may need to be implemented depending on the models. This file contains both the network architecture and weights and can easily be used with TensorRT using for instance this light wrapper. Typically when using PyTorch to train the network, the model can also be exported in ONNX format.

This library is not mandatory but is advised to get the best possible performance, especially on smaller devices such as Jetson, where the built-in quantification to fp16/int8 provides optimum run times. For more samples on different network architectures, you can refer to this repository. TensorRT library is installed when installing the ZED SDK AI module. We also provide a sample that can run Yolov4 (and more!) with the OpenCV DNN module, which can be trained using darknet. We provide optimized inference samples that use any Yolov5 model using the TensorRT library. Ideally, the inference requirements such as available memory or compute constraints should be considered from the training step to select the most pertinent model architecture. Several model variants can be chosen to improve accuracy or inference speed. You can refer to this tutorial to train a custom model based on Yolov5 for instance. They can be trained on any dataset with annotated information, known as ground truth to teach the detector what to look for. Several state-of-the-art object detection algorithms can be used to identify and localize objects. This means that the detected object will keep the same ID through the sequence, as depicted below:įor more information on Object Detection, see the Using the API page. If the positional tracking module is activated, the ZED SDK can track the objects within the environment.

From there, the ZED can output the 2D bounding boxes of the objects and accurately compute the 3D bounding boxes with the help of the depth map. The ZED SDK also computes a 2D mask that indicates which pixels of the left image belong to the object. meters) and calculated from the back of the left eye of the camera to the scene object. Similar to the Object Detection module, the distance of the object from the camera is expressed in metric units (ex. Using the 2D bounding boxes given by your own detection algorithm, the ZED SDK identifies the objects and computes their 3D position and velocity. The objects can also be tracked within the environment over time, even if the camera is in motion, thanks to data from the positional tracking module. The detections can then be fed into the ZED SDK that computes the 3D position of each object, as well as their 3D bounding box, using data from the depth module. You can use your own bounding box detector, tailored for each custom need. The 2D detections are ingested and the 3D information such as position, 3D bounding box, and more are computed. Since ZED SDK 3.6, a custom detector can be used with the API. Now even with any 2D bounding box detector.

Thanks to depth sensing and 3D information, the ZED camera can provide the 2D and 3D positions of the objects in the scene. Object detection is the ability to identify objects present in an image. The close() function is also called automatically by the destructor if necessary.Using the Object Detection API with a Custom Detector Note: zed.close() can also disable properly all active modules. enableObjectTracking ) Disable modules and exitīefore exiting the application, modules need to be disabled and the camera closed. enable2DMask = true if ( detection_parameters. enableObjectTracking = true detection_parameters. Define the Object Detection module parameters ObjectDetectionParameters detection_parameters = new ObjectDetectionParameters () detection_parameters.

Zed lol model obj