This project is the code of Southeast Missouri State University Robust End-to-end Multimodal Detection with Confidence Fusion by Eashwar Ediga, Monesh Venkul Vommi, Harish Reddy.
In the application of autonomous driving, there are times when unexpected and severe weather (fog, snow, night) occurs in outdoor environments, making the detection tasks less effective. Therefore, this Project is about novel multimodal object detection network called MT-DETR. It achieves state-of-the-art performance using the camera, lidar and radar, and additional time information. The experimental results demonstrate that the MT-DETR is robust and performs well in various weather conditions. The good generalization and scalability confirm future applicability to different multimodal tasks.
The repository is based on mmdetection and cbnetv2. Many thanks for their awesome open-source project.
To run the code:
data/
and checkpoint/
and read the instructions there to download.bash run_script/${script_name}
You can comment training/inference block in shell scripts if you want.
The following are the important directories of this project:
data
: download the dataset herecheckpoint
: download model weights hererun_script
: shell files for running models, change your path and GPU_id hereconfigs
: configs of models, adjust models’ setting heremmdet/models/backbones/mt_detr.py
,mmdet/models/backbones/fusion_module.py
: core model architecture of MT-DETR (this paper)@InProceedings{Chu_2023_WACV, author = {Eashwar Ediga, Monesh Venkul Vommi, Harish Reddy }, Project = {MT-DETR: Robust End-to-End Multimodal Detection With Confidence Fusion}, month = {January}, year = {2024},
} ```