I-MuPPET:
Interactive Multi-Pigeon Pose Estimation and Tracking
GCPR 2022, oral

Urs Waldmann1,2, Hemal Naik1,2,3,4, Máté Nagy1,2,3,5,6, Fumihiro Kano2,3, Iain D. Couzin1,2,3, Oliver Deussen1,2, and Bastian Goldlücke1,2

1 University of Konstanz, Germany
2 Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Germany
3 Max Planck Institute of Animal Behavior, Konstanz, Germany
4 Technische Universität München, Munich, Germany
5 Hungarian Academy of Sciences, Budapest, Hungary
6 Eötvös Loránd University, Budapest, Hungary

News

2023/10/20 We launched our 3D-MuPPET project page. Check it out here, especially our Hugging Face Demo.
2023/08/30 We extend I-MuPPET to 3D. We show that 3D-MuPPET also works in natural environments without model fine-tuning on additional annotations. [arXiv].
2023/05/31 Annotated single pigeon data to reproduce the results of the paper is available here.
2023/05/31 Benchmark on 2D multi-pigeon tracking is available here.

Abstract

I-MuPPET Framework

Most tracking data encompasses humans, the availability of annotated tracking data for animals is limited, especially for multiple objects. To overcome this obstacle, we present I-MuPPET, a system to estimate and track 2D keypoints of multiple pigeons at interactive speed. We train a Keypoint R-CNN on single pigeons in a fully supervised manner and infer keypoints and bounding boxes of multiple pigeons with that neural network. We use a state of the art tracker to track the individual pigeons in video sequences. I-MuPPET is tested quantitatively on single pigeon motion capture data, and we achieve comparable accuracy to state of the art 2D animal pose estimation methods in terms of Root Mean Square Error (RMSE). Additionally, we test I-MuPPET to estimate and track poses of multiple pigeons in video sequences with up to four pigeons and obtain stable and accurate results with up to 17 fps. To establish a baseline for future research, we perform a detailed quantitative tracking evaluation, which yields encouraging results.

Additional Results

Pose Estimation and Tracking of Multiple Pigeons

I-MuPPET is trained on our labeled single pigeon data set. Nevertheless we are able to estiamte and track poses for multiple pigeons.


Single pigeon. In this video I-MuPPET estimates and tracks the pose of one pigeon. This single pigeon is not from our labeled single pigeon data set that we used for training.



Multiple pigeons. In the following videos I-MuPPET estimates and tracks the pose of multiple pigeons, although only trained on data containing one single pigeon.

Failure case: From 'ID 1' we see that in some cases of the multi-pigeon video sequences, pose estimation is not accurate. Please keep in mind that I-MuPPET is trained on our labeled single pigeon data set only.


Pose Estimation of a 'DeepLabCut Mouse' with I-MuPPET

I-MuPPET is trained on the odor trail tracking data set from DeepLabCut.

Cite us 

                            
    @inproceedings{waldmann2022imuppet,
      title={I-MuPPET: Interactive Multi-Pigeon Pose Estimation and Tracking},
      author={Waldmann, Urs and Naik, Hemal and M\'{a}t\'{e}, Nagy and Kano, Fumihiro and Couzin, Iain D. and Deussen, Oliver and Goldl\"{u}cke, Bastian},
      booktitle={DAGM German Conference on Pattern Recognition},
      year={2022},
      pages={513--528}
      }