MS3D: mean-shift object tracking boosted by joint back projection of color and depth

In this paper, we present MS3D tracker, which extends the mean-shift tracking algorithm in several ways when RGB-D data is available. We fuse color and depth distribution efficiently in the mean-shift tracking scheme. In addition, in order to improve the robustness of the description of the object to be tracked, we further process the pixels in the rectangular region of interest (ROI) returned by mean-shift. We apply depth distribution analysis to pixels of the ROI in order to separate background pixels from pixels belonging to the object to be tracked (i.e. the target region). Then, we use the color histogram of the target region and its surroundings to create a discriminative color model, which has the capability to distinguish the object from background. The proposed algorithm is evaluated on the RGB-D tracking dataset proposed by [1]. It ranked in the first position and it runs in real-time showing both accuracy and robustness in the challenge sequences of background clutter, occlusion, scale variation and shape deformation.