Title:Temporally Coherent Chinese Restaurant Process for Discovery of Persons and Corresponding Tracklets from User-generated Videos

Abstract:A tracklet is a short sequence of detections of an entity of interest, such as a person's face, in contiguous frames of a video. In this paper we consider clustering tracklets in long videos, an important problem of current interest in Computer Vision. It involves identifying tracklets in short segments of the video and finding associations between tracklets over the entire video. These tasks require careful modelling of spatio-temporal cues. State of the art [27] proposes a probabilistic model which incorporates the spatio-temporal cues by parametrizing them through clique potentials in HMRF. The attendant learning and inference problems make the model un-wieldy resulting in failure to handle long videos with many cluster labels. In this paper we circumvent the problem of explicitly encoding spatio-temporal cues by exploiting Temporal Coherence (TC). The major contribution of the paper is to develop Temporally Coherent Chinese Restaurant Process (TC-CRP), a novel Bayesian Non-parametric (BNP) prior which models Temporal Coherence. To the best of our knowledge this is the first work which models TC in a BNP framework and thus makes a significant addition to BNP priors suited for video analytics. On an interesting problem of discovering persons and their tracklets from user-uploaded videos of long TV series episodes from Youtube we show that TC-CRP is very effective. It can also filter out tracklets resulting from false detections. We explore alternative approaches based on low-rank matrix recovery and constrained subspace clustering, but find these to be very slow and less accurate than our method. Finally, we show that TC-CRP can be useful in Low-rank Matrix Recovery when the desired matrix has sets of identical columns.