Title:Online Real time Multiple Spatiotemporal Action Localisation and Prediction

Abstract:We present a method for multiple spatiotemporal (S/T) action localisation, classification and early prediction based on a single deep learning framework, able to perform in an online fashion and in real time. Our online action detection pipeline comprises of three main steps. In step one, two end-to-end trainable SSD (Single Shot MultiBox Detector) convolutional neural networks are employed to regress and classify detection boxes in each video frame potentially containing an action of interest, one from optical flow and one from RBG images. In step two, appearance and motion cues are combined by merging the detection boxes and classification scores generated by the two networks. In step three, sequences of detection boxes most likely to be associated with a single action instance, called 'action tubes', are constructed incrementally and efficiently in an online fashion, allowing the system to perform early action class prediction and spatial localisation in real time. Empirical results on the challenging UCF101 and J-HMDB-21 datasets demonstrate new state-of-the-art results in S/T action localisation on UCF101, even when compared to offline competitors, and superior results on early action label prediction across the board. To the best of our knowledge, ours is the first real-time framework (up to 52fps) able to perform online spatiotemporal action localisation.