Proc. Asilomar Conference on Signals, Systems and Computers, Nov. 2-5, 2014, pp. 673-677, Pacific Grove, CA.

Real-Time 3D Rotation Smoothing for Video Stabilization

Chao Jia, Zeina Sinno and Brian L. Evans

Department of Electrical and Computer Engineering, Wireless Networking and Communications Group, The University of Texas at Austin, Austin, TX 78712 USA
kurtjc@gmail.com - zeina@utexas.edu - bevans@ece.utexas.edu

Paper(1 MB) - PDF Slides (1 MB) - PowerPoint Slides (160 MB) - Project Site

Videos:
Figure 1: (a) Original Video - (b) Proposed IIR-like smoothing - (c) Proposed UKF-like smoothing
Figure 2: (a) Original Video - (b) Proposed IIR-like smoothing - (c) Proposed UKF-like smoothing
Original videos are 480 x 720 and stablized videos are 360 x 540

Abstract

We propose two real-time motion smoothing algorithms for video stabilization using a pure 3D rotation motion model with known camera projection parameters. Both proposed algorithms aim at smoothing 3D rotation matrix sequences in a causal way. The first algorithm smooths the 3D rotation sequences in a way similar to 1st-order IIR filtering. The second algorithm uses sequential probabilistic estimation under a constant angular velocity model. These two algorithms are generalized from classical 2D motion smoothing algorithms. We exploit the manifold structure of the rotation matrices so that the proposed algorithms directly smooth the 3D rotation sequences on the manifold. In addition, we introduce a simple projection step in order to guarantee that no black borders intrude into the stabilized video frames. Experimental results show that the proposed algorithms are able to effectively stabilize video sequences and outperform their 2D counterparts with less jitter and distortion.

Questions and Answers

  1. Why do you consider the methods presented real time?
    Answer: The first method, the IIR-based, executes a frame every 1.57 ms while the second one, the KF based, takes 6.97 ms. Given that the acquisition is done with a frame rate below 30 frames/s, the delay for the first method would be less than 50 ms and for the second one less than 250 ms. As a result we can consider the methods to be online.

  2. The presented videos are subject to very high distortions. Are the videos always distorted as such? If you have less distortion would you expect good results?
    Answer: Handheld devices are used by a wide variety of users, so we would expect a wide range of distortions. We would expect such distorted videos under some conditions, e.g if the road is not smooth. For a proof-of-concept, we present highly distorted cases. We would expect our algorithm to perform as well when we have less distortions.

  3. What is the frame size of the new video?
    Answer: 540x360. Originally it was 720x480 in the two provided examples.

COPYRIGHT NOTICE: All the documents on this server have been submitted by their authors to scholarly journals or conferences as indicated, for the purpose of non-commercial dissemination of scientific work. The manuscripts are put on-line to facilitate this purpose. These manuscripts are copyrighted by the authors or the journals in which they were published. You may copy a manuscript for scholarly, non-commercial purposes, such as research or instruction, provided that you agree to respect these copyrights.

Last updated 01/31/16.