Proc. IEEE Multimedia Signal Processing Workshop, Sep. 17-20, 2012, Banff, Canada.

Probabilistic 3-D Motion Estimation for Rolling Shutter Video Rectification from Visual and Inertial Measurements

Chao Jia and Brian L. Evans

Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712 USA -

Paper - Poster - Slides - Software - Project Site

Images: Fig 1a Distortion - Fig 1b Correction - Fig 9a Distortion - Fig 9b Correction

Paper won IEEE MMSP Top 10% Paper Award

Guitar Oscillations Captured with iPhone 4 showing rolling shutter artifacts


Video acquired by handheld CMOS cameras may suffer from rolling shutter artifacts. Rolling shutter artifacts, which are due to the rows in the image sensor array being exposed sequentially from top to bottom, increase with the speed of the relative motion between the scene and camera. To rectify these artifacts, one needs to recover the projection parameters for each row. In this paper, we propose a probabilistic method to estimate 3-D camera rotation by using video and inertial measurements on the handheld platform, such as a smart phone. Our contributions are
  1. an efficient sensor fusion algorithm using an extended Kalman filter, and
  2. a quality assessment method using vanishing point detection.
Experiments indicate that the proposed sensor fusion algorithm produces a more accurate orientation estimate and better rectifies rolling shutter artifacts.

Video Demonstrations

After the paper was published, we developed video demonstrations to show the difference between the original and the original after rolling shutter artifact rectification had been applied. The rolling shutter artifacts in these two examples are primarily caused by panning the camera. In each video demonstration, the original video is on the left and the rectified video is on the right.

Video #1 - Video #2

In the third video example, we compare the results of video stabilization with and without rolling shutter artifact rectification. The rolling shutter artifacts in this example is caused by jitter of a walking man's hands. Original video is on the top. Bottom left is the stabilized video without rolling shutter artifact rectification, and bottom right is the stabilized video with rolling shutter artifact rectification.

Video #3


I would ask you how have you obtained match_idx,match_x1,match_x2 data to calibrate the camera?

In Matlab, I used SIFT matching with VLFeat. You can find it in our 'sift_track' function in the released software mentioned above. The SIFT feature is robust but a little bit slow. You can also try BRISK or ORB. Both of them can be found in OpenCV. Please check this repo if you need a wrapper for Matlab:

COPYRIGHT NOTICE: All the documents on this server have been submitted by their authors to scholarly journals or conferences as indicated, for the purpose of non-commercial dissemination of scientific work. The manuscripts are put on-line to facilitate this purpose. These manuscripts are copyrighted by the authors or the journals in which they were published. You may copy a manuscript for scholarly, non-commercial purposes, such as research or instruction, provided that you agree to respect these copyrights.

Slides on Early Ideas for Rolling Shutter Artifact Reduction (March 7, 2012)

Last Updated 06/24/15.