Proc. IEEE Multimedia Signal Processing Workshop,
Sep. 17-20, 2012, Banff, Canada.
Probabilistic 3-D Motion Estimation for Rolling Shutter
Video Rectification from Visual and Inertial Measurements
Chao Jia
and
Brian L. Evans
Department of Electrical
and Computer Engineering,
The University of Texas at Austin,
Austin, TX 78712 USA
kurtjc@gmail.com -
bevans@ece.utexas.edu
Paper -
Poster -
Slides -
Software -
Project Site
Images:
Fig 1a Distortion -
Fig 1b Correction -
Fig 9a Distortion -
Fig 9b Correction
Paper
won IEEE MMSP Top 10% Paper Award
Guitar Oscillations Captured with iPhone 4 showing rolling shutter artifacts
Abstract
Video acquired by handheld CMOS cameras may suffer from rolling shutter
artifacts.
Rolling shutter artifacts, which are due to the rows in the image sensor
array being exposed sequentially from top to bottom, increase with the
speed of the relative motion between the scene and camera.
To rectify these artifacts, one needs to recover the projection parameters
for each row.
In this paper, we propose a probabilistic method to estimate 3-D camera
rotation by using video and inertial measurements on the handheld platform,
such as a smart phone.
Our contributions are
- an efficient sensor fusion algorithm using an extended Kalman filter,
and
- a quality assessment method using vanishing point detection.
Experiments indicate that the proposed sensor fusion algorithm produces a
more accurate orientation estimate and better rectifies rolling shutter
artifacts.
Video Demonstrations
After the paper was published, we developed video demonstrations to show
the difference between the original and the original after rolling shutter
artifact rectification had been applied.
The rolling shutter artifacts in these two examples are primarily caused by
panning the camera.
In each video demonstration, the original video is on the left and the rectified
video is on the right.
Video #1 -
Video #2
In the third video example, we compare the results of video stabilization
with and without rolling shutter artifact rectification.
The rolling shutter artifacts in this example is caused by jitter of a
walking man's hands.
Original video is on the top.
Bottom left is the stabilized video without rolling shutter artifact
rectification, and bottom right is the stabilized video with rolling shutter
artifact rectification.
Video #3
Questions
I would ask you how have you obtained match_idx,match_x1,match_x2 data to calibrate the camera?
In Matlab, I used SIFT matching with VLFeat.
You can find it in our 'sift_track' function in the released software
mentioned above.
The SIFT feature is robust but a little bit slow.
You can also try BRISK or ORB.
Both of them can be found in OpenCV.
Please check this repo if you need a wrapper for Matlab:
https://github.com/kyamagu/mexopencv
COPYRIGHT NOTICE: All the documents on this server
have been submitted by their authors to scholarly journals or conferences
as indicated, for the purpose of non-commercial dissemination of
scientific work.
The manuscripts are put on-line to facilitate this purpose.
These manuscripts are copyrighted by the authors or the journals in which
they were published.
You may copy a manuscript for scholarly, non-commercial purposes, such
as research or instruction, provided that you agree to respect these
copyrights.
Slides on
Early Ideas for Rolling Shutter Artifact Reduction (March 7, 2012)
Last Updated 06/24/15.
