An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using RGBD Data

Bryan Willimon, Steven Hickson, Stan Birchfield, and Ian Walker

Abstract

We propose an algorithm that uses energy minimization to estimate the current configuration of a non-rigid object. Our approach utilizes an RGBD image to calculate corresponding SURF features, depth, and boundary information. We do not use predetermined features, thus enabling our system to operate on unmodified objects. Our approach relies on a 3D nonlinear energy minimization framework to solve for the configuration using a semi-implicit scheme. Results show various scenarios of dynamic posters and shirts in different configurations to illustrate the performance of the method. In particular, we show that our method is able to estimate the configuration of a textureless nonrigid object with no correspondences available.

Overview

Our algorithm consists of minimizing an energy equation with four terms:

Our goal is to locate the mesh that best explains the data while adhering to the smoothness prior. To find the configuration of minimum energy, we compute the partial derivative of the energy with respect to the vectors X_t, Y_t, and Z_t, and set the result to zero. In order to calculate the minimal solution, we employ the semi-implicit scheme used by Kass et al. (IJCV 1988). The smoothness term is treated explicitly, while the data terms are treated implicitly. Let V_t represent the mesh at iteration t and V_{t - 1} the mesh at the previous iteration, t - 1. Then the terms are given in the following table.

Smoothness Term

Correspondence Term

Depth Term

Boundary Term

Energy Minimization

Below are the three equations for X_t, Y_t, and Z_t that we set to zero and minimize.

Experimental Results

We captured RGBD video sequences of shirts and posters to test our proposed method’s ability to handle different nonrigid objects in a variety of scenarios. We also verified the contributions made by the novel depth and boundary energy terms to the accuracy of the estimated object configuration. For our experiments, the weights were set according to λ_C = 1.3, λ_B = 0.8, and λ_D = 0.6.

Contribution of Each Energy Term to Improve the Final Mesh

The improvement resulting from the various terms. From left to right:

(1) Texture mapped point cloud of RGBD input data
(2) The mesh resulting from just the smoothness and correspondence terms
(3) The mesh from all but the boundary term
(4) The mesh resulting from all the terms

From top to bottom:

Top view, front view, and 45° pan view

Textured Shirt Experiment

Four frames of a sequence in which a shirt partially occludes itself. Top: The estimated mesh (V) overlaid on the RGB color image. Bottom: a 45° pan view of the 3D mesh

Untextured Shirt Experiment

Five frames from a sequence in which a completely textureless shirt with no pattern is rotated out of the image plane, while the algorithm estimates the configuration of the shirt as a triangular mesh estimate. Even without data correspondences, the proposed energy function is able to accurately estimate the configuration of a dynamic non-rigid object.

Publications

Bryan Willimon, Steven Hickson, Stan Birchfield, and Ian Walker, An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using RGBD Data, Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), 2012.

Acknowledgements

This research was supported by the U.S. National Science Foundation under grants IIS-1017007, and IIS-0904116.