Transfer4D: A framework for frugal motion capture and deformation transfer

Shubh Maheshwari1, Rahul Narain2, Ramya Hebbalaguppe1
1TCS Research, India, 2Indian Institute of Technology Delhi, India
Paper Supplementary Poster Code Data Slides

Transfer4D transfers motion from a commodity depth sensor to a virtual model.

Summary

Animating a virtual character based on a real performance of an actor is a challenging task that currently requires expensive motion capture setups and additional effort by expert animators, rendering it accessible only to large production houses. The goal of our work is to democratize this task by developing a frugal alternative termed Transfer4D that uses only commodity depth sensors and further reduces animators' effort by automating the rigging and animation transfer process.

Our approach can transfer motion from an incomplete, single-view depth video to a semantically similar target mesh, unlike prior works that make a stricter assumption on the source to be noise-free and watertight.

  • We use non-rigid reconstruction to track motion from the depth sequence, and then we rig the source object using skinning decomposition. Finally, the rig is embedded into the target object for motion retargeting.
  • To handle sparse, incomplete videos from depth video inputs and variations between source and target objects, we propose to use skeletons as an intermediary representation between motion capture and transfer.
  • We propose a novel unsupervised skeleton extraction pipeline from single-view depth sequence that incorporates additional geometric information, resulting in superior performance in motion reconstruction and transfer in comparison to the contemporary methods and making our approach generic.

Video

How it works

Skeletonization

Curve skeleton

Extracted Curve Skeleton

(i) Notice that the extracted joint position from Local separators lies on the surface of the incomplete mesh. Our optimization aligns the joint position to the medial axis of the object.

Motion Skeleton

Interpolation end reference image.

(ii) To incorporate the motion information, each curve of the extracted curve skeleton is split into multiple bones

Our proposed skeletonization method, (a) does not require predifined template or markers, (b) works on single-view incomplete mesh sequence, and (c) uses the geometry and motion cues to estimate skeleton motion.

Comparisions

Static Skeleton Comparision

Interpolation end reference image.

Static skeletonisation methods do not incorporate motion information and produce temporally incoherent skeletons


Skeleton embedding Comparision (Same object)

Interpolation end reference image.

Compared to motion skeleton extracted from other methods, by incorporating structural cues, ours is more effective at embedding skeleton from incomplete mesh sequence.

BibTeX

@article{transfer4D,
  author    = {Maheshwari, Shubh and Narain, Rahul and Hebbalaguppe, Ramya},
  title     = {Transfer4D: A framework for frugal motion capture and deformation transfer},
  journal   = {CVPR},
  year      = {2023},
}