Overview of our motion pose and shape network (MPS-Net). MPS-Net estimates pose, shape, and camera parameters Θ in the video sequence based on the static feature extractor, temporal encoder, temporal feature integration, and SMPL parameter regressor to generate 3D human pose and shape.

Learning to capture human motion is essential to 3D human pose and shape estimation from monocular video. However, the existing methods mainly rely on recurrent or convolutional operation to model such temporal information, which limits the ability to capture non-local context relations of human motion. To address this problem, we propose a motion pose and shape network (MPS-Net) to effectively capture humans in motion to estimate accurate and temporally coherent 3D human pose and shape from a video. Specifically, we first propose a motion continuity attention (MoCA) module that leverages visual cues observed from human motion to adaptively recalibrate the range that needs attention in the sequence to better capture the motion continuity dependencies. Then, we develop a hierarchical attentive feature integration (HAFI) module to effectively combine adjacent past and future feature representations to strengthen temporal correlation and refine the feature representation of the current frame. By coupling the MoCA and HAFI modules, the proposed MPS-Net excels in estimating 3D human pose and shape in the video. Though conceptually simple, our MPS-Net not only outperforms the state-of-the-art methods on the 3DPW, MPI-INF-3DHP, and Human3.6M benchmark datasets, but also uses fewer network parameters.

	[Paper] CVPR 2022 [Code]

We visualize the 3D human body estimated by our MPS-Net from different viewpoints. The results show that our MPS-Net is able to estimate the correct global body rotation.

Qualitative comparison of TCMR [6] (left) and our MPS-Net (right) on the challenging in-the-wild 3DPW [37] and MPI-INF-3DHP [27] datasets.

Qualitative comparison of TCMR [6] (middle row) and our MPS-Net (bottom row) on the challenging in-the-wild 3DPW [37] dataset.

Qualitative comparison of TCMR [6] (middle row) and our MPS-Net (bottom row) on the challenging in-the-wild 3DPW [37] dataset.

Qualitative comparison of TCMR [6] (middle row) and our MPS-Net (bottom row) on the MPI-INF-3DHP [27] dataset.

Qualitative comparison of TCMR [6] (middle row) and our MPS-Net (bottom row) on the Human3.6M [16] dataset.

Table 1. Evaluation of state-of-the-art video-based methods on 3DPW [37], MPI-INF-3DHP [27], and Human3.6M [16] datasets. Following Choi et al. [6], all methods are trained on the training set including 3DPW, but do not use the Human3.6M SMPL parameters obtained from Mosh [23]. The number of input frames follows the original protocol of each method.

Table 2. Comparison of the number of network parameters, FLOPs, and model size.

Table 3. Ablation study for different modules of the MPS-Net on the 3DPW [37] dataset. The training and evaluation settings are the same as the experiments on the 3DPW dataset in Table 1.

Table 4. Evaluation of state-of-the-art single image-based and video-based methods on the 3DPW [37] dataset. All methods do not use 3DPW for training.