Abstract:
Human Pose Estimation (HPE) aims to identify human body parts from input data such as images and videos and construct representations like human skeletons. 2D HPE methods are often affected by occlusion and the problem of similar actions, while 3D HPE methods can provide more abundant spatial information. Therefore, this paper proposes a 2D-3D dual-modal feature fusion architecture that integrates 2D and 3D pose features. Through the effective alignment of local and global features of the two modalities, it achieves accurate prediction of human joint points.This architecture consists of two core modules: the local region alignment module and the 2D-3D cross-modal spatial attention module. The local region alignment module strengthens the modal matching ability between 2D and 3D features through consistent Patch partitioning and skeleton mapping, thus capturing fine-grained joint relationships. The 2D-3D cross-modal spatial attention enhances the perception of depth information in 2D features through a cross-selection strategy of query (
Q), key (
K), and value (
V), while preserving the spatial relationships of 2D features and constructing complementary fused features.Experimental results on the Human3.6M and MPI-INF-3DHP datasets show that the model performs excellently in the task of video human pose estimation. It not only significantly improves the prediction accuracy but also effectively balances the relationship between computational cost and performance. This cross-modal fusion design provides an efficient and accurate solution for human pose modeling in complex dynamic scenes.