• 基于2D-3D双模态Transformer人体姿态估计

    2D-3D dual-modal pose transformer for human pose estimation

    • 人体姿态估计(Human Pose Estimation, HPE)旨在从图像和视频等输入数据中识别人体部位并构建人体骨架等表示。2D HPE方法常受遮挡和动作相似性问题的影响,而3D HPE方法能够提供更为丰富的空间信息。因此,提出了一种2D-3D双模态特征融合架构,整合2D和3D姿态特征,通过两种模态局部特征与全局特征的有效对齐,实现精准的人体关节点预测。该架构由局部区域对齐和2D-3D跨模态空间注意力两个核心模块组成,局部区域对齐模块通过一致的Patch划分和骨架映射,强化了2D和3D特征之间的模态匹配能力,从而捕获细粒度的关节点关系。2D-3D跨模态空间注意力通过查询(Q)、键(K)、值(V)交叉选择策略增强2D特征深度信息感知,同时保留2D特征空间关系,构建互补融合特征。在Human3.6M和MPI-INF-3DHP数据集上的实验结果表明:该模型在视频人体姿态估计任务中表现出色,不仅显著提高了预测精度,还有效平衡了计算成本与性能的关系。这种跨模态融合的设计为复杂动态场景下的人体姿态建模提供了一种高效、准确的解决方案。

       

      Abstract: Human Pose Estimation (HPE) aims to identify human body parts from input data such as images and videos and construct representations like human skeletons. 2D HPE methods are often affected by occlusion and the problem of similar actions, while 3D HPE methods can provide more abundant spatial information. Therefore, this paper proposes a 2D-3D dual-modal feature fusion architecture that integrates 2D and 3D pose features. Through the effective alignment of local and global features of the two modalities, it achieves accurate prediction of human joint points.This architecture consists of two core modules: the local region alignment module and the 2D-3D cross-modal spatial attention module. The local region alignment module strengthens the modal matching ability between 2D and 3D features through consistent Patch partitioning and skeleton mapping, thus capturing fine-grained joint relationships. The 2D-3D cross-modal spatial attention enhances the perception of depth information in 2D features through a cross-selection strategy of query (Q), key (K), and value (V), while preserving the spatial relationships of 2D features and constructing complementary fused features.Experimental results on the Human3.6M and MPI-INF-3DHP datasets show that the model performs excellently in the task of video human pose estimation. It not only significantly improves the prediction accuracy but also effectively balances the relationship between computational cost and performance. This cross-modal fusion design provides an efficient and accurate solution for human pose modeling in complex dynamic scenes.

       

    /

    返回文章
    返回