• 融合知识图谱与动态门控机制的作物病虫害多模态识别方法

    Multimodal recognition method for crop diseases and pests integrating knowledge graph and dynamic gating mechanism

    • 针对农业图文咨询中图像质量波动大、文本描述口语化所造成的语义模糊与模态对齐困难,提出一种融合领域知识图谱与动态门控机制的改进型多模态模型ALBEF-DKG。该模型包含两项核心创新:一是引入领域知识图谱为先验知识,以增强模型对专业术语与症状关联的语义理解能力;二是设计动态门控机制,根据图像质量指标自适应调整多模态特征的融合权重,从而提升对低质量图像输入的鲁棒性。在真实农业场景数据集AgriMix上的多维度实验结果表明:与CLIP、ALBEF等基准模型相比,所提模型在术语识别与语义对齐任务上均取得显著优势,展现出良好的鲁棒性与场景泛化能力。

       

      Abstract: To address the challenges of semantic ambiguity and modal misalignment caused by fluctuating image quality and colloquial text descriptions in agricultural image-text consultation, this paper proposes an enhanced multimodal model named ALBEF-DKG, which integrates a domain knowledge graph and a dynamic gating mechanism. The model incorporates two key innovations. First, it introduces a domain knowledge graph as prior knowledge to enhance the model's semantic understanding of professional terminology and symptom correlations. Second, it designs a dynamic gating mechanism that adaptively adjusts multimodal fusion weights based on image quality metrics, thereby improving robustness against low-quality image inputs. Extensive multi-dimensional experimental results on the real-world agricultural dataset AgriMix demonstrate that, compared with baseline models such as CLIP and ALBEF, the proposed model achieves significant advantages in both term recognition and semantic alignment tasks, exhibiting satisfactory robustness and strong scene generalization capability.

       

    /

    返回文章
    返回