Abstract:
To address the challenges of semantic ambiguity and modal misalignment caused by fluctuating image quality and colloquial text descriptions in agricultural image-text consultation, this paper proposes an enhanced multimodal model named ALBEF-DKG, which integrates a domain knowledge graph and a dynamic gating mechanism. The model incorporates two key innovations. First, it introduces a domain knowledge graph as prior knowledge to enhance the model's semantic understanding of professional terminology and symptom correlations. Second, it designs a dynamic gating mechanism that adaptively adjusts multimodal fusion weights based on image quality metrics, thereby improving robustness against low-quality image inputs. Extensive multi-dimensional experimental results on the real-world agricultural dataset AgriMix demonstrate that, compared with baseline models such as CLIP and ALBEF, the proposed model achieves significant advantages in both term recognition and semantic alignment tasks, exhibiting satisfactory robustness and strong scene generalization capability.