• 融入孪生网络与自适应注意力的句子嵌入方法

    Sentence embedding method with siamese network and adaptive attention

    • 在句子嵌入研究中,当前主流方法是基于BERT的特殊分类标记CLS和基于Sentence-BERT的全局平均标记MEAN。但前者着重于句子的全局信息,表示能力存在不足,后者则存在过度表达的问题。对此,提出了一种融合孪生网络与自适应注意力的句子嵌入方法。首先,对BERT模型采用孪生网络结构,用来将句子对分别独立地映射为词嵌入向量。其次,将向量通过均衡聚合层得到词均标记MEAN-WORD,然后将此标记与词嵌入向量结合后再通过自适应注意力层得到动态标记DYNAMIC。最后,根据具体的下游任务对得到的标记结果进行相应的后处理以便应用。在数据集STS-B与SNLI、SST-2上的实验证明:与近年来多种句子嵌入方法相比,该方法能够生成更好的句子嵌入,具有更好的性能表现。另外,对该方法的动态标记进行了可视化分析,以便直观理解。

       

      Abstract: In the study of sentence embedding, the current mainstream methods are BERT-based special classification token CLS and Sentence-BERT-based global average token MEAN. But the former focus on the global information of sentences and the representation ability is insufficient, while the latter has the problem of over-expression. In order to solve the above problems, a sentence embedding method combining siamese network and adaptive attention is proposed. Firstly, the siamese network structure is adopted for the BERT model, which is used to map sentence pairs into word embedding vectors independently. Secondly, the vectors are used to obtain the word average token MEAN-WORD through the balanced aggregation layer, and then the token is combined with the word embedding vectors to obtain the dynamic token DYNAMIC through the adaptive attention layer. Finally, the token results are post-processed according to the specific downstream tasks for application. Experiments on datasets STS-B, SNLI and SST-2 show that compared with many sentence embedding methods in recent years, this method can generate better sentence embedding and has better performance. In addition, the dynamic token is visually analyzed for intuitive understanding.

       

    /

    返回文章
    返回