Abstract:
In the study of sentence embedding, the current mainstream methods are BERT-based special classification token CLS and Sentence-BERT-based global average token MEAN. But the former focus on the global information of sentences and the representation ability is insufficient, while the latter has the problem of over-expression. In order to solve the above problems, a sentence embedding method combining siamese network and adaptive attention is proposed. Firstly, the siamese network structure is adopted for the BERT model, which is used to map sentence pairs into word embedding vectors independently. Secondly, the vectors are used to obtain the word average token MEAN-WORD through the balanced aggregation layer, and then the token is combined with the word embedding vectors to obtain the dynamic token DYNAMIC through the adaptive attention layer. Finally, the token results are post-processed according to the specific downstream tasks for application. Experiments on datasets STS-B, SNLI and SST-2 show that compared with many sentence embedding methods in recent years, this method can generate better sentence embedding and has better performance. In addition, the dynamic token is visually analyzed for intuitive understanding.