• 一种基于大模型的代码检索分块策略

    A code retrieval chunking strategy based on LLM

    • 针对现有代码分块策略处理长代码文档时存在语义割裂、结构依赖性强及对模型性能敏感等问题,提出一种代码分块语义损失量化模型,并对分块策略的优化目标进行形式化定义。在此基础上,提出基于大语言模型的代码分解策略,将传统分割式代码分块转化为重组式代码分解,以增强分块语义完整度并降低语义损失;同时分析代码注释对语义表征与分块过程的干扰。在APPS与CodeSearchNet-Python数据集上的实验结果表明:所提代码分解策略使语义检索的NDCG@1最高提升约10个百分点,在注释干扰下性能保持稳定,对轻量级模型具有良好适应性,且对提示词的依赖性显著降低。该策略可为代码检索增强生成中的高质量分块提供有效方法。

       

      Abstract: To address the problems of semantic fragmentation, strong structural dependency, and sensitivity to model performance in existing code chunking strategies for long code documents, a semantic loss quantification model for code chunking is proposed, and the optimization objective of chunking strategies is formally defined. On this basis, a code decomposition strategy based on large language models is proposed, which transforms traditional segmented code chunking into restructured code decomposition to enhance semantic integrity and reduce semantic loss; the interference of code comments on semantic representation and the chunking process is also analyzed. Experimental results on the APPS and CodeSearchNet-Python datasets show that the proposed code decomposition strategy improves NDCG@1 in semantic retrieval by up to approximately 10 percentage points, maintains stable performance under comment interference, adapts well to lightweight models, and significantly reduces dependence on prompts. The proposed strategy provides an effective method for high-quality chunking in retrieval-augmented code generation.

       

    /

    返回文章
    返回