Abstract:
To address the problems of semantic fragmentation, strong structural dependency, and sensitivity to model performance in existing code chunking strategies for long code documents, a semantic loss quantification model for code chunking is proposed, and the optimization objective of chunking strategies is formally defined. On this basis, a code decomposition strategy based on large language models is proposed, which transforms traditional segmented code chunking into restructured code decomposition to enhance semantic integrity and reduce semantic loss; the interference of code comments on semantic representation and the chunking process is also analyzed. Experimental results on the APPS and CodeSearchNet-Python datasets show that the proposed code decomposition strategy improves NDCG@1 in semantic retrieval by up to approximately 10 percentage points, maintains stable performance under comment interference, adapts well to lightweight models, and significantly reduces dependence on prompts. The proposed strategy provides an effective method for high-quality chunking in retrieval-augmented code generation.