Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation

arXiv:2503.06868v2 Announce Type: replace-cross Abstract: Existing long-text generation methods produce lengthy outputs from short inputs, leaving long-input-to-long-output generation underexplored. As input length increases, LLMs increasingly overlook information in the middle of the context--a limitation known as the "lost-in-the-middle" phenomenon--leading to inconsistent and incoherent outputs. To address this problem, we propose Retrieval-Augmented Long-Text Writer (RAL-Writer), a training-free framework consisting of a Planner that generates writing steps and a Writer that produces content according to these steps. The Writer jointly models semantic relevance and positional bias to compute importance scores, dynamically retrieve critical input segments, and strategically restate them. We also construct a benchmark dataset for long-input-to-long-output generation and introduce three evaluation metrics covering length, consistency, and quality. We evaluate RAL-Writer against comparable baselines on this dataset. The results demonstrate the effectiveness of our approach. Code is available at https://github.com/OnlyAR/RAL-Writer.