A Hybrid Approach for Long Document Summarization: Optimizing Cross-attention and Abstractive Performance

Shishir, Tanvir Hossen; Hossain, M. Mohtasim; Rashad, Fahim Sadik

dc.contributor.author	Shishir, Tanvir Hossen
dc.contributor.author	Hossain, M. Mohtasim
dc.contributor.author	Rashad, Fahim Sadik
dc.date.accessioned	2025-03-10T05:49:46Z
dc.date.available	2025-03-10T05:49:46Z
dc.date.issued	2024-06-26
dc.identifier.citation	[1] A. Bajaj, P. Dangati, K. Krishna, et al., “Long document summarization in a low resource setting using pretrained language models,” in ACL, 2021. [2] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document trans former,” arXiv preprint arXiv:2004.05150, 2020. [3] S. Borgeaud, A. Mensch, J. Hoffmann, et al., “Improving language models by retrieving from trillions of tokens,” in Proceedings of the 2022 International Con ference on Learning Representations (ICLR), 2022. [Online]. Available: https: //arxiv.org/abs/2112.04426. [4] S. Cho, K. Song, X. Wang, F. Liu, and D. Yu, “Toward unifying text segmentation and long document summarization,” in EMNLP, 2022. [5] K. Choromanski, V. Likhosherstov, D. Dohan, et al., “Rethinking attention with performers,” arXiv preprint arXiv:2009.14794, 2020. [6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2019. [7] A. Drozdov, U. Alon, A. Veit, R. McDonald, and G. Durrett, “Efficient nearest neighbor language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [Online]. Available: https://arxiv.org/abs/2204.08481. [8] N. Giarelis, C. Mastrokostas, and N. Karacapilidis, “Abstractive vs. extractive summarization: An experimental review,”Applied Sciences, vol. 13, no. 13, p. 7620, 2023. doi: 10.3390/app13137620. [Online]. Available: https://www.mdpi. com/2076-3417/13/13/7620. [9] L. Huang, S. Cao, N. Parulian, H. Ji, and L. Wang, “Efficient attentions for long document summarization,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu man Language Technologies, 2021. 38 [10] B. Ivgi, S. Jain, M. Caccia, V. Ivanov, C. Raffel, and T. Linzen, “Fusion-in-decoder: Improved context fusion for long-context question answering,” arXiv preprint arXiv:2207.02108, 2022. [11] N. Kitaev, L. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” arXiv preprint arXiv:2001.04451, 2020. [12] W. Kryściński, B. McCann, C. Xiong, and R. Socher, “Evaluating the factual consistency of abstractive text summarization,” ArXiv, 2020. [13] K. Lee, M.-W. Chang, and K. Toutanova, “Latent retrieval for weakly supervised open domain question answering,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019. [Online]. Available: https://arxiv.org/abs/1906.00300. [14] K. Lee, M.-W. Chang, and K. Toutanova, “Latent retrieval for weakly supervised open domain question answering,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [15] M. Lewis, Y. Liu, N. Goyal, et al., “Bart: Denoising sequence-to-sequence pre training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. [16] X. Li, J. Xu, L. Huang, K. Xu, J. Gao, and B. Xu, “Simcls: Simulating contrastive learning in summarization,” ArXiv, 2022. [17] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 2004. [18] S. Liu, Y. Hu, F. Liu, L. Liu, and H. Wu, “Improving coherence and consistency in abstractive summarization with contrastive learning,” ArXiv, 2021. [19] Y. Liu, M. Ott, N. Goyal, et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [20] P. Manakul and M. Gales, “Long-span summarization via local attention and content selection,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021. [21] S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the sum mary! topic-aware convolutional neural networks for extreme summarization,” in ArXiv, 2018. [22] B. Pang, E. Nijkamp, W. Kryściński, S. Savarese, Y. Zhou, and C. Xiong, “Long document summarization with top-down and bottom-up inference,”ArXiv, 2022. 39 [23] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002. [24] D. Pu, Y.Wang, and V. Demberg, “Incorporating distributions of discourse struc ture for long document abstractive summarization,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. [25] L. Qin and B. Van Durme, “Retriever-reader: Generative retrieval models for machine reading comprehension,” arXiv preprint arXiv:2303.05129, 2023. [26] C. Raffel, N. Shazeer, A. Roberts, et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [27] Unlimiformer: Efficiently scaling transformers for long document summarization using knn indexing, [Available online], 2023. [28] A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [29] V. Vyas, J. Chen, J. Yan, T. Wang, H. Shu, and H. Wang, “Cluster-gcn: An ef ficient algorithm for training graph convolutional networks,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, 2020. [30] A. Wang, A. Fan, M. Lewis, and P.-S. Yang, “Longformer: Optimizing trans formers for long-context nlp tasks,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, 2021. [31] D. Yadav, J. Desai, and A. K. Yadav, “Automatic text summarization methods: A comprehensive review,” arXiv preprint arXiv:2204.01849, 2022. [Online]. Avail able: https://arxiv.org/abs/2204.01849. [32] J. Zhang, Y. Liu, P. Wei, X. Zhou, and J. Zhao, “Pegasus: Pre-training with ex tracted gap-sentences for abstractive summarization,” arXiv preprint arXiv:1912.08777, 2020. [33] T. Zhang, A. Sklar, and K. Toutanova, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:2005.03775, 2020. [34] Y. Zhang, M. Ding, L. Meng, J. Han, and K. Chang, “Text summarization with contrastive learning,” ArXiv, 2020.	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/2369
dc.description	Supervised by Mr. Ishmam Tashdeed, Lecturer, Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT) Board Bazar, Gazipur, Bangladesh This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Computer Science and Engineering, 2024	en_US
dc.description.abstract	This thesis addresses the challenge of long document summarization within the do main of Natural Language Processing (NLP), with a primary focus on enhancing fac tual consistency and text coherence while effectively managing extensive input con texts. To achieve these objectives, we propose a novel hybrid methodology that inte grates both extractive and abstractive summarization techniques. The methodology commences with an extractive phase, where a BART model is fine-tuned and aug mented with K-Nearest Neighbors (KNN) indexing, substantially increasing the con text length of the input and facilitating the retention of more comprehensive informa tion from the source document. Following the extractive phase, the abstractive phase leverages a pretrained BART model, coupled with contrastive learning, generating more coherent and factually accurate summaries. This two-stage approach ensures that the initial extraction captures essential information, which is then refined and articulated in the abstractive phase. Our experimental results demonstrate the suc cessful implementation of this methodology, with significant improvements observed in factual consistency and text coherence, as validated by higher BERTScore metrics. Despite the promising outcomes, we acknowledge that further human evaluation is necessary to fully validate our findings, which remains beyond the current research scope. Nonetheless, our research signifies a major advancement in long document summarization, presenting a strong framework that merges the benefits of extractive and abstractive techniques to generate high-quality summaries. This hybrid approach not only overcomes the shortcomings of individual methods but also paves the way for future progress in NLP-based summarization.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.subject	Knn, Extractive, Abstractive,Contrastive,Summarization	en_US
dc.title	A Hybrid Approach for Long Document Summarization: Optimizing Cross-attention and Abstractive Performance	en_US
dc.type	Thesis	en_US