Zero-Shot Evaluation of Large Language Models on Medical Query Summarization Tasks

Chowdhury, Abrar; Sakib, Md. Sadman; Fahim, Farhan Shahriar

dc.contributor.author	Chowdhury, Abrar
dc.contributor.author	Sakib, Md. Sadman
dc.contributor.author	Fahim, Farhan Shahriar
dc.date.accessioned	2025-03-11T07:19:13Z
dc.date.available	2025-03-11T07:19:13Z
dc.date.issued	2024-07-03
dc.identifier.citation	[1] Q. A. R. Adib and S. B. Alam, “Bnclinical-sum: Benchmarking datasets for bangla long & short clinical dialogue summarization,” Ph.D. dissertation, Brac University, 2024. [2] S. Akter, A. S. Asa, M. P. Uddin, M. D. Hossain, S. K. Roy, and M. I. Afjal, “An extractive text summarization technique for bengali document (s) using k-means clustering algorithm,” in 2017 ieee international conference on imaging, vision & pattern recognition (icivpr), IEEE, 2017, pp. 1–6. [3] A. Ben Abacha and D. Demner-Fushman, “On the summarization of consumer health questions,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds., Flo rence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 2228–2234. DOI: 10.18653/v1/P19-1215. [Online]. Available: https://aclanthology. org/P19-1215. [4] A. Bhattacharjee, T. Hasan, W. U. Ahmad, and R. Shahriyar, “BanglaNLG and BanglaT5: Benchmarks and resources for evaluating low-resource natural language generation in Bangla,” in Findings of the Association for Computational Linguis tics: EACL 2023, A. Vlachos and I. Augenstein, Eds., Dubrovnik, Croatia: Asso ciation for Computational Linguistics, May 2023, pp. 726–735. DOI: 10.18653/ v1/2023.findings-eacl.54. [Online]. Available: https://aclanthology. org/2023.findings-eacl.54. [5] M. Gambhir and V. Gupta, “Recent automatic text summarization techniques: A survey,” Artificial Intelligence Review, vol. 47, pp. 1–66, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:24465182. [6] M. M. Haque, S. Pervin, A. Hossain, and Z. Begum, “Approaches and trends of automatic bangla text summarization: Challenges and opportunities,” IJTD, vol. 11, no. 4, pp. 1–17, 2020. DOI: 10.4018/IJTD.20201001.oa. [Online]. Available: https://doi.org/10.4018/IJTD.20201001.oa. [7] S. M. A. I. Hayat, A. Das, and M. M. Hoque, “Abstractive bengali text summa rization using transformer-based learning,” in 2023 6th International Conference 27 on Electrical Information and Communication Technology (EICT), 2023, pp. 1–6. DOI: 10.1109/EICT61409.2023.10427906. [8] Y. E. Işıkdemir, “Nlp transformers: Analysis of llms and traditional approaches for enhanced text summarization,” Eskişehir Osmangazi Üniversitesi Mühendislik ve Mimarlık Fakültesi Dergisi, vol. 32, no. 1, pp. 1140–1151, 2024. [9] M. Kabir, M. S. Islam, M. T. R. Laskar, M. T. Nayeem, M. S. Bari, and E. Hoque, “Benllmeval: A comprehensive evaluation into the potentials and pitfalls of large language models on bengali nlp,” arXiv preprint arXiv:2309.13173, 2023. [10] A. Khan, F. Kamal, M. A. Chowdhury, T. Ahmed, M. T. R. Laskar, and S. Ahmed, “BanglaCHQ-summ: An abstractive summarization dataset for medical queries in Bangla conversational speech,” in Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), F. Alam, S. Kar, S. A. Chowdhury, F. Sadeque, and R. Amin, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 85–93. DOI: 10.18653/v1/2023.banglalp-1.10. [Online]. Available: https://aclanthology.org/2023.banglalp-1.10. [11] Y. Kumar, K. A. Kaur, and S. Kaur, “Study of automatic text summarization ap proaches in different languages,” Artificial Intelligence Review, vol. 54, pp. 5897– 5929, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID: 233936657. [12] M. Moradi and N. Ghadiri, “Text summarization in the biomedical domain,” ArXiv, vol. abs/1908.02285, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:199472647. [13] M. F. Mridha, A. A. Lima, K. Nur, S. C. Das, M. Hasan, and M. M. Kabir, “A survey of automatic text summarization: Progress, process and challenges,” IEEE Access, vol. 9, pp. 156 043–156 070, 2021. DOI: 10.1109/ACCESS.2021.3129786. [14] S. R. Razu, T. Yasmin, T. B. Arif, et al., “Challenges faced by healthcare pro fessionals during the covid-19 pandemic: A qualitative inquiry from bangladesh,” Frontiers in public health, vol. 9, p. 647 315, 2021. [15] A. Sarkar and M. S. Hossen, “Automatic bangla text summarization using term fre quency and semantic similarity approach,” in 2018 21st International Conference of Computer and Information Technology (ICCIT), 2018, pp. 1–6. DOI: 10.1109/ ICCITECHN.2018.8631934. [16] A. Sarker, Y.-C. Yang, M. A. Al-garadi, and A. Abbas, “A light-weight text sum marization system for fast access to medical evidence,” Frontiers in Digital Health, vol. 2, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID: 227257712. 28 [17] M. A. I. Talukder, S. Abujar, A. K. M. Masum, F. Faisal, and S. A. Hossain, “Ben gali abstractive text summarization using sequence to sequence rnns,” in 2019 10th International Conference on Computing, Communication and Networking Tech nologies (ICCCNT), 2019, pp. 1–5. DOI: 10.1109/ICCCNT45670.2019.8944839. [18] A. Trewartha, N. Walker, H. Huo, et al., “Quantifying the advantage of domain specific pre-training on named entity recognition tasks in materials science,” Pat terns, vol. 3, p. 100 488, Apr. 2022. DOI: 10.1016/j.patter.2022.100488. [19] S. Yadav, D. Gupta, and D. Demner-Fushman, Chq-summ: A dataset for consumer healthcare question summarization, Jun. 2022. DOI: 10.48550/arXiv.2206. 06581.	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/2383
dc.description	Supervised by Dr. Md Moniruzzaman, Assistant Professor, Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT) Board Bazar, Gazipur, Bangladesh This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Software Engineering, 2024	en_US
dc.description.abstract	Online health consultations are becoming increasingly popular as a way for patients to discuss their medical health inquiries. In Bangladesh, patients are also using online health care solutions and thus providing medical queries in Bangla language. The COVID-19 pandemic has accelerated the use of these platforms, leading to a significant influx of questions and placing a heavy burden on the limited number of healthcare professionals available to respond. Text summarization offers a promising solution by condensing Bangla medical queries to highlight only the essential information needed for answers. This not only reduces the time healthcare professionals spend parsing unnecessary details but also serves as a crucial step toward developing automated medical question-answering systems. This research presents a comprehensive zero-shot evaluation of several state-ofthe-art Bangla and multilingual text generation models on the task of summarizing Bangla Consumer Health Questions (CHQs). The models we evaluated include BanglaT5, mT5, GPT-3.5, and GPT-4. The evaluation was conducted using ‘BanglaCHQ-Summ,’ which is currently the only available dataset specifically designed for summarizing Bangla CHQs, comprising 2350 pairs of questions and their corresponding summaries. The study aimed to determine which model performs best in terms of accurately and concisely summarizing Bangla medical queries. Among the models tested, GPT-4 demonstrated superior performance, achieving a BERTScore of 90.25%	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.subject	NLP, LLM, ROUGE, BLEU, CHQ	en_US
dc.title	Zero-Shot Evaluation of Large Language Models on Medical Query Summarization Tasks	en_US
dc.type	Thesis	en_US