Efficient Ensemble-Based Approaches to Personal Health Mention Detection

Nower, Nuzhat; Kamal, Fida; Khan, Alvi Aveen

dc.contributor.author	Nower, Nuzhat
dc.contributor.author	Kamal, Fida
dc.contributor.author	Khan, Alvi Aveen
dc.date.accessioned	2024-09-02T05:53:33Z
dc.date.available	2024-09-02T05:53:33Z
dc.date.issued	2023-04-30
dc.identifier.citation	[1] Tasnim Ahmed, Mohsinul Kabir, Shahriar Ivan, Hasan Mahmud, and Kamrul Hasan. Am i being bullied on social media? an ensemble approach to cat egorize cyberbullying. In 2021 IEEE International Conference on Big Data (Big Data), pages 2442–2453. IEEE, 2021. doi: 10.1109/BigData52589.2021. 9671594. [2] Tasnim Ahmed, Shahriar Ivan, Mohsinul Kabir, Hasan Mahmud, and Kamrul Hasan. Performance analysis of transformer-based architectures and their ensembles to detect trait-based cyberbullying. Social Network Analysis and Mining, 12(1):1–17, 2022. doi: 10.1007/s13278-022-00934-4. [3] Eiji Aramaki, Sachiko Maskawa, and Mizuki Morita. Twitter catches the flu: detecting influenza epidemics using twitter. In Proceedings of the 2011 Conference on empirical methods in natural language processing, pages 1568– 1576, 2011. [4] Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/ S17-2126. [5] Rhys Biddle, Aditya Joshi, Shaowu Liu, Cecile Paris, and Guandong Xu. Leveraging sentiment distributions to distinguish figurative from literal health reports on twitter. In Proceedings of The Web Conference 2020, pages 1217– 1227, 2020. doi: 10.1145/3366423.3380198. [6] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. “ O’Reilly Media, Inc.”, 2009. 53 [7] Liangzhe Chen, KSM Tozammel Hossain, Patrick Butler, Naren Ramakrish nan, and B Aditya Prakash. Syndromic surveillance of flu on twitter using weakly supervised temporal topic models. Data mining and knowledge dis covery, 30(3):681–710, 2016. doi: 10.1007/s10618-015-0434-x. [8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. doi: 10.48550/arXiv.1604.06174. [9] Rumi Chunara, Jason R Andrews, and John S Brownstein. Social and news media enable estimation of epidemiological patterns early in the 2010 haitian cholera outbreak. The American journal of tropical medicine and hygiene, 86 (1):39, 2012. doi: 10.4269/ajtmh.2012.11-0597. [10] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. doi: 10.48550/arXiv.2003.10555. [11] Omar B Da’ar, Faisel Yunus, Nassif Md Hossain, and Mowafa Househ. Impact of twitter intensity, time, and location on message lapse of bluebird’s pursuit of fleas in madagascar. Journal of Infection and Public Health, 10(4):396–402, 2017. doi: 10.1016/j.jiph.2016.06.011. [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019. doi: 10.18653/ v1/N19-1423. [13] Syed Mohammed Sartaj Ekram, Adham Arik Rahman, Md Sajid Altaf, Mo hammed Saidul Islam, Mehrab Mustafy Rahman, Md Mezbaur Rahman, Md Azam Hossain, and Abu Raihan Mostofa Kamal. Banglarqa: A bench mark dataset for under-resourced bangla language reading comprehension 54 based question answering with diverse question-answer types. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2518– 2532, 2022. [14] Samah Jamal Fodeh, Mohammed Al-Garadi, Osama Elsankary, Jeanmarie Perrone, William Becker, and Abeed Sarker. Utilizing a multi-class classi fication approach to detect therapeutic and recreational misuse of opioids on twitter. Computers in biology and medicine, 129:104132, 2021. doi: 10.1016/j.compbiomed.2020.104132. [15] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007. github.io/OpenWebTextCorpus, 2019. Accessed: 2022-11-22. [16] Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. doi: 10.48550/arXiv.1706.02677. [17] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Con ference on Learning Representations, 2020. [18] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving de berta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021. doi: 10.48550/arXiv.2111. 09543. [19] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems, 30:1731–1741, 2017. [20] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International con ference on machine learning, pages 448–456. PMLR, 2015. 55 [21] Adith Iyer, Aditya Joshi, Sarvnaz Karimi, Ross Sparks, and Cecile Paris. Figurative usage detection of symptom words to improve personal health mention detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1142–1147, 2019. doi: 10.18653/v1/P19-1108. [22] Keyuan Jiang, Ricardo Calix, and Matrika Gupta. Construction of a personal experience tweet corpus for health surveillance. In Proceedings of the 15th workshop on biomedical natural language processing, pages 128–135, 2016. doi: 10.18653/v1/W16-2917. [23] Keyuan Jiang, Shichao Feng, Qunhao Song, Ricardo A Calix, Matrika Gupta, and Gordon R Bernard. Identifying tweets of personal health experience through word embedding and lstm neural network. BMC bioinformatics, 19 (8):67–74, 2018. doi: 10.1186/s12859-018-2198-y. [24] Aditya Joshi, Sarvnaz Karimi, Ross Sparks, C´ecile Paris, and C Raina Mac intyre. Survey of text-based epidemic intelligence: A computational linguis tics perspective. ACM Computing Surveys (CSUR), 52(6):1–19, 2019. doi: 10.1145/3361141. [25] Aditya Joshi, Ross Sparks, Sarvnaz Karimi, Sheng-Lun Jason Yan, Abrar Ah mad Chughtai, Cecile Paris, and C Raina MacIntyre. Automated monitoring of tweets for early detection of the 2014 ebola epidemic. PloS one, 15(3): e0230322, 2020. doi: 10.1371/journal.pone.0230322. [26] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tom´aˇs Mikolov. Bag ´ of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, 2017. [27] Mohsinul Kabir, Tasnim Ahmed, Md Bakhtiar Hasan, Md Tahmid Rah man Laskar, Tarun Kumar Joarder, Hasan Mahmud, and Kamrul Hasan. Deptweet: A typology for social media texts to detect depression severities. 56 Computers in Human Behavior, 139:107503, 2023. doi: 10.1016/j.chb.2022. 107503. [28] Payam Karisani and Eugene Agichtein. Did you really just have a heart attack? towards robust detection of personal health mentions in social media. In Proceedings of the 2018 World Wide Web Conference, pages 137–146, 2018. doi: 10.1145/3178876.3186055. [29] Makoto P Kato, Kazuaki Kishida, Noriko Kando, Tetsuya Saka, and Mark Sanderson. Report on ntcir-12: The twelfth round of nii testbeds and commu nity for information access research. In ACM SIGIR Forum, volume 50, pages 18–27. ACM New York, NY, USA, 2017. doi: 10.1145/3053408.3053413. [30] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyan skiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016. doi: 10.48550/arXiv.1609.04836. [31] Alvi Aveen Khan, Fida Kamal, Nuzhat Nower, Tasnim Ahmed, and Tareque Mohmud Chowdhury. An evaluation of transformer-based models in personal health mention detection. In 2022 25th International Conference on Computer and Information Technology (ICCIT), pages 1–6. IEEE, 2022. doi: 10.1109/ICCIT57492.2022.10054937. [32] Pervaiz Iqbal Khan, Imran Razzak, Andreas Dengel, and Sheraz Ahmed. Improving personal health mention detection on twitter using permutation based word representation learning. In International Conference on Neu ral Information Processing, pages 776–785. Springer, 2020. doi: 10.1007/ 978-3-030-63830-6 65. [33] Pervaiz Iqbal Khan, Imran Razzak, Andreas Dengel, and Sheraz Ahmed. A novel approach to train diverse types of language models for health mention classification of tweets. In Artificial Neural Networks and Machine Learning – ICANN 2022, pages 136–147, 2022. doi: 10.1007/978-3-031-15931-2 12. 57 [34] Pervaiz Iqbal Khan, Imran Razzak, Andreas Dengel, and Sheraz Ahmed. Per formance comparison of transformer-based models on twitter health mention classification. IEEE Transactions on Computational Social Systems, pages 1–10, 2022. doi: 10.1109/TCSS.2022.3143768. [35] Pervaiz Iqbal Khan, Shoaib Ahmed Siddiqui, Imran Razzak, Andreas Dengel, and Sheraz Ahmed. Improving health mention classification of social media content using contrastive adversarial training. IEEE Access, 10:87900–87910, 2022. doi: 10.1109/ACCESS.2022.3200159. [36] Anders Krogh and John Hertz. A simple weight decay can improve gen eralization. Advances in neural information processing systems, 4:950–957, 1991. [37] Alex Lamb, Michael Paul, and Mark Dredze. Separating fact from fear: Tracking flu infections on twitter. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 789–795, 2013. [38] Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque. Contextu alized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5505–5514, 2020. [39] Joffrey L Leevy, Taghi M Khoshgoftaar, Richard A Bauder, and Naeem Seliya. A survey on addressing high-class imbalance in big data. Journal of Big Data, 5(1):1–30, 2018. doi: 10.1186/s40537-018-0151-6. [40] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. doi: 10.48550/arXiv.1907.11692. [41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. 58 [42] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30:4765– 4774, 2017. [43] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estima tion of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. doi: 10.48550/arXiv.1301.3781. [44] Saif Mohammad. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th annual meet ing of the association for computational linguistics (volume 1: Long papers), pages 174–184, 2018. doi: 10.18653/v1/P18-1017. [45] Martin M¨uller, Marcel Salath´e, and Per E Kummervold. Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. arXiv preprint arXiv:2005.07503, 2020. doi: 10.48550/arXiv.2005.07503. [46] Sebastian Nagel. Cc-news. https://commoncrawl.org/2016/10/ news-dataset-available/, 2016. Accessed: 2022-11-22. [47] Usman Naseem, Matloob Khushi, Jinman Kim, and Adam G Dunn. Rhmd: A real-world dataset for health mention classification on reddit. IEEE Transactions on Computational Social Systems, pages 1–10, 2022. doi: 10.1109/TCSS.2022.3186883. [48] Usman Naseem, Jinman Kim, Matloob Khushi, and Adam G Dunn. Iden tification of disease or symptom terms in reddit to improve health mention classification. In Proceedings of the ACM Web Conference 2022, pages 2573– 2581, 2022. doi: 10.1145/3485447.3512129. [49] Usman Naseem, Jinman Kim, Matloob Khushi, and Adam G Dunn. Robust identification of figurative language in personal health mentions on twitter. IEEE Transactions on Artificial Intelligence, pages 1–1, 2022. doi: 10.1109/ TAI.2022.3175469. 59 [50] Usman Naseem, Byoung Chan Lee, Matloob Khushi, Jinman Kim, and Adam Dunn. Benchmarking for public health surveillance tasks on social media with a domain-specific pretrained language model. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 22–31, 2022. doi: 10.18653/v1/2022.nlppower-1.3. [51] Dat Quoc Nguyen, Thanh Vu, and Anh-Tuan Nguyen. Bertweet: A pre trained language model for english tweets. In Proceedings of the 2020 Confer ence on Empirical Methods in Natural Language Processing: System Demon strations, pages 9–14, 2020. doi: 10.18653/v1/2020.emnlp-demos.2. [52] Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Long Doan, et al. Wnut-2020 task 2: Identification of informative covid-19 english tweets. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 314–318, 2020. doi: 10.18653/v1/2020.wnut-1.41. [53] Robert T Olszewski. Bayesian classification of triage diagnoses for the early detection of epidemics. pages 412–416, 2003. [54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026–8037, 2019. [55] Michael Paul and Mark Dredze. You are what you tweet: Analyzing twitter for public health. In Proceedings of the International AAAI Conference on Web and Social Media, volume 5, pages 265–272, 2011. doi: 10.1609/icwsm. v5i1.14137. [56] Michael J Paul and Mark Dredze. Social monitoring for public health. Syn thesis Lectures on Information Concepts, Retrieval, and Services, 9(5):1–183, 2017. doi: 10.1007/978-3-031-02311-8. 60 [57] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532– 1543, 2014. doi: 10.3115/v1/D14-1162. [58] Yao Qian, Yuchen Fan, Wenping Hu, and Frank K Soong. On the train ing aspects of deep neural network (dnn) for parametric tts synthesis. In 2014 IEEE International Conference on Acoustics, Speech and Signal Pro cessing (ICASSP), pages 3829–3833. IEEE, 2014. doi: 10.1109/ICASSP.2014. 6854318. [59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. Ope nAI blog, 1(8):9, 2019. [60] Lior Rokach. Ensemble methods for classifiers. In Data mining and knowl edge discovery handbook, pages 957–980. Springer, 2005. doi: 10.1007/ 0-387-25465-X 45. [61] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine trans lation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. Association for Computational Linguistics (ACL), 2016. doi: 10.18653/v1/P16-1162. [62] Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. doi: 10.48550/arXiv.1806.02847. [63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30:5998–6008, 2017. [64] Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. Advances in neural information processing systems, 26:351– 359, 2013. 61 [65] Chen-Kai Wang, Onkar Singh, Zhao-Li Tang, and Hong-Jie Dai. Using a recurrent neural network model for classification of tweets conveyed influenza related information. In Proceedings of the International Workshop on Digital Disease Detection Using Social Media 2017 (DDDSM-2017), pages 33–38, 2017. [66] Davy Weissenbacher, Abeed Sarker, Arjun Magge, Ashlynn Daughton, Karen O’Connor, Michael Paul, and Graciela Gonzalez. Overview of the fourth social media mining for health (smm4h) shared tasks at acl 2019. In Proceedings of the fourth social media mining for health applications (# SMM4H) workshop & shared task, pages 21–30, 2019. doi: 10.18653/v1/W19-3203. [67] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhut dinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32:5753–5763, 2019. [68] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceed ings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489, 2016. doi: 10.18653/v1/N16-1174. [69] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015. doi: 10.1109/ICCV.2015.11.	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/2148
dc.description	Supervised by Mr. Tareque Mohmud Chowdhury, Assistant Professor, Co-Supervisor Ms. Tasnim Ahmed Lecturer, Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.description.abstract	An important component of public health surveillance is the analysis of personal health-related posts on social media platforms, known as Personal Health Mention (PHM) Detection. PHM detection is essential to quickly detecting epidemics, allowing health organisations to prepare themselves and warn the general public to take precautionary steps. One of the key complexities of this task is the informal nature of the language used in social media, which makes it difficult to understand their context. The architectures that are capable of discerning context are also computationally expensive to use and often mistrusted in the medical community due to their decision-making strategies being hidden behind a black box. In this thesis, we address each of these issues separately. We introduce four transformer-based ensemble architectures that have not been previously explored in the PHM domain and show that these architectures can achieve state-of-the art results across the domain. Combined with computationally efficient training mechanisms, our architectures also use fewer resources than existing ones. Addi tionally, we provide methods to explain the outputs produced by the architectures in order to address the concerns related to explainability.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.title	Efficient Ensemble-Based Approaches to Personal Health Mention Detection	en_US
dc.type	Thesis	en_US