PrivacyExtractor: Automated Annotation and Classification of Privacy Concerns in User-generated App Reviews

Farah, Labiba; Islam, Md. Sakibul; Anam, MD Mohaymen Ul

dc.contributor.author	Farah, Labiba
dc.contributor.author	Islam, Md. Sakibul
dc.contributor.author	Anam, MD Mohaymen Ul
dc.date.accessioned	2025-03-11T05:48:43Z
dc.date.available	2025-03-11T05:48:43Z
dc.date.issued	2024-06-05
dc.identifier.citation	[1] A. Ant, J. Earp, Q. He, W. Stufflebeam, D. Bolchini, and C. Jensen, “Financial privacy policies and the need for standardization,” IEEE Security Privacy, vol. 2, pp. 36–45, Mar. 2004. doi: 10.1109/MSECP.2004.1281243. [2] A. R. Besmer, J. Watson, and M. S. Banks, “Investigating user perceptions of mobile app privacy: An analysis of user-submitted app reviews,” International Journal of Information Security and Privacy, vol. 14, no. 4, p. 74, 2020. [3] B. Bonné, S. T. Peddinti, I. Bilogrevic, and N. Taft, “Exploring decision making with android’s runtime permission dialogs using in-context surveys,” in Sym posium on Usable Privacy and Security, ser. SOUPS, USENIX Association, 2017, pp. 195–210. [4] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [5] A. Cavoukian, “The security-privacy paradox: Issues, misconceptions, and strate gies,” 2003. [6] N. Chen, J. Lin, S. C. H. Hoi, X. Xiao, and B. Zhang, “Ar-miner: Mining infor mative reviews for developers from mobile app marketplace,” New York, NY, USA: Association for Computing Machinery, 2014, isbn: 9781450327565. doi: 10 . 1145 / 2568225 . 2568263. [Online]. Available: https : / / doi . org / 10 . 1145/2568225.2568263. [7] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psy chological measurement, vol. 20, no. 1, pp. 37–46, 1960. [8] C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995. [9] M. Deng, K. Wuyts, R. Scandariato, B. Preneel, and W. Joosen, “A privacy threat analysis framework: Supporting the elicitation and fulfillment of privacy re quirements,” Requirements Engineering, vol. 16, no. 1, pp. 3–32, 2011. 35 [10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2019. [11] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd. Springer, 2009, isbn: 978-0-387- 84857-0. [12] D. W. Hosmer, S. Lemeshow, and R. X. Sturdivant, Applied Logistic Regression. John Wiley & Sons, 2013. [13] K. S. Jones, “A statistical interpretation of term specificity and its application in retrieval,” Journal of Documentation, vol. 28, no. 1, pp. 11–21, 1972. doi: 10. 1108/eb026526. [14] J. Jung, S. Han, and D. Wetherall, “Enhancing mobile application permissions with runtime feedback and constraints,” in Proceedings of the Second ACMWork shop on Security and Privacy in Smartphones and Mobile Devices, ser. SPSM ’12, ACM, 2012. [15] A. Karim, “Classification of google play store application reviews using ma chine learning,” Jul. 2020. doi: 10.20944/preprints202007.0646.v1. [16] P. G. Kelley, L. F. Cranor, and N. Sadeh, “Privacy as part of the app decision making process,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’13, Paris, France: ACM, 2013, pp. 3393–3402, isbn: 978-1-4503-1899-0. doi: 10 . 1145 / 2470654 . 2466466. [Online]. Avail able: https://doi.org/10.1145/2470654.2466466. [17] B. Liu, M. S. Andersen, F. Schaub, et al., “Follow my recommendations: A per sonalized privacy assistant for mobile app permissions,” in Twelfth Symposium on Usable Privacy and Security, ser. SOUPS 2016, USENIX Association, 2016, pp. 27–41. [18] Y. Liu, M. Ott, N. Goyal, et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [19] D. Mukherjee, A. Ahmadi, M. Vahdat Pour, and J. Reardon, “An empirical study on user reviews targeting mobile apps’ security & privacy,” 2020. arXiv: 2010. 06371 [cs.CR]. [20] P. Nema, P. Anthonysamy, N. Taft, and S. T. Peddinti, “Analyzing user perspec tives on mobile app privacy at scale,” 2020. [21] P. Nema, P. Anthonysamy, N. Taft, and S. T. Peddinti, “Analyzing user per spectives on mobile app privacy at scale,” New York, NY, USA: Association for 36 Computing Machinery, 2022, isbn: 9781450392211. doi: 10.1145/3510003. 3510079. [Online]. Available: https://doi.org/10.1145/3510003.3510079. [22] D. C. Nguyen, E. Derr, M. Backes, and S. Bugiel, “Short text, large effect: Mea suring the impact of user reviews on android app security & privacy,” in 2019 IEEE Symposium on Security and Privacy (SP), IEEE Press, 2019. [23] D. M. W. Powers, “Evaluation: From precision, recall and f-measure to roc, in formedness, markedness and correlation,” in Journal of Machine Learning Tech nologies, vol. 2, 2011, pp. 37–63. [24] A. Qureshi, M. Ahmad, S. Ullah, M. Yasir, F. Rustam, and I. Ashraf, “Perfor mance evaluation of machine learning models on large dataset of android ap plications reviews,” Multimedia Tools and Applications, vol. 82, pp. 1–23, Mar. 2023. doi: 10.1007/s11042-023-14713-6. [25] G. L. Scoccia, M. Autili, G. Stilo, and P. Inverardi, “An empirical study of privacy labels on the apple ios mobile app store,” New York, NY, USA: Association for Computing Machinery, 2022, isbn: 9781450393010. doi: 10.1145/3524613. 3527813. [Online]. Available: https://doi.org/10.1145/3524613.3527813. [26] M. Tavakoli, L. Zhao, A. Heydari, and G. Nenadic, “Extracting useful software development information from mobile application reviews: A survey of intelli gent mining techniques and tools,” Expert Systems with Applications, vol. 113, Jun. 2018. doi: 10.1016/j.eswa.2018.05.037. [27] C. Thompson, M. Johnson, S. Egelman, D. Wagner, and J. King, “When it’s bet ter to ask forgiveness than get permission: Attribution mechanisms for smart phone resources,” in Proceedings of the Ninth Symposium on Usable Privacy and Security, ser. SOUPS ’13, ACM, New York, NY, USA, 2013. [28] L. Tsai, P. Wijesekera, J. Reardon, et al., “Turtleguard: Helping android users apply contextual privacy preferences,” in Proceedings of the Thirteenth USENIX Conference on Usable Privacy and Security, vol. 1, 2017, pp. 145–162. [29] C. J. Van Rijsbergen, “The foundation of evaluation in retrieval systems,” Jour nal of Documentation, vol. 30, no. 4, pp. 365–373, 1974. [30] C. Wang, J. Jiang, M. Daneva, and M. Van Sinderen, “Coolted: A web-based collaborative labeling tool for the textual dataset,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2022, pp. 613–617. [31] J. Wei, K. Zou, Z. Chen, Y. Chen, L. Li, and P. Chen, “Eda: Easy data augmen tation techniques for boosting performance on text classification tasks,” in Pro ceedings of the 2019 Conference on Empirical Methods in Natural Language Pro 37 cessing and the 9th International Joint Conference on Natural Language Process ing (EMNLP-IJCNLP), 2019, pp. 6383–6389. [32] H. Xu, S. Gupta, M. B. Rosson, and J. Carroll, “Measuring mobile users’ con cerns for information privacy,” in International Conference on Information Sys tems, ser. ICIS 2012, Elsevier, 2012. [33] J. Zhang, J. Hua, Y. Chen, N. Niu, and C. Liu, “Mining user privacy concern topics from app reviews,” International Journal of Information Security and Pri vacy, vol. 14, no. 4, p. 74, 20	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/2379
dc.description	Supervised by Mr. Shohel Ahmed, Assistant Professor, Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT) Board Bazar, Gazipur, Bangladesh This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Software Engineering, 2024	en_US
dc.description.abstract	This thesis presents the development of a sophisticated tool designed to automatically annotate and classify privacy-related user reviews from the Google Play Store and a novel dataset. With privacy concerns becoming increasingly significant in the digital age, our tool aims to streamline the process of identifying and categorizing privacy- related issues and suggestions from user feedback. Through extensive experimenta- tion, we found that ensemble models, particularly those incorporating Random Forest classifiers, outperformed transformer-based models in accurately identifying and cat- egorizing these privacy issues. The tool demonstrated robust performance even with shorter datasets, indicating its potential applicability in real-world scenarios. Addi- tionally, the study highlights the importance of tailored data augmentation techniques for different machine learning algorithms. Our findings suggest that integrating this tool can provide developers with actionable insights to enhance the privacy aspects of their applications. Future research can explore the use of larger datasets and further optimization of data augmentation strategies to improve model performance.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.subject	Privacy, Requirement Engineering, Software Engineering, Tool, Security	en_US
dc.title	PrivacyExtractor: Automated Annotation and Classification of Privacy Concerns in User-generated App Reviews	en_US
dc.type	Thesis	en_US