ChartSumm: A large scale benchmark for Chart to Text Summarization

Rahman, Raian; Hasan, Rizvi; Farhad, Abdullah Al

dc.contributor.author	Rahman, Raian
dc.contributor.author	Hasan, Rizvi
dc.contributor.author	Farhad, Abdullah Al
dc.date.accessioned	2023-04-28T06:59:33Z
dc.date.available	2023-04-28T06:59:33Z
dc.date.issued	2022-05-30
dc.identifier.citation	[1] L. Gong, J. Crego, and J. Senellart, “Enhanced transformer model for data-totext generation,” in Proceedings of the 3rd Workshop on Neural Generation and Translation, Nov. 2019. [2] K. Chen, F. Li, B. Hu, W. Peng, Q. Chen, H. Yu, and Y. Xiang, “Neural datato-text generation with dynamic content planning,” Knowledge-Based Systems, 2021. [3] J. Sreevalsan-Nair, K. Dadhich, and S. C. Daggubati, “Tensor fields for data extraction from chart images: bar charts and scatter plots,” in Topological Methods in Data Analysis and Visualization VI. Springer, 2021. [4] J. Luo, Z. Li, J. Wang, and C.-Y. Lin, “Chartocr: Data extraction from charts images via a deep hybrid framework,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021. [5] F. Zhou, Y. Zhao, W. Chen, Y. Tan, Y. Xu, Y. Chen, C. Liu, and Y. Zhao, “Reverseengineering bar charts using neural networks,” Journal of Visualization, vol. 24, no. 2, 2021. [6] K. Dadhich, S. C. Daggubati, and J. Sreevalsan-Nair, “Barchartanalyzer: Digitizing images of bar charts.” in BarChartAnalyzer, 2021. [7] J. Obeid and E. Hoque, “Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model,” arXiv preprint arXiv:2010.09142, 2020. [8] J. Zhu, J. Ran, R. K.-w. Lee, K. Choo, and Z. Li, “Autochart: A dataset for chartto-text generation task,” arXiv preprint arXiv:2108.06897, 2021. [9] T.-Y. Hsu, C. L. Giles, and T.-H. Huang, “SciCap: Generating captions for scientific figures,” in Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana, Dominican Republic: Association for 48 Computational Linguistics, Nov. 2021, pp. 3258–3264. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.277 [10] S. Kanthara, R. T. K. Leong, X. Lin, A. Masry, M. Thakkar, E. Hoque, and S. Joty, “Chart-to-text: A large-scale benchmark for chart summarization,” arXiv preprint arXiv:2203.06486, 2022. [11] J. Obeid and E. Hoque, “Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model,” in Proceedings of the 13th International Conference on Natural Language Generation. Dublin, Ireland: Association for Computational Linguistics, Dec. 2020, pp. 138–147. [Online]. Available: https://aclanthology.org/2020.inlg-1.20 [12] D. H. Kim, E. Hoque, and M. Agrawala, Answering Questions about Charts and Generating Visual Explanations. New York, NY, USA: Association for Computing Machinery, 2020, p. 1–13. [Online]. Available: https: //doi.org/10.1145/3313831.3376467 [13] G. Carenini, C. Conati, E. Hoque, and B. Steichen, “User task adaptation in multimedia presentations.” in UMAP Workshops. Citeseer, 2013. [14] L. Ferres, G. Lindgaard, L. Sumegi, and B. Tsuji, “Evaluating a tool for improving accessibility to charts and graphs,” ACM Trans. Comput.- Hum. Interact., vol. 20, no. 5, nov 2013. [Online]. Available: https: //doi.org/10.1145/2533682.2533683 [15] S. Carberry, S. Elzer, and S. Demir, “Information graphics: an untapped resource for digital libraries,” in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006. [16] Z. Li, M. Stagitis, S. Carberry, and K. F. McCoy, “Towards retrieving relevant information graphics,” in Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval, 2013, pp. 789–792. [17] E. Reiter, S. G. Sripada, J. Hunter, J. Yu, and I. P. Davy, “Choosing words in computer-generated weather forecasts,” Artif. Intell., vol. 167, pp. 137–169, 2005. [18] M. Fasciano and G. Lapalme, “Intentions in the coordinated generation of graphics and text from tabular data,” Knowledge and Information Systems, vol. 2, no. 3, pp. 310–339, 2000. [19] V. O. Mittal, J. D. Moore, G. Carenini, and S. Roth, “Describing complex charts in natural language: A caption generation system,” Computational 49 Linguistics, vol. 24, no. 3, pp. 431–467, 1998. [Online]. Available: https://aclanthology.org/J98-3004 [20] “Quillbot,” https://quillbot.com/. [Online]. Available: https://quillbot.com/ [21] “Wordsmith,” https://lexically.net/wordsmith/index.html. [Online]. Available: https://lexically.net/wordsmith/index.html [22] H. Mei, M. Bansal, and M. R. Walter, “What to talk about and how? selective generation using lstms with coarse-to-fine alignment,” arXiv preprint arXiv:1509.00838, 2015. [23] J. Zhu, J. Ran, R. K.-W. Lee, Z. Li, and K. Choo, “AutoChart: A dataset for chart-to-text generation task,” in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). Held Online: INCOMA Ltd., Sep. 2021, pp. 1636–1644. [Online]. Available: https://aclanthology.org/2021.ranlp-main.183 [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017. [25] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013. [26] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014. [27] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. [28] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with lstm,” Neural computation, 2000. [29] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoderdecoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014. [30] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. [31] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, 2019. 50 [32] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. [33] S. Wiseman, S. M. Shieber, and A. M. Rush, “Challenges in data-to-document generation,” arXiv preprint arXiv:1707.08052, 2017. [34] R. Puduppully, L. Dong, and M. Lapata, “Data-to-text generation with entity modeling,” arXiv preprint arXiv:1906.03221, 2019. [35] D. Bahdanau, D. Serdyuk, P. Brakel, N. R. Ke, J. Chorowski, A. Courville, and Y. Bengio, “Task loss estimation for sequence prediction,” arXiv preprint arXiv:1511.06456, 2015. [36] C. Gulcehre, S. Ahn, R. Nallapati, B. Zhou, and Y. Bengio, “Pointing the unknown words,” arXiv preprint arXiv:1603.08148, 2016. [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. [38] R. Lebret, D. Grangier, and M. Auli, “Neural text generation from structured data with application to the biography domain,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1203–1213. [Online]. Available: https://aclanthology.org/D16-1128 [39] C. Thomson, E. Reiter, and S. Sripada, “SportSett:basketball - a robust and maintainable data-set for natural language generation,” in Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation. Santiago de Compostela, Spain: Association for Computational Lingustics, Sep. 2020, pp. 32–40. [Online]. Available: https://aclanthology.org/ 2020.intellang-1.4 [40] “knoema,” https://knoema.com/atlas. [Online]. Available: https://knoema.com/ atlas [41] “statista,” https://www.statista.com/. [Online]. Available: https://www.statista. com/ [42] L. Gong, J. M. Crego, and J. Senellart, “Enhanced transformer model for data-totext generation,” in Proceedings of the 3rd Workshop on Neural Generation and Translation, 2019, pp. 148–156. 51 [43] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. [44] M. Post, “A call for clarity in reporting BLEU scores,” in Proceedings of the Third Conference on Machine Translation: Research Papers. Brussels, Belgium: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://aclanthology.org/W18-6319 [45] T. Sellam, D. Das, and A. Parikh, “BLEURT: Learning robust metrics for text generation,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 7881–7892. [Online]. Available: https://aclanthology.org/2020. acl-main.704 [46] L. Azzopardi, M. Girolami, and K. van Risjbergen, “Investigating the relationship between language model perplexity and ir precision-recall measures,” ser. SIGIR ’03. New York, NY, USA: Association for Computing Machinery, 2003. [Online]. Available: https://doi.org/10.1145/860435.860505 [47] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/1867
dc.description	Supervised by Mr. Md. Hamjajul Ashmafee, Lecturer, Department of Computer Science and Engineering(CSE), Islamic University of Technology (IUT) Board Bazar, Gazipur-1704, Bangladesh. This thesis is submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Science and Engineering, 2022.	en_US
dc.description.abstract	Information visualization such as bar- and line-charts are quite popular for understanding large tabular data. But, interpreting information solely with different visualization techniques can also be difficult due to different reasons like visual impairment or the requirement of prior domain knowledge to understand the chart. Automatic "chart to text summarization" can be promising and effective tool for providing accessibility as well as precised insights of chart data in natural language. In spite of having a good potential, there have not been a lot of works on chart to text summarization making it a low resource task. Scarcity of large scale datasets for chart to text summarization is one of the reason behind this. The human written descriptions in the available dataset also contains information beyond the knowledge of the chart making it difficult for us to have an unbiased evaluation. In our thesis, we propose ChartSumm a large scale dataset for chart to text summarization consisting of 84,363 charts along with their metadata and descriptions. We also propose two test sets: test-e and test-h for evaluating the performance of the trained models available in this domain. Our experiment shows that a T5 model trained on our dataset has achieved BLEU score of 75.72 in test-e set and 64.78 in test-h set. From our analysis we can conclude that large language models like T5 and BART can generate short precised deception from given chart metadata.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur, Bangladesh	en_US
dc.subject	Chart summarization; Natural language generation; Low resource task	en_US
dc.title	ChartSumm: A large scale benchmark for Chart to Text Summarization	en_US
dc.type	Thesis	en_US