Bangla Sign Language Generation From Videos

Sameen, Ahmed Shadab; Soiree, Sumaya Sanjana; Shahriar, Sajid

dc.contributor.author	Sameen, Ahmed Shadab
dc.contributor.author	Soiree, Sumaya Sanjana
dc.contributor.author	Shahriar, Sajid
dc.date.accessioned	2025-03-06T06:32:53Z
dc.date.available	2025-03-06T06:32:53Z
dc.date.issued	2024-09-04
dc.identifier.citation	[1] “Deafness and hearing loss.” https://www.who.int/news-room/fact sheets/detail/deafness-and-hearing-loss. [2] https://wfdeaf.org/our-work/. [3] R. C. Poonia, “List: A lightweight framework for continuous indian sign language translation,” Information, vol. 14, no. 2, p. 79, 2023. [4] W. C. Stokoe Jr, “Sign language structure: An outline of the visual com munication systems of the american deaf,” Journal of deaf studies and deaf education, vol. 10, no. 1, pp. 3–37, 2005. [5] W. C. Stokoe, D. C. Casterline, and C. G. Croneberg, A Dictionary of Amer ican Sign Language on Linguistic Principles. Linstok Press, 1976. [6] S. S. Kumar, T. Wangyal, V. Saboo, and R. Srinath, “Time series neural net works for real-time sign language translation,” in 2018 17th IEEE Interna tional Conference on Machine Learning and Applications (ICMLA), pp. 243– 248, IEEE, 2018. [7] A. Hasib, S. S. Khan, J. F. Eva, M. Khatun, A. Haque, N. Shahrin, R. Rah man, H. Murad, M. Islam, M. R. Hussein, et al., “Bdsl 49: A comprehensive dataset of bangla sign language,” arXiv preprint arXiv:2208.06827, 2022. [8] K. K. Podder, M. E. H. Chowdhury, A. M. Tahir, Z. B. Mahbub, A. Khan dakar, M. S. Hossain, and M. A. Kadir, “Bangla sign language (bdsl) al phabets and numerals classification using a deep learning model,” Sensors, vol. 22, no. 2, 2022. [9] M. S. Islam, S. Sultana Sharmin, N. Jessan, A. S. A. Rabby, and S. Hossain, “Ishara-lipi: The first complete multipurposeopen access dataset of isolated characters for bangla sign language,” pp. 1–4, 09 2018. 38 [10] S. M. Rayeed, G. Akram, S. Tuba, G. Zilani, H. Mahmud, and M. K. Hasan, “Bangla sign digits recognition using depth information,” p. 36, 03 2022. [11] S. N. Hasan, M. J. Hasan, and K. S. Alam, “Shongket: A comprehensive and multipurpose dataset for bangla sign language detection,” in 2021 Interna tional Conference on Electronics, Communications and Information Technol ogy (ICECIT), pp. 1–4, 2021. [12] H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Li, “Improving sign language trans lation with monolingual data by sign back-translation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1316–1325, 2021. [13] A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i Nieto, “How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2021. [14] N. Adaloglou, T. Chatzis, I. Papastratis, A. Stergioulas, G. T. Papadopoulos, V. Zacharopoulou, G. J. Xydopoulos, K. Atzakas, D. Papazachariou, and P. Daras, “A comprehensive study on sign language recognition methods,” CoRR, vol. abs/2007.12530, 2020. [15] A. Imashev, M. Mukushev, V. Kimmelman, and A. Sandygulova, “A dataset for linguistic understanding, visual evaluation, and recognition of sign lan guages: The k-rsl,” pp. 631–640, 01 2020. [16] O. M. Sincan and H. Y. Keles, “AUTSL: A large scale multi-modal turkish sign language dataset and baseline methods,” CoRR, vol. abs/2008.00932, 2020. [17] O. Ozdemir, A. A. Kındıro˘glu, N. Cihan Camgoz, and L. Akarun, “Bospho- ¨ rusSign22k Sign Language Recognition Dataset,” in Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Lan 39 guages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, 2020. [18] N. C. Camg¨oz, A. A. Kındıro˘glu, S. Karab¨ukl¨u, M. Kelepir, A. S. Ozsoy, ¨ and L. Akarun, “BosphorusSign: a Turkish sign language recognition corpus in health and finance domains,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 1383– 1388, 2016. [19] T. Hanke, M. Schulder, R. Konrad, and E. Jahn, “Extending the Public DGS Corpus in size and depth,” in Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Re sources in the Service of the Language Community, Technological Challenges and Application Perspectives, (Marseille, France), pp. 75–82, European Lan guage Resources Association (ELRA), May 2020. [20] D. Li, C. Rodriguez, X. Yu, and H. Li, “Word-level deep sign language recog nition from video: A new large-scale dataset and methods comparison,” in The IEEE Winter Conference on Applications of Computer Vision, pp. 1459– 1469, 2020. [21] “Asl university.” http://asluniversity.com/. [22] N. K. Caselli, Z. S. Sehyr, A. M. Cohen-Goldberg, and K. Emmorey, “Asl-lex: A lexical database of american sign language,” Behavior research methods, vol. 49, pp. 784–801, 2017. [23] N. C. Camg¨oz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, “Rwth phoenix-weather 2014 t: Parallel corpus of sign language video, gloss and translation,” CVPR, Salt Lake City, UT, vol. 3, p. 6, 2018. [24] T. Hanke, “iLex - a tool for sign language lexicography and corpus analysis,” in Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), (Las Palmas, Canary Islands - Spain), European Language Resources Association (ELRA), May 2002. 40 [25] O. Crasborn and H. Sloetjes, “Enhanced elan functionality for sign language coropora,” 01 2008. [26] C. Neidle, S. Sclaroff, and V. Athitsos, “Signstream: A tool for linguistic and computer vision research on visual-gestural language data,” Behavior research methods, instruments, computers : a journal of the Psychonomic Society, Inc, vol. 33, pp. 311–20, 09 2001. [27] M. Kipp, “Anvil - a generic annotation tool for multimodal dialogue,” pp. 1367–1370, 09 2001. [28] Y. Chen, R. Zuo, F. Wei, Y. Wu, S. Liu, and B. Mak, “Two-stream network for sign language recognition and translation,” 2023. [29] H. R. V. Joze and O. Koller, “Ms-asl: A large-scale data set and benchmark for understanding american sign language,” 2019. [30] D. Li, X. Yu, C. Xu, L. Petersson, and H. Li, “Transferring cross-domain knowledge for video sign language recognition,” 2020. [31] R. Zuo, F. Wei, and B. Mak, “Natural language-assisted sign language recog nition,” 2023. [32] S. Albanie, G. Varol, L. Momeni, T. Afouras, J. S. Chung, N. Fox, and A. Zisserman, “Bsl-1k: Scaling up co-articulated sign language recognition using mouthing cues,” 2021. [33] A. Duarte, S. Albanie, X. G. i Nieto, and G. Varol, “Sign language video retrieval with free-form textual queries,” 2022. [34] L. Momeni, G. Varol, S. Albanie, T. Afouras, and A. Zisserman, “Watch, read and lookup: learning to spot signs from multiple supervisors,” 2020. [35] G. Varol, L. Momeni, S. Albanie, T. Afouras, and A. Zisserman, “Read and attend: Temporal localisation in sign language videos,” 2021. 41 [36] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, “Neural sign language translation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7784–7793, 2018. [37] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Multi-channel trans formers for multi-articulatory sign language translation,” 2020. [38] Y. Chen, F. Wei, X. Sun, Z. Wu, and S. Lin, “A simple multi-modality transfer learning baseline for sign language translation,” 2023. [39] D. Li, C. Xu, X. Yu, K. Zhang, B. Swift, H. Suominen, and H. Li, “Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation,” 2020. [40] Y. Cheng, F. Wei, J. Bao, D. Chen, and W. Zhang, “Cico: Domain-aware sign language retrieval via cross-lingual contrastive learning,” 2023. [41] A. Imashev, M. Mukushev, V. Kimmelman, and A. Sandygulova, “A dataset for linguistic understanding, visual evaluation, and recognition of sign lan guages: The k-RSL,” in Proceedings of the 24th Conference on Computational Natural Language Learning, (Online), pp. 631–640, Association for Compu tational Linguistics, Nov. 2020. [42] R. Cui, H. Liu, and C. Zhang, “Recurrent convolutional neural networks for continuous sign language recognition by staged optimization,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1610– 1618, 2017. [43] O. Koller, N. C. Camgoz, H. Ney, and R. Bowden, “Weakly supervised learn ing with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 9, pp. 2306–2320, 2020. [44] O. Koller, S. Zargaran, and H. Ney, “Re-sign: Re-aligned end-to-end se quence modelling with deep recurrent cnn-hmms,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3416–3424, 2017. 42 [45] J. Pu, W. Zhou, and H. Li, “Iterative alignment network for continuous sign language recognition,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4160–4169, 2019. [46] H. Zhou, W. Zhou, Y. Zhou, and H. Li, “Spatial-temporal multi-cue network for continuous sign language recognition,” 2020. [47] H. Hu, W. Zhao, W. Zhou, Y. Wang, and H. Li, “Signbert: Pre-training of hand-model-aware representation for sign language recognition,” 2021. [48] H. Hu, W. Zhou, and H. Li, “Hand-model-aware sign language recogni tion,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1558–1566, May 2021. [49] H. Hu, W. Zhou, J. Pu, and H. Li, “Global-local enhancement network for NMF-aware sign language recognition,” ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 17, pp. 1–19, jul 2021. [50] S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Sign language recog nition via skeleton-aware multi-model ensemble,” 2021. [51] S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Skeleton aware multi modal sign language recognition,” in Proceedings of the IEEE/CVF Confer ence on Computer Vision and Pattern Recognition, pp. 3413–3423, 2021. [52] Y. Min, A. Hao, X. Chai, and X. Chen, “Visual alignment constraint for continuous sign language recognition,” 2021. [53] R. Zuo and B. Mak, “Local context-aware self-attention for continuous sign language recognition,” Proc. Interspeech 2022, pp. 4810–4814, 2022. [54] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” 2018. [55] Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3d residual networks,” 2017. 43 [56] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Sign language trans formers: Joint end-to-end sign language recognition and translation,” in Pro ceedings of the IEEE/CVF conference on computer vision and pattern recog nition, pp. 10023–10033, 2020. [57] K. Yin and J. Read, “Better sign language translation with stmc transformer,” 2020. [58] H. Zhou, W. Zhou, Y. Zhou, and H. Li, “Spatial-temporal multi-cue net work for continuous sign language recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13009–13016, 2020. [59] H. Fan, Y. Li, B. Xiong, W.-Y. Lo, and C. Feichtenhofer, “Pyslowfast.” https://github.com/facebookresearch/slowfast, 2020. [60] C. Feichtenhofer, “X3d: Expanding architectures for efficient video recogni tion,” 2020. [61] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,” 2018. [62] R. Goyal, S. E. Kahou, V. Michalski, J. Materzy´nska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic, “The ”something something” video database for learning and evaluating visual common sense,” 2017. [63] F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–970, 2015. [64] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, “The kinetics human action video dataset,” 2017. 44 [65] A. N´u˜nez-Marcos, O. Perez-de Vi˜naspre, and G. Labaka, “A survey on sign language machine translation,” Expert Systems with Applications, p. 118993, 2022. [66] K.-Y. Su, M.-W. Wu, and J.-S. Chang, “A new quantitative quality measure for machine translation systems.,” pp. 433–439, 08 1992. [67] De Coster, Mathieu and D’Oosterlinck, Karel and Pizurica, Marija and Rabaey, Paloma and Verlinden, Severine and Van Herreweghe, Mieke and Dambre, Joni, “Frozen pretrained transformers for neural sign language translation,” in Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pp. 88–97, Associ ation for Machine Translation in the Americas, 2021. [68] A. Almohimeed, M. Wald, and R. Damper, “A new evaluation approach for sign language machine translation,” 2009. [69] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” 10 2002. [70] C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, and H. Sawaf, “Accelerated dp based search for statistical translation.,” 01 1997. [71] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, (Cambridge, Massachusetts, USA), pp. 223–231, Associa tion for Machine Translation in the Americas, Aug. 8-12 2006. [72] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, (Barcelona, Spain), pp. 74–81, Associa tion for Computational Linguistics, July 2004. [73] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evalu ation with improved correlation with human judgments,” in Proceedings of 45 the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Ma chine Translation and/or Summarization, (Ann Arbor, Michigan), pp. 65–72, Association for Computational Linguistics, June 2005. [74] G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics,” pp. 138–145, 01 2002. [75] M. Post, “A call for clarity in reporting bleu scores,” 2018. [76] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero reference deep curve estimation for low-light image enhancement,” 2020. [77] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4960–4964, IEEE, 2016. [78] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014. [79] S. Zhang, Y. Feng, and L. Li, “Future-guided incremental transformer for si multaneous translation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14428–14436, 2021. [80] Y. Liu, H. Xiong, Z. He, J. Zhang, H. Wu, H. Wang, and C. Zong, “End-to-end speech translation with knowledge distillation,” arXiv preprint arXiv:1904.08075, 2019. [81] I. E. Murtagh, A linguistically motivated computational framework for Irish sign language. PhD thesis, Trinity College Dublin. School of Linguistic Speech & Comm Sci. CLCS, 2019	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/2361
dc.description	Supervised by Dr. Hasan Mahmud, Associate Professor, Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT) Board Bazar, Gazipur, Bangladesh This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Computer Science and Engineering, 2024	en_US
dc.description.abstract	Sign language serves as a crucial communication tool for the Deaf community, representing a fully developed linguistic system. As research into sign language via deep learning expands, the need for an efficient and robust toolset for sign lan- guage data collection becomes increasingly important. This abstract introduces the Sign Language Data Collection Toolset (SLDCT), which is designed to stream- line the processes of collecting and annotating sign language data. The SLDCT includes various interconnected components such as user registration, video record- ing, image extraction, keypoint extraction, and the selection of multiple regions of interest. The video recording software, a central feature of the toolset, facilitates the easy recording of sign language performances. A significant challenge in sign language data collection is the annotation process, which is often time-consuming and requires detailed analysis by experts. To mitigate this issue, the SLDCT in- tegrates annotation tools that allow for real-time annotation during the recording process, thereby merging the recording and annotation phases. This integration helps researchers annotate data more efficiently. One notable feature of the toolset is the ability to collect keypoints. The toolset can extract frames from videos, cap- ture facial landmarks, hand poses, and other keypoints, and save this information in a .csv file. These keypoints are valuable for training lightweight deep-learning models for various research applications. Additionally, the toolset allows for the capture of multiple regions of interest from video frames, with the bounding box coordinates saved as metadata alongside the dataset.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.subject	Bangla Sign Language, Toolkit, Dataset Generation, MediaPipe, Videos	en_US
dc.title	Bangla Sign Language Generation From Videos	en_US
dc.type	Thesis	en_US