Optical Flow Based Facial Expression Recognition from Video Sequences

Salekin, Md Sirajus

dc.contributor.author	Salekin, Md Sirajus
dc.date.accessioned	2020-09-18T08:54:29Z
dc.date.available	2020-09-18T08:54:29Z
dc.date.issued	2015-11-15
dc.identifier.citation	[1] Andrew J Calder, A Mike Burton, Paul Miller, Andrew W Young, and Shigeru Akamatsu. A principal component analysis of facial expressions. Vision research, 41(9):1179{1208, 2001. [Cited on pages 3 and 10] [2] Paul Ekman and Wallace V Friesen. Facial action coding system. 1977. [Cited on page 8] [3] Joseph C Hager, Paul Ekman, and Wallace V Friesen. Facial action coding system. Salt Lake City, UT: A Human Face, 2002. [Cited on page 8] [4] Zhengyou Zhang. Feature-based facial expression recognition: Sensitivity analysis and experiments with a multilayer perceptron. International journal of pattern recognition and Arti cial Intelligence, 13(06):893{911, 1999. [Cited on page 8] [5] Guodong Guo and Charles R Dyer. Simultaneous feature selection and classi er training via linear programming: A case study for face expression recognition. In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, volume 1, pages I{346. IEEE, 2003. [Cited on page 8] [6] Michel F Valstar, I Patras, and Maja Pantic. Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)-Workshops, pages 76{76. IEEE, 2005. [Cited on page 8] [7] Michel Valstar and Maja Pantic. Fully automatic facial action unit detection and temporal analysis. In 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06), pages 149{149. IEEE, 2006. [Cited on page 8] [8] Maja Pantic and Leon J. M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on pattern analysis and machine intelli- gence, 22(12):1424{1445, 2000. [Cited on page 8] [9] Irene Kotsia and Ioannis Pitas. Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE transac- tions on image processing, 16(1):172{187, 2007. [Cited on page 9] 46 Bibliography 47 [10] Curtis Padgett and Garrison W Cottrell. Representing face images for emotion classi cation. Advances in neural information processing systems, pages 894{900, 1997. [Cited on page 10] [11] Gianluca Donato, Marian Stewart Bartlett, Joseph C. Hager, Paul Ekman, and Terrence J. Sejnowski. Classifying facial actions. IEEE transactions on pattern analysis and machine intelligence, 21(10):974{989, 1999. [Cited on page 10] [12] S everine Dubuisson, Franck Davoine, and Myl ene Masson. A solution for facial expression representation and recognition. Signal Processing: Image Communication, 17(9):657{673, 2002. [Cited on page 10] [13] Ioan Buciu, I Pitas, et al. Ica and gabor representation for facial expression recognition. In Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, volume 2, pages II{855. IEEE, 2003. [Cited on page 10] [14] CHEN Fan and Kazunori Kotani. Facial expression recognition by supervised independent component analysis using map estimation. IEICE TRANSACTIONS on Information and Systems, 91(2):341{350, 2008. [Cited on page 10] [15] Michael J Lyons, Julien Budynek, and Shigeru Akamatsu. Automatic classi cation of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12):1357{1362, 1999. [Cited on page 10] [16] Md Zia Uddin, JJ Lee, and T-S Kim. An enhanced independent component-based human facial expression recognition from video. IEEE Transactions on Consumer Electronics, 55(4), 2009. [Cited on pages 10 and 11] [17] Timo Ojala, Matti Pietikainen, and Topi Maenpaa. Multiresolution gray-scale and rotation invariant texture classi cation with local binary patterns. IEEE Transac- tions on pattern analysis and machine intelligence, 24(7):971{987, 2002. [Cited on pages 11, 12, 13, and 30] [18] Taskeed Jabid, Md Hasanul Kabir, and Oksam Chae. Local directional pattern (ldp) for face recognition. In Consumer Electronics (ICCE), 2010 Digest of Technical Papers International Conference on, pages 329{330. IEEE, 2010. [Cited on pages 11 and 14] [19] Ngoc-Son Vu and Alice Caplier. Face recognition with patterns of oriented edge magnitudes. Computer Vision{ECCV 2010, pages 313{326, 2010. [Cited on pages 11 and 16] [20] Caifeng Shan, Shaogang Gong, and Peter W McOwan. Robust facial expression recognition using local binary patterns. In Image Processing, 2005. ICIP 2005. IEEE International Conference on, volume 2, pages II{370. IEEE, 2005. [Cited on page 12] Bibliography 48 [21] Guoying Zhao and Matti Pietikainen. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE transactions on pattern analysis and machine intelligence, 29(6), 2007. [Cited on page 12] [22] Caifeng Shan, Shaogang Gong, and Peter W McOwan. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 27(6):803{816, 2009. [Cited on page 12] [23] Timo Ahonen, Abdenour Hadid, and Matti Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence, 28(12):2037{2041, 2006. [Cited on page 12] [24] Taskeed Jabid, Md Hasanul Kabir, and Oksam Chae. Robust facial expression recognition based on local directional pattern. ETRI journal, 32(5):784{794, 2010. [Cited on pages 14 and 18] [25] Taskeed Jabid, Md Hasanul Kabir, and Oksam Chae. Local directional pattern (ldp){a robust image descriptor for object recognition. In Advanced Video and Signal Based Surveillance (AVSS), 2010 Seventh IEEE International Conference on, pages 482{487. IEEE, 2010. [Cited on page 14] [26] Md Zia Uddin. An e cient local feature-based facial expression recognition system. Arabian Journal for Science and Engineering, 39(11):7885{7893, 2014. [Cited on pages 14, 18, 35, 36, 39, 41, and 42] [27] Ngoc-Son Vu, Hannah M Dee, and Alice Caplier. Face recognition using the poem descriptor. Pattern Recognition, 45(7):2478{2488, 2012. [Cited on page 16] [28] Ngoc-Son Vu, Huu-Tuan Nguyen, and Alice Caplier. Multiple patterns of gradient magnitudes for face recognition. In Image Processing (ICIP), 2012 19th IEEE International Conference on, pages 589{592. IEEE, 2012. [Cited on page 16] [29] Ngoc-Son Vu and Alice Caplier. Enhanced patterns of oriented edge magnitudes for face recognition and image matching. IEEE Transactions on Image Processing, 21(3):1352{1365, 2012. [Cited on page 16] [30] Ngoc-Son Vu. Exploring patterns of gradient orientations and magnitudes for face recognition. IEEE Transactions on Information Forensics and Security, 8(2):295{ 304, 2013. [Cited on page 16] [31] E. Silva, C. Esparza, and Y. Meja. Poem-based facial expression recognition, a new approach. In 2012 XVII Symposium of Image, Signal Processing, and Arti cial Vision (STSIVA), pages 162{167, Sept 2012. [Cited on pages 16 and 39] [32] Bruce D Lucas, Takeo Kanade, et al. An iterative image registration technique with an application to stereo vision. 1981. [Cited on pages 17, 25, 26, 27, and 30] Bibliography 49 [33] Berthold KP Horn and Brian G Schunck. Determining optical ow. Arti cial intelligence, 17(1-3):185{203, 1981. [Cited on pages 17, 25, 26, and 27] [34] Padmanabhan Anandan. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3): 283{310, 1989. [Cited on page 17] [35] James R. Bergen, P. Anandan, Th J. Hanna, and Rajesh Hingorani. Hierarchical model-based motion estimation. pages 237{252. Springer-Verlag, 1992. [Cited on page 17] [36] MASE Kenji. Recognition of facial expression from optical ow. IEICE TRANS- ACTIONS on Information and Systems, 74(10):3474{3483, 1991. [Cited on pages 18 and 24] [37] Andreas Lanitis, Christopher J Taylor, and Timothy F Cootes. A uni ed approach to coding and interpreting face images. In Computer Vision, 1995. Proceedings., Fifth International Conference on, pages 368{373. IEEE, 1995. [Cited on page 18] [38] Michael J Black and Yaser Yacoob. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Computer Vision, 1995. Proceedings., Fifth International Conference on, pages 374{381. IEEE, 1995. [Cited on pages 18 and 24] [39] Yaser Yacoob and Larry S Davis. Recognizing human facial expressions from long image sequences using optical ow. IEEE Transactions on pattern analysis and machine intelligence, 18(6):636{642, 1996. [Cited on pages 18 and 24] [40] Irfan A. Essa and Alex Paul Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE transactions on pattern analysis and machine intelligence, 19(7):757{763, 1997. [Cited on pages 18 and 24] [41] Md Zia Uddin, Tae-Seong Kim, and Byung Cheol Song. An optical ow featurebased robust facial expression recognition with hmm from video. International Journal of Innovative Computing, Information and Control, 9(4):1409{1421, 2013. [Cited on pages 18, 38, and 39] [42] Yoseph Linde, Andres Buzo, and Robert Gray. An algorithm for vector quantizer design. IEEE Transactions on communications, 28(1):84{95, 1980. [Cited on pages 18, 19, and 20] [43] Y-I Tian, Takeo Kanade, and Je rey F Cohn. Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence, 23(2):97{115, 2001. [Cited on page 18] Bibliography 50 [44] Fabrice Bourel, Claude C Chibelushi, and Adrian A Low. Robust facial expression recognition using a state-based model of spatially-localised facial dynamics. In Au- tomatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, pages 113{118. IEEE, 2002. [Cited on page 18] [45] James MacQueen et al. Some methods for classi cation and analysis of multivariate observations. In Proceedings of the fth Berkeley symposium on mathematical statis- tics and probability, volume 1, pages 281{297. Oakland, CA, USA., 1967. [Cited on pages 19, 32, and 41] [46] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273{297, 1995. [Cited on pages 20, 21, and 42] [47] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550{1560, 1990. [Cited on pages 20 and 21] [48] Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of nite state markov chains. The annals of mathematical statistics, 37(6):1554{1563, 1966. [Cited on pages 20 and 22] [49] Leonard E Baum, Ted Petrie, George Soules, and Norman Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. The annals of mathematical statistics, 41(1):164{171, 1970. [Cited on pages 20 and 22] [50] Maodi Hu, Yunhong Wang, Zhaoxiang Zhang, De Zhang, and James J Little. Incremental learning for video-based gait recognition with lbp ow. IEEE transactions on cybernetics, 43(1):77{89, 2013. [Cited on page 22] [51] Kelson RT Aires, Andre M Santana, and Adelardo AD Medeiros. Optical ow using color information: preliminary results. In Proceedings of the 2008 ACM symposium on Applied computing, pages 1607{1611. ACM, 2008. [Cited on page 26] [52] Takeo Kanade, Je rey F Cohn, and Yingli Tian. Comprehensive database for facial expression analysis. In Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, pages 46{53. IEEE, 2000. [Cited on pages 35 and 38] [53] Ira Cohen, Nicu Sebe, Ashutosh Garg, Lawrence S Chen, and Thomas S Huang. Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and image understanding, 91(1):160{187, 2003. [Cited on page 41] [54] Y Zhu, Liyanage C De Silva, and Chi Chung Ko. Using moment invariants and hmm in facial expression recognition. Pattern Recognition Letters, 23(1):83{91, 2002. [Cited on page 41] Bibliography 51 [55] Thomas Cover and Peter Hart. Nearest neighbor pattern classi cation. IEEE transactions on information theory, 13(1):21{27, 1967. [Cited on page 42] [56] Andrew McCallum, Kamal Nigam, et al. A comparison of event models for naive bayes text classi cation. In AAAI-98 workshop on learning for text categorization, volume 752, pages 41{48. Citeseer, 1998. [Cited on page 42]	en_US
dc.identifier.uri	http://hdl.handle.net/123456789/340
dc.description	Supervised by Prof. Dr. Md. Hasanul Kabir	en_US
dc.description.abstract	Facial expression is one of the most powerful masses of non-verbal communication through which we can easily enter the world of one's instant emotions or intuitions. As most of the time, it elicits naturally, so it brings out a lot of applications in the eld of machine intelligence, behavioral science, clinical practices, biometric security, gaming, human computer interactions, psychological research, data-driven animation etc. Proper expression recognition can lead to an intelligent machine for taking commands more e ectively, can show the insight psychological condition of a patient to a psychiatrist or researcher, can show the next suggested path for any computer game. But automatic facial expression recognition is a challenging task due to the di erent factors such as variations in illumination, pose, facial expression, alignment, di erent ages, occlusions etc. In this thesis paper, we propose a novel feature representation by a new feature descriptor, named Patterns of Oriented Motion Flow (POMF) from the optical ow information, to recognize the proper facial expression from a facial video. The POMF computes di erent directional motion information and encodes those directional ow information with enhanced local texture micro pattern. As it captures the spatial temporal changes of facial movements through optical ow and enables to observe both local and global structures, it shows its robustness for the facial expression. Finally, the POMF histogram is used to train the expression model by Hidden Markov Model (HMM). To train through the HMM, the objective sequences are produced by the generation of codebook using K-means clustering technique. The performance of the proposed method has been evaluated over the RGB camera based and Depth camera based video. We also compare the proposed method with the other promising appearance based methods. Experimental results demonstrate that the proposed POMF descriptor is more robust in extracting facial information and provides higher classi cation rate compared to other existing promising methods.	en_US
dc.language.iso	en	en_US
dc.publisher	Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT), Board Bazar, Gazipur-1704, Bangladesh	en_US
dc.title	Optical Flow Based Facial Expression Recognition from Video Sequences	en_US
dc.type	Thesis	en_US