By Xiao-Ping Zhang and Zhu Liu
Ryerson University, Canada
AT&T-Research, USA
zliu@research.att.com
xzhang@ee.ryerson.ca
Abstract
With the rapid growing popularity of multimedia content creation and sharing enabled by the high quality mobile video capture devices and the broadband wire or wireless connection, the volume of new content that is available to us is simply beyond our consumption capacity. For example, in the last year, video material uploaded to the YouTube video sharing service has increased from about 20 hours per minute to about 35 hours per minute. Tools that can automatically find the most relevant content according to our interests, specified manually or learned from our viewing history, are desired. Multimedia content analysis and understanding is an indispensable component in such tools and many other multimedia applications and services. It has been an active research area in the last two decades, and it has evolved to a combination and interconnection of many subjects such as audio and music processing, image and video processing, natural language processing, computer vision, and machine learning.
Graphical models are the combination of probability theory and graph theory, and they provide a way to represent the joint distribution over all of the random variables by a product of factors each depending only on a subset of the variables. Such models are flexible and scalable to capture complex dependencies among large number of random variables. Many existing statistical models and methods find a unifying representation in graphical models. For example, Hidden Markov models, random Markov field, Kalman filter, etc. As an emerging framework in machine learning with enormous potential, graphical models have been introduced in the multimedia content analysis area, and are adopted widely in many applications. The intrinsic nature of multimedia content analysis tasks, including high dimension of the low level features, rich prior knowledge on the structure of multimedia content, as well as the complex temporal-spatial correlation among multiple modalities, find a perfect match for graphical models. There is no doubt that the application of graphical models in multimedia content analysis will keep thriving.
The purpose of this tutorial is twofold: introducing the graphical models as a new framework of machine learning, and demonstrating the applications of graphical models in multimedia content processing domain. This tutorial is intended for researchers in the multimedia content analysis and understanding area as well as professionals working in related fields. Fundamentals in both graphical models and the multimedia content analysis will be covered in this tutorial, and there are no prerequisites for the audience. The tutorial is designed to present a refreshing, broad perspective on graphic models, and in-depth examples on their application in multimedia content analysis. It will be valuable to experts in the constituent technologies such as multimedia indexing and search, content-based analysis who are looking to broaden their knowledge beyond their current areas of expertise. Specifically, the audience will: 1) Understand the basic of the graph theory and graphical models; 2) Learn special graphical models, including Hidden Markov models, Markov Random Field, and Conditional Random Field; 3) Get familiar with the general approaches in multimedia content analysis systems; and 4) Study a few examples on how to apply graphical models in multimedia content analysis tasks, for example, video event detection, video sequence matching, image labeling, etc; 5) Be able to determine which topics in multimedia content analysis are of interest to them for further study.
Outline
This tutorial will be half day, and it includes two parts: Part I – Introduction of Basic Theory, and Part II – Advances and Applications in Multimedia Content Analysis. Following are detailed outlines for each part.
Part I: Introduction of Basic Theory
- Bayesian Methods
- Graphical Models
- Hidden Markov Models (HMM)
- Markov Random Field (MRF)
- Conditional Random Field (CRF)
Part II: Advances and Applications in Multimedia Content Analysis
- Video Temporal Dynamics
- Identification of Digital Video based on Shot-level Sequence Matching
- Independent Component Analysis Mixture Hidden Markov Models (ICAMHMM)
- Video Event Detection Using ICAMHMM
- ICA Mixture Hidden Conditional Random Field Model (ICAMHCRF) for Sports Event Classification
- Gaussian Mixture Conditional Random Field Modeling for Indoor Image Labeling
- Laplacian Mixture Conditional Random Field Model for Image Labeling
- Future Research Directions
Audience
Introductory and intermediate, specifically, graduate students and researchers interested in the graphical models and their applications in multimedia content analysis.
Organizers/Presenters
Xiao-Ping Zhang received the B.S. and Ph.D. degrees from Tsinghua University, in 1992 and 1996, respectively, all in electronic engineering. He received the M.B.A. degree in finance and economics with Honors from the University of Chicago Booth School of Business, Chicago, IL. Since fall 2000, he has been with the Department of Electrical and Computer Engineering, Ryerson University, where he is now Professor and Director of Communication and Signal Processing Applications Laboratory (CASPAL). Prior to joining Ryerson, from 1996 to 1998, he was a Postdoctoral Fellow at the University of Texas, San Antonio, and then at the Beckman Institute, the University of Illinois at Urbana-Champaign. He held research and teaching positions with the Communication Research Laboratory, McMaster University, Canada, in 1999. From 1999 to 2000, he was a Senior DSP Engineer with SAM Technology, Inc., San Francisco, CA, and a consultant with the San Francisco Brain Research Institute. His research interests include signal processing for communications, multimedia retrieval and video content analysis, computational intelligence, and applications in bioinformatics, finance, and marketing. He is a frequent consultant for biotech companies and investment firms. Dr. Zhang is a senior member of IEEE and a registered Professional Engineer in Ontario, Canada and a member of Beta Gamma Sigma Honor Society. He is the Publicity Co-Chair for ICME’06 and program Co-Chair for ICIC’05. He has served as guest editor for the Multimedia Tools and Applications Journal. He is currently an Associate Editor for IEEE Signal Processing Letters.
Zhu Liu received the B.S. and M.S. degrees in Electronic Engineering from Tsinghua University, Beijing, China, in 1994 and 1996, respectively, and the Ph.D. degree in Electrical Engineering from Polytechnic University, Brooklyn, NY (now part of New York University), in 2001. He joined AT&T Labs - Research, Middletown, NJ, in 2000, and is currently a Principle Member of Technical Staff in the Video and Multimedia Technologies and Services Research Department. He is an adjunct professor of the Electrical Engineering Department of Columbia University. His research interests include multimedia content analysis, multimedia databases, video search, pattern recognition, machine learning, and natural language understanding. He holds 13 U.S. patents and has published more than 60 papers in international conferences and journals. Dr. Liu is a senior member of IEEE, and a member of ACM. Dr. Liu and his colleagues won the best demonstration award in the Consumer Communication & Networking Conference 2007. He is on the editorial board of the IEEE Transaction on Multimedia and the Peer-to-peer Networking and Applications Journal. He has served as guest editor for the IEEE Transaction on Multimedia, the Multimedia Tools and Applications Journal, and the International Journal of Semantic Computing. He was also on the organizing committee and technical committee for many IEEE International Conferences.