Tutorial 6 : Crowdsourcing Applications for Multimedia and Signal Processing

Presenters:

Dinei Florêncio, Microsoft Research, USA
Flávio Ribeiro, Universidade de São Paulo, Brazil

Abstract

Crowdsourcing is a game-changing technology which will play a significantly stronger role in multimedia and DSP over the next several years. It allows one to bring human intelligence and subjectivity into the available tools, with deep impact in many areas. Furthermore, recent progress in infrastructure providers (such as Amazon Mechanical Turk) allow developers to introduce crowdsourcing almost as transparently as other processing tasks, albeit with a larger delay. In this tutorial, we take a look at the crowdsourcing landscape, with a focus in tasks related to, or of interest to, multimedia and signal processing research. In particular, we will look at crowdsourcing as a processing block, which can be combined with a number of traditional computational methods to produce desired results. Understanding early on the characteristics, limitations, and underlying concepts of crowdsourcing will bring a strong advantage to researchers in the area, as they deploy applications using this novel tool.

Outline

Part 1 - Introduction: What is crowdsourcing?
Part 2 - Variations:

• Games with a Purpose.
• Crowd workers: Amazon’s Mechanical Turk, CrowdFlower, and others.
• Volunteers and interest groups.

Part 3 - Multimedia Applications:

• Subjective quality assessment.
• Image tagging, classification, segmentation.
• Relevance detection and foveation.

Part 4 - Games with a Purpose: advantages and limitations.
Part 5 - Mechanical Turk: advantages and limitations.
Part 6 - Case study 1: A simple experiment on user preferences.
Part 7 - Case study 2: Automating mean opinion score testing using the crowd.

• Scripts
• Interface
• Test designs
• Experiments/results
• Lessons

Part 8 - Case study 3: region of interest tracking for videos.
Part 9 - What to expect from crowdsourcing in the next several years.
Part 10 - Conclusions.

Biography of the presenters

Dinei Florêncio received the B.S. and M.S. from University of Brasília (Brazil), and the Ph.D. from Georgia Tech, all in Electrical Engineering. He has been a researcher with Microsoft Research since 1999, currently with the Communication and Collaboration Systems group. From 1996 to 1999, he was a member of the research staff at the David Sarnoff Research Center. He was also a co-op student with AT&T Human Interface Lab (now part of NCR) from 1994 to 1996, and a summer intern at the (now defunct) Interval Research in 1994. Dr. Florêncio’s current research focus includes signal processing and computer security. In the area of signal processing, he works in audio and video processing, with particular focus to real time communication. He has numerous contributions in speech enhancement, microphone arrays, image and video coding, spectral analysis, and non-linear algorithms. In the area of computer security, his interest focuses in cybercrime and problems that can be assisted by algorithmic research. Topics include phishing prevention, user authentication, sender authentication, human interactive proofs, and economics of cybercrime. A more recent interest is in harnessing the power of the web to help DSP, either by exploiting large databases, or by incorporating humans in the loop by using crowdsourcing. Dr. Florêncio is a senior member of the IEEE, and has published over 50 refereed papers, and 36 granted US patents (with another 13 currently pending). He received the 1998 Sarnoff Achievement Award and an NCR inventor award. His papers have won several awards, including at SOUPS’2010, ICME’2010, and a SAIC best paper award in 1996. His research has enhanced the lives of millions of people, through high impact technology transfers to many Microsoft products, including Live Messenger, Exchange Server, RoundTable, and the MSN toolbar. Dr. Florêncio was general co-chair of CBSP’08, MMSP’09, and WIFS’11 and technical co-chair of Hot3D’2010, WIFS’2010, and ICME’2011. He is also a track chair for ICIP’11.

Flávio Ribeiro received the B.S. degree in electrical engineering from Escola Politécnica, University of São Paulo, Brazil, in 2005. He is currently pursuing the Ph.D. degree in electrical engineering, also at Escola Politécnica, University of São Paulo. On the summers of 2009 and 2010, he was a research intern at Microsoft Research Redmond. Mr. Ribeiro won the ICME 2010 best student paper award. His research interests include array signal processing, multimedia signal processing, active noise control and applied linear algebra. His current passion is how to harness the power of the web and crowdsourcing to help signal processing algorithms and research.