Skip to main navigation Skip to search Skip to main content

Deep multimodal speaker naming

  • Yongtao Hu
  • , Jimmy Sj Ren
  • , Jingwen Dai
  • , Chang Yuan
  • , Li Xu
  • , Wenping Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

47 Citations (Scopus)

Abstract

Automatic speaker naming is the problem of localizing as well as identifying each speaking character in a TV/movie/live show video. This is a challenging problem mainly attributes to its multimodal nature, namely face cue alone is insufficient to achieve good performance. Previous multimodal approaches to this problem usually process the data of different modalities individually and merge them using handcrafted heuristics. Such approaches work well for simple scenes, but fail to achieve high performance for speakers with large appearance variations. In this paper, we propose a novel con-volutional neural networks (CNN) based learning framework to automatically learn the fusion function of both face and audio cues. We show that without using face tracking, fa-cial landmark localization or subtitle/transcript, our system with robust multimodal feature extraction is able to achieve state-of-The-Art speaker naming performance evaluated on two diverse TV series. The dataset and implementation of our algorithm are publicly available online.

Original languageEnglish
Title of host publicationMM 2015 - Proceedings of the 2015 ACM Multimedia Conference
Pages1107-1110
Number of pages4
ISBN (Electronic)9781450334594
DOIs
Publication statusPublished - 13 Oct 2015
Externally publishedYes
Event23rd ACM International Conference on Multimedia, MM 2015 - Brisbane, Australia
Duration: 26 Oct 2015 → 30 Oct 2015

Publication series

NameMM 2015 - Proceedings of the 2015 ACM Multimedia Conference

Conference

Conference23rd ACM International Conference on Multimedia, MM 2015
Country/TerritoryAustralia
CityBrisbane
Period26/10/15 → 30/10/15

Keywords

  • CNN
  • Deep Learning
  • Multimodal
  • Speaker Naming

Fingerprint

Dive into the research topics of 'Deep multimodal speaker naming'. Together they form a unique fingerprint.

Cite this