A PUBLIC AUDIO IDENTIFICATION EVALUATION FRAMEWORK FOR BROADCAST MONITORING

Figures & data

FIGURE 1 Illustration of the audio fingerprinting workflow. (Figure is provided in color online.)

TABLE 1 Comparative List of the Evaluation Protocols in the Literature, Specifying Corpus and Queries Sizes, Protocols and Score Metrics

Articles	Corpus	Queries	Protocol	Metric
(Allamanche et al. 2001) (Fraunhofer)	15 k	NA	Subsets	Acc + Top 10
(Cano et al. 2002)	50 k	12 h (104)	Subsets + Broad	FP/FN
(Haitsma and Kalker 2002) (Philips)	4	4	Subsets + BER	Nb hits
(Wang 2003) (Shazam)	10 k	250	Subsets	Acc + FP
(Pinquier and Andre-Obrecht 2004)	200	10 h (132)	Broad	Acc
(Seo et al. 2006)	8 k	NA	Subsets	ROC (Pre/Rec)
(Covell and Baluja 2007) (Google)	10 k	1000	Subsets	Acc + ROC (FP/FN)
(Betser, Collen, and Rault 2007)	30	18 h (243)	Broad	Recall
(Kim and Yoo 2007)	NA	7 M	Pairs	ROC (FP/FN)
(Mohri, Moreno, and Weinstein 2008)	15 k	3,600	Subsets	Acc
(Liu, Yun, and Kim 2009)	13 k	2400	Pairs	Acc + ROC (FP/FN)
(Jie, Gang, and Jun 2009)	500	NA	Subsets	Top 1,5,10,20,50
(Jang et al. 2009)	100	44 k	Pairs	Acc + ROC (FP/FN)
(Belletini and Mazzini 2010)	100 k	NA	Subsets	Acc
(Li, Liu, and Xue 2010)	1,822	100	Subsets + BER	Top 1,5,10 + FP/FN
(Smeaton, Over, and Kraaij 2006)(TRECVid)	400 h	12000	Subsets	Balanced FP/FN

FIGURE 2 Possible evaluation scopes.

FIGURE 3 Comparison of the false alarms counting methodologies. (Figure is provided in color online.)

TABLE 2 Pilot Quaero Evaluation Results

System	S OK/N	S FA,1 (S ^out)	R 1	S FA,1.5 (S ^out)	R 1.5	S FA,2 (S ^out)	R 2
Participant 1	445/459	0 (2)	96.5%	0 (2)	96.5%	0 (2)	96.5%
Participant 2	381/459	0 (0)	83.0%	0 (0)	83.0%	0 (0)	83.0%
Participant 3	442/459	0 (2)	95.9%	0 (2)	95.9%	0 (2)	95.9%

Allamanche , E. , J. Herre , O. Hellmuth , B. Fröba , T. Kastner , and M. Cremer . 2001 . Content-based identification of audio material using MPEG-7 low level description. In Proceeding of the international symposium on music information retrieval (ISMIR '01). Bloomington, Indiana, USA .

 Google Scholar

Cano , P. , E. Batlle , H. Mayer , and H. Neuschmied . 2002 . Robust sound modeling for song detection in broadcast audio. In Proc. 112th AES Convention, 1–7. Munich, Germany .

 Google Scholar

Haitsma , J. , and T. Kalker . 2002 . A highly robust audio fingerprinting system. In Proceeding of the international symposium on music information retrieval (ISMIR '02). Paris, France .

 Google Scholar

Wang , A. L. C. 2003 . An industrial-strength audio search algorithm. In Proceedings of the International Symposium on Music Information Retrieval (ISMIR '03). Washington, D.C., USA .

 Google Scholar

Pinquier , J. , and R. André-Obrecht . 2004. Jingle detection and identification in audio documents. In Proceedings of the international conference on acoustics, speech and signal processing (ICASSP '04), vol. 4, 329–332.

 Google Scholar

Seo , J. S. , M. Jin , S. Lee , D. Jang , S. Lee , and C. D. Yoo . 2006 . Audio fingerprinting based on normalized spectral subband moments . IEEE Signal Processing Letters 13 : 209 – 212 .

 Web of Science ®Google Scholar

Covell , M. , and S. Baluja . 2007 . Known-audio detection using waveprint: Spectrogram fingerprinting by wavelet hashing. In Proceeding of the international conference on acoustics, speech and signal processing (ICASSP '07), 237–240. Honolulu, Hawaii, USA .

 Google Scholar

Betser , M. , P. Collen , and J.-B. Rault . 2007 . Audio identification using sinusoidal modeling and application to jingle detection. In Proceeding of the international symposium on music information retrieval (ISMIR '07). Vienna, Austria .

 Google Scholar

Kim , S. , and C. D. Yoo . 2007 . Boosted binary audio fingerprint based on spectral subband moments. In Proceeding of the international conference on acoustics, speech and signal processing (ICASSP '07), vol. 1, 241–244. Honolulu, Hawaii, USA .

 Google Scholar

Mohri , M. , P. Moreno , and E. Weinstein . 2008 . Efficient and robust music identification with weighted finite-state transducers . IEEE Transactions on Audio, Speech and Language Processing 18 : 197 – 207 .

 Web of Science ®Google Scholar

Liu , Y. , H. S. Yun , and N. S. Kim . 2009 . Audio fingerprinting based on multiple hashing in DCT domain . IEEE Signal Processing Letters 16 : 525 – 528 .

 Web of Science ®Google Scholar

Jie , T. , L. Gang , and G. Jun . 2009 . Improved algorithms of music information retrieval based on audio fingerprint. In Proceedings of the 3rd international symposium on intelligent information technology application workshops (IITAW '09) .

 Google Scholar

Jang , D. , C. D. Yoo , S. Lee , S. Kim , and T. Kalker . 2009 . Pairwise boosted audio fingerprint . IEEE Transactions on Information Forensics and Security 4 : 995 – 1004 .

 Web of Science ®Google Scholar

Belletini , C. , and G. Mazzini . 2010 . A framework for robust audio fingerprinting . Journal of Communications 5 : 409 – 424 .

 Google Scholar

Li , W. , Y. Liu , and X. Xue . 2010 . Robust audio identification for MP3 popular music. In Proceeding of the 33rd international ACM SIGIR conference on research and development in information retrieval, 627–634. Geneva, Switzerland .

 Google Scholar