PHRASE AND NGRAM-BASED STATISTICAL MACHINE TRANSLATION SYSTEM COMBINATION

Figures & data

TABLE 1 BTEC Chinese-English Corpus. Basic Statistics for the Training, Development and Test Datasets

		2006 Evaluation
		Chinese		English
Train	Sentences		40.0k
	Words	342.3 k		367.3 k
	Vocabulary	11.2 k		7.2 k
Dev	Sentences		489
	Words	5.7 k		–
	Vocabulary	1,1 k		–
	References		16
Test	Sentences		500
	Words	6,1		–
	Vocabulary	1,3 k		–
	References		16

TABLE 4 Phrase (Left) and Ngram (Right) Default Parameters

	Phrase-Based System
Phrase-length	10
Max target per source	20/30
Language model	4gram Kneser-Ney interpolation
Features	s2 t and t2s lexicon models, word and phrase bonus
Beam	50
Search	Monotonic
	Ngram-Based System
Tuple-length	no limit
Max target per source	20/30
Language model	4gram Kneser-Ney interpolation
Features	s2 t and t2s lexicon models, word bonus
Beam	50
Search	Monotonic

TABLE 2 Results Obtained Using TALPphr (Phrase-Based), TALPtup (Ngram-Based System) and Their Combination

		Dev		Test		Eval
Language	System	BLEU	NIST	BLEU	NIST	BLEU	NIST
Zh2En	TALPphr primary	19.29	6.57	46.33	8.95	20.08	6.42
	TALPtup primary	19.75	6.64	44.63	8.99	20.34	6.22
	TALPcom	21.19	6.69	49.72	8.36	20.21	5.97
Ar2En	TALPphr primary	27.07	7.15	55.34	10.28	22.20	6.54
	TALPtup primary	29.27	7.52	55.11	10.45	23.83	6.80
	TALPcom	30.29	7.41	57.34	10.46	23.95	6.60
It2En	TALPphr primary	41.66	9.08	62.68	10.69	35.55	8.32
	TALPtup primary	43.05	9.21	63.40	10.76	37.38	8.59
	TALPcom	44.13	9.04	63.38	10.43	37.74	8.41
Jp2En	TALPphr primary	15.37	6.01	48.93	9.54	14.51	5.58
	TALPtup primary	16.59	6.34	47.14	9.42	14.61	5.27
	TALPcom	19.36	6.42	51.73	8.8	15.66	5.51

FIGURE 1 Analysis of phrase and Ngram-based systems. From top to bottom, training sentences with the corresponding alignment (source word position hyphen target word position), unit extraction, test sentence, and translation were computed with each system using the extracted units. Word marked with ∗ is unknown for the system.

TABLE 3 TC-STAR Corpus (Spanish-English Task). Basic Statistics for the Training, Development, and Test Datasets

		2nd Evaluation
		Spanish		English
Train	Sentences		1.28 M
	Words	32 M		31 M
	Vocabulary	159 K		111 K
Dev	Sentences	430		430
	Words	15.7 k		16 k
	Vocabulary	3.2 k		2.7 k
	References		3
Test	Sentences	840		1094
	Words	22.7 k		26.8 k
	Vocabulary	4 k		3.9 k
	References		2

TABLE 5 Results of Rescoring and System Combination on the Development Set. PB-Phrase-Based, NB-Ngram-Based

System	n-best	BLEU	λ pb	λ nb
Es2En
PB	1	45.45	1	–
PB	100	45.84	1	2.67
PB	1000	46.0	1	2.5
NB	1	46.01	–	1
NB	100	46.32	0.23	1
NB	1000	46.32	0.21	1
PB+NB	2	46.37	0.25	1
PB+NB	200	46.36	0.89	1
PB+NB	2000	46.47	0.18	1
PB	1	47.87	1	–
En2Es
PB	100	48.57	1	1.96
PB	1000	48.66	1	1.16
NB	1	48.08	–	1
NB	100	48.3	0.50	1
NB	1000	48.35	1	1.16
PB+ NB	2	48.32		1
PB+ NB	200	48.51	0.55	1
PB+ NB	2000	48.54	0.50	1

TABLE 6 Rescoring and System Combination Results

System	n-best	BLEU	NIST	mWER
Es2En
PB	1	51.90	10.54	37.50
PB	100	52.45	10.59	37.17
PB	1000	52.55	10.61	37.12
NB	1	51.63	10.46	37.88
NB	100	52.16	10.54	37.53
NB	1000	52.25	10.55	37.43
PB+NB	2	51.77	10.49	37.68
PB+NB	200	52.12	10.54	37.58
PB+NB	2000	52.31	10.56	37.32
En2Es
PB	1	47.75	9.94	41.2
PB	100	48.17	10.07	40.30
PB	1000	48.46	10.13	39.98
NB	1	47.73	10.09	40.50
NB	100	48.27	10.15	40.11
NB	1000	48.33	10.15	40.13
PB+NB	2	48.26	10.05	40.61
PB+NB	200	48.49	10.15	40.05
PB+NB	2000	48.54	10.16	40.00

TABLE 7 Impact on Efficiency of the Beam Size in PB (Top) and NB System (Bottom)

Task	Beam	Time(s)	Units
Es2En	50	2,677
	10	852	537 k
	5	311
En2Es	50	2,689
	10	903	594 k
	5	329
Es2En	50	1,264
	10	281	104 k 288 k 145 k
	5	138
En2Es	50	1,508
	10	302	118 k 355 k 178 k
	5	155

TABLE 8 Impact on Accuracy of the Beam Size in PB (Top) and NB System (Bottom)

Task	Beam	BLEU	NIST	mWER
	50	51.90	10.53	37.54
Es2En	10	51.93	10.54	37.49
	5	51.87	10.55	37.47
	50	47.75	9.94	41.20
En2Es	10	47.77	9.96	41.09
	5	47.86	10.00	40.74
	50	51.63	10.46	37.88
Es2En	10	51.50	10.45	37.83
	5	51.39	10.45	37.85
	50	47.73	10.08	40.50
En2Es	10	46.82	9.97	41.04
	5	45.59	9.83	41.04