Intelligent Team Building Recommendation System Perplexity Rules

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 363?374,
Edinburgh, Scotland, UK, July 27?31, 2011. c?2011 Association for Computational Linguistics
Language Models for Machine Translation:
Original vs. Translated Texts
Gennadi Lembersky and Noam Ordan and Shuly Wintner
Department of Computer Science, University of Haifa, 31905 Haifa, Israel
glembers@campus.haifa.ac.il, noam.ordan@gmail.com, shuly@cs.haifa.ac.il
Abstract
We investigate the differences between
language models compiled from original
target-language texts and those compiled
from texts manually translated to the tar-
get language. Corroborating established
observations of Translation Studies, we
demonstrate that the latter are signifi-
cantly better predictors of translated sen-
tences than the former, and hence fit the
reference set better. Furthermore, trans-
lated texts yield better language mod-
els for statistical machine translation than
original texts.
1 Introduction
Statistical machine translation (MT) uses large
target language models (LMs) to improve the
fluency of generated texts, and it is commonly
assumed that for constructing language mod-
els, ?more data is better data? (Brants and Xu,
2009). Not all data, however, are created the
same. In this work we explore the differences be-
tween LMs compiled from texts originally writ-
ten in the target language and LMs compiled
from translated texts.
The motivation for our work stems from much
research in Translation Studies that suggests
that original texts are significantly different
from translated ones in various aspects (Geller-
stam, 1986). Recently, corpus-based compu-
tational analysis corroborated this observation,
and Kurokawa et al (2009) apply it to sta-
tistical machine translation, showing that for
an English-to-French MT system, a transla-
tion model trained on an English-translated-to-
French parallel corpus is better than one trained
on French-translated-to-English texts. Our re-
search question is whether a language model
compiled from translated texts may similarly
improve the results of machine translation.
We test this hypothesis on several translation
tasks, where the target language is always En-
glish. For each language pair we build two En-
glish language models from two types of corpora:
texts originally written in English, and human
translations from the source language into En-
glish. We show that for each language pair, the
latter language model better fits a set of refer-
ence translations in terms of perplexity. We also
demonstrate that the differences between the
two LMs are not biased by content but rather
reflect differences on abstract linguistic features.
Research in Translation Studies suggests that
all translated texts, irrespective of source lan-
guage, share some so-called translation univer-
sals. Consequently, translated texts from sev-
eral languages to a single target language resem-
ble each other along various axes. To test this
hypothesis, we compile additional English LMs,
this time using texts translated to English from
languages other than the source. Again, we use
perplexity to assess the fit of these LMs to refer-
ence sets of translated-to-English sentences. We
show that these LMs depend on the source lan-
guage and differ from each other. Whereas they
outperform original-based LMs, LMs compiled
from texts that were translated from the source
language still fit the reference set best.
Finally, we train phrase-based MT systems
(Koehn et al, 2003) for each language pair. We
use four types of LMs: original; translated from
363
the source language; translated from other lan-
guages; and a mixture of translations from sev-
eral languages. We show that the translated-
from-source-language LMs provide a significant
improvement in the quality of the translation
output over all other LMs, and that the mix-
ture LMs always outperform the original LMs.
This improvement persists even when the orig-
inal LMs are up to ten times larger than the
translated ones.
The main contributions of this work are there-
fore a computational corroboration of the hy-
potheses that
1. original and translated texts exhibit signif-
icant, measurable differences;
2. LMs compiled from translated texts better
fit translated references than LMs compiled
from original texts of the same (and much
larger) size (and, to a lesser extent, LMs
compiled from texts translated from lan-
guages other than the source language); and
3. MT systems that use LMs based on man-
ually translated texts significantly outper-
form LMs based on originally written texts.
It is important to emphasize that translated
texts abound: Many languages, especially low-
resource ones, are more likely to have translated
texts (religious scripts, educational materials,
etc.) than original ones. Some numeric data
are listed in Pym and Chrupa la (2005). Fur-
thermore, such data can be automatically identi-
fied (see Section 2). The practical impact of our
work on MT is therefore potentially dramatic.
This paper is organized as follows: Section 2
provides background and describes related work.
We explain our research methodology and re-
sources in Section 3 and detail our experiments
and results in Section 4. Section 5 discusses the
results and their implications.
2 Background and Related Work
Numerous studies suggest that translated texts
are different from original ones. Gellerstam
(1986) compares texts written originally in
Swedish and texts translated from English into
Swedish. He notes that the differences between
them do not indicate poor translation but rather
a statistical phenomenon, which he terms trans-
lationese. He focuses mainly on lexical dif-
ferences, for example less colloquialism in the
translations, or foreign words used in the trans-
lations ?with new shades of meaning taken from
the English lexeme? (p.91). Only later studies
consider grammatical differences (see, e.g., San-
tos (1995)). The features of translationese were
theoretically organized under the terms laws of
translation and translation universals.
Toury (1980, 1995) distinguishes between two
laws: the law of interference and the law of
growing standardization. The former pertains
to the fingerprints of the source text that are
left in the translation product. The latter per-
tains to the effort to standardize the translation
product according to existing norms in the tar-
get language (and culture). Interestingly, these
two laws are in fact reflected in the architecture
of statistical machine translation: interference
corresponds to the translation model and stan-
dardization to the language model.
The combined effect of these laws creates a hy-
brid text that partly corresponds to the source
text and partly to texts written originally in the
target language but in fact belongs to neither
(Frawley, 1984). Baker (1993, 1995, 1996) sug-
gests several candidates for translation univer-
sals, which are claimed to appear in any trans-
lated text, regardless of the source language.
These include simplification, the tendency of
translated texts to simplify the language, the
message or both; and explicitation, their ten-
dency to spell out implicit utterances that occur
in the source text.
Baroni and Bernardini (2006) use machine
learning techniques to distinguish between origi-
nal and translated Italian texts, reporting 86.7%
accuracy. They manage to abstract from con-
tent and perform the task using only morpho-
syntactic cues. Ilisei et al (2010) perform the
same task for Spanish but enhance it theoreti-
cally in order to check the simplification hypoth-
esis. The most informative features are lexical
variety, sentence length and lexical density.
van Halteren (2008) focuses on six languages
from Europarl (Koehn, 2005): Dutch, English,
French, German, Italian and Spanish. For each
364
of these languages, a parallel six-lingual sub-
corpus is extracted, including an original text
and its translations into the other five languages.
The task is to identify the source language of
translated texts, and the reported results are ex-
cellent. This finding is crucial: as Baker (1996)
states, translations do resemble each other; how-
ever, in accordance with the law of interference,
the study of van Halteren (2008) suggests that
translation from different source languages con-
stitute different sublanguages. As we show in
Section 4.2, LMs based on translations from the
source language outperform LMs compiled from
non-source translations, in terms of both fitness
to the reference set and improving MT.
Kurokawa et al (2009) show that the direction
of translation affects the performance of statis-
tical MT. They train systems to translate be-
tween French and English (and vice versa) us-
ing a French-translated-to-English parallel cor-
pus, and then an English-translated-to-French
one. They find that in translating into French
it is better to use the latter parallel corpus, and
when translating into English it is better to use
the former. Whereas they focus on the trans-
lation model, we focus on the language model
in this work. We show that using a LM trained
on a text translated from the source language of
the MT system does indeed improve the results
of the translation.
3 Methodology and Resources
3.1 Hypotheses
We investigate the following three hypotheses:
1. Translated texts differ from original texts;
2. Texts translated from one language differ
from texts translated from other languages;
3. LMs compiled from manually translated
texts are better for MT as measured using
BLEU than LMs compiled from original texts.
We test our hypotheses by considering trans-
lations from several languages to English. For
each language pair we create a reference set com-
prising several thousands of sentences written
originally in the source language and manually
translated to English. Section 3.4 provides de-
tails on the reference sets.
To investigate the first hypothesis, we train
two LMs for each language pair, one created
from original English texts and the other from
texts translated into English. Then, we check
which LM better fits the reference set.
Fitness of a LM to a set of sentences is mea-
sured in terms of perplexity (Jelinek et al, 1977;
Bahl et al, 1983). Given a language model and
a test (reference) set, perplexity measures the
predictive power of the language model over the
test set, by looking at the average probability
the model assigns to the test data. Intuitively,
a better model assigns higher probablity to the
test data, and consequently has a lower perplex-
ity; it is less surprised by the test data. For-
mally, the perplexity PP of a language model L
on a test set W = w1 w2 . . . wN is the probabil-
ity of W normalized by the number of words N
Jurafsky and Martin (2008, page 96):
PP(L,W ) = N
????
N?
i=1
1
PL(wi|w1 . . . wi?1)
(1)
For the second hypothesis, we extend the ex-
periment to LMs created from texts translated
from other languages to English. For exam-
ple, we test how well a LM trained on French-
to-English-translated texts fits the German-to-
English reference set; and how well a LM trained
on German-to-English-translated texts fits the
French-to-English reference set.
Finally, for the third hypothesis, we use these
LMs for statistical MT (SMT). For each lan-
guage pair we build several SMT systems. All
systems use a translation model extracted from
a parallel corpus which is oblivious to the direc-
tion of the translation; and one of the above-
mentioned LMs. Then, we compare the trans-
lation quality of these systems in terms of the
BLEU metric (Papineni et al, 2002).
3.2 Language Models
In all the experiments, we use SRILM (Stolcke,
2002) to train 4-gram language models (with
the default backoff model) from various corpora.
Our main corpus is Europarl (Koehn, 2005),
specifically portions collected over years 1996 to
365
1999 and 2001 to 2009. This is a large multi-
lingual corpus, containing sentences translated
from several European languages. However, it
is organized as a collection of bilingual corpora
rather than as a single multilingual one, and it
is hard to identify sentences that are translated
to several languages.
We therefore treat each bilingual sub-corpus
in isolation; each such sub-corpus contains sen-
tences translated from various languages. We
rely on the language attribute of the speaker
tag to identify the source language of sentences
in the English part of the corpus. Since this tag
is rarely used with English-language speakers,
we also exploit the ID attribute of the speaker
tag, which we match against the list of British
members of the European parliament.
We focus on the following languages: Ger-
man (DE), French (FR), Italian (IT), and Dutch
(NL). For each of these languages, L, we con-
sider the L-English Europarl sub-corpus. In
each sub-corpus, we extract chunks of approx-
imately 2.5 million English tokens translated
from each of these source languages (T-L), as
well as sentences written originally in English
(O-EN). The mixture corpus (MIX), which is
designed to represent ?general? translated lan-
guage, is constructed by randomly selecting sen-
tences translated from any language (excluding
original English sentences). Table 1 lists the
number of sentences, number of tokens and av-
erage sentence length, for each sub-corpus and
each original language.
In addition, we use the Hansard corpus, con-
taining transcripts of the Canadian parliament
from 1996?20071. This is a bilingual French?
English corpus comprising about 80% original
English texts (EO) and about 20% texts trans-
lated from French (FO). We first separate orig-
inal English from the original French and then,
for each original language, we randomly extract
portions of texts of different sizes: 1M, 5M and
10M tokens from the FO corpus and 1M, 5M,
10M, 25M, 50M and 100M tokens from the EO
corpus; see Table 2.
1We are grateful to Cyril Goutte, George Foster and
Pierre Isabelle for providing us with an annotated version
of this corpus.
German?English
Orig. Lang. Sent?s Tokens Len
MIX 82,700 2,325,261 28.1
O-EN 91,100 2,324,745 25.5
T-DE 87,900 2,322,973 26.4
T-FR 77,550 2,325,183 30.0
T-IT 65,199 2,325,996 35.7
T-NL 94,000 2,323,646 24.7
French?English
Orig. Lang. Sent?s Tokens Len
MIX 90,700 2,546,274 28.1
O-EN 99,300 2,545,891 25.6
T-DE 94,900 2,546,124 26.8
T-FR 85,750 2,546,085 29.7
T-IT 72,008 2,546,984 35.4
T-NL 103,350 2,545,645 24.6
Italian?English
Orig. Lang. Sent?s Tokens Len
MIX 87,040 2,534,793 29.1
O-EN 93,520 2,534,892 27.1
T-DE 90,550 2,534,867 28.0
T-FR 82,930 2,534,930 30.6
T-IT 69,270 2,535,225 36.6
T-NL 96,850 2,535,053 26.2
Dutch?English
Orig. Lang. Sent?s Tokens Len
MIX 90,500 2,508,265 27.7
O-EN 97,000 2,475,652 25.5
T-DE 94,200 2,503,354 26.6
T-FR 86,600 2,523,055 29.1
T-IT 73,541 2,518,196 34.2
T-NL 101,950 2,513,769 24.7
Table 1: Europarl corpus statistics
To experiment with a non-European language
(and a different genre) we choose Hebrew (HE).
We use two English corpora: The original (O-
EN) corpus comprises articles from the Interna-
tional Herald Tribune, downloaded over a pe-
riod of seven months (from January to July
2009). The articles cover four topics: news
(53.4%), business (20.9%), opinion (17.6%) and
arts (8.1%). The translated (T-HE) corpus con-
sists of articles collected from the Israeli news-
paper HaAretz over the same period of time.
HaAretz is published in Hebrew, but portions of
366
Original French
Size Sent?s Tokens Len
1M 54,851 1,000,076 18.23
5M 276,187 5,009,157 18.14
10M 551,867 10,001,716 18.12
Original English
Size Sent?s Tokens Len
1M 54,216 1,006,275 18.56
5M 268,806 5,006,482 18.62
10M 537,574 10,004,191 18.61
25M 1,344,580 25,001,555 18.59
50M 2,689,332 50,009,861 18.60
100M 5,376,886 100,016,704 18.60
Table 2: Hansard corpus statistics
it are translated to English. The O-corpus was
downsized, so both corpora had approximately
the same number of tokens in each topic. Ta-
ble 3 lists basic statistics for these corpora.
Hebrew?English
Orig. Lang. Sent?s Tokens Len
O-EN 135,228 3,561,559 26.3
T-HE 147,227 3,561,556 24.2
Table 3: Hebrew-to-English corpus statistics
3.3 SMT Training Data
To focus on the effect of the language model
on translation quality, we design SMT train-
ing corpora to be oblivious to the direction of
translation. Again, we use Europarl (January
2000 to September 2000) as the main source of
our parallel corpora. We also use the Hansard
corpus: We randomy extract 50,000 sentences
from the original French sub-corpora and an-
other 50,000 sentences from the original English
sub-corpora. For Hebrew we use the Hebrew?
English parallel corpus (Tsvetkov and Wintner,
2010) which contains sentences translated from
Hebrew to English (54%) and from English to
Hebrew (46%). The English-to-Hebrew part
comprises many short sentences (approximately
6 tokens per sentence) taken from a movie sub-
title database. This explains the small token to
sentence ratio of this particular corpus. Table 4
lists some details on those corpora.
Lang?s Side Sent?s Tokens Len
DE-EN DE 92,901 2,439,370 26.3EN 92,901 2,602,376 28.0
FR-EN FR 93,162 2,610,551 28.0EN 93,162 2,869,328 30.8
IT-EN IT 85,485 2,531,925 29.6EN 85,485 2,517,128 29.5
NL-EN NL 84,811 2,327,601 27.4EN 84,811 2,303,846 27.2
Hansard FR 100,000 2,167,546 21.7EN 100,000 1,844,415 18.4
HE-EN HE 95,912 726,512 7.6EN 95,912 856,830 8.9
Table 4: SMT training data details
3.4 Reference Sets
The reference sets have two uses. First, they
are used as the test sets in the experiments that
measure the perplexity of the language models.
Second, in the MT experiments we use them to
randomly extract 1000 sentences for tuning and
1000 (different) sentences for evaluation.
For each language L we use the L-English sub-
corpus of Europarl (over the period of October
to December 2000), containing only sentences
originally produced in language L. The Hansard
reference set is completely disjoint from the LM
and SMT training sets and comprises only orig-
inal French sentences. The Hebrew-to-English
reference set is an independent (disjoint) part
of the Hebrew-to-English parallel corpus. This
set mostly comprises literary data (88.6%) and a
small portion of news (11.4%). All sentences are
originally written in Hebrew and are manually
translated to English. See Table 5.
4 Experiments and Results
We detail in this section the experiments per-
formed to test the three hypotheses: that trans-
lated texts can be distinguished from original
ones, and provide better language models of
other translated texts; that texts translated
from other languages than the source are still
better predictors of translations than original
texts (Section 4.1); and that these differences
are important for SMT (Section 4.2).
367
Lang?s Side Sent?s Tokens Len
DE-EN DE 6,675 161,889 24.3EN 6,675 178,984 26.8
FR-EN FR 8,494 260,198 30.6EN 8,494 271,536 32.0
IT-EN IT 2,269 82,261 36.3EN 2,269 78,258 34.5
NL-EN NL 4,593 114,272 24.9EN 4,593 105,083 22.9
Hansard FR 8,926 193,840 21.72EN 8,926 163,448 18.3
HE-EN HE 7,546 102,085 13.5EN 7,546 126,183 16.7
Table 5: Reference sets
4.1 Translated vs. Original texts
We train several 4-gram LMs for each Europarl
sub-corpus, based on the corpora described in
Section 3.2. For each language L, we train a
LM based on texts translated from L, from lan-
guages other than L as well as texts originally
written in English. The LMs are applied to the
reference set of texts translated from L, and we
compute the perplexity: the fitness of the LM
to the reference set. Table 6 details the results,
where for each sub-corpus and LM we list the
number of unigrams in the test set, the num-
ber of out-of-vocabulary items (OOV) and the
perplexity (PP). The lowest perplexity (reflect-
ing the best fit) in each sub-corpus is typeset in
boldface, and the highest (worst fit) is slanted.
These results overwhelmingly support our hy-
pothesis. For each language L, the perplexity
of the LM that was created from L transla-
tions is lowest, followed immediately by the MIX
LM. Furthermore, the perplexity of the LM cre-
ated from originally-English texts is highest in
all experiments. In addition, the perplexity of
LMs constructed from texts translated from lan-
guages other than L always lies between these
two extremes: it is a better fit of the refer-
ence set than original texts, but not as good
as texts translated from L (or mixture trans-
lations). This corroborates the hypothesis that
translations form a language in itself, and trans-
lations from L1 to L2, form a sub-language,
related to yet different from translations from
German to English translations
Orig. Lang. Unigrams OOV PP
MIX 32,238 961 83.45
O-EN 31,204 1161 96.50
T-DE 27,940 963 77.77
T-FR 29,405 1141 92.71
T-IT 28,586 1122 95.14
T-NL 28,074 1143 89.17
French to English translations
Orig. Lang. Unigrams OOV PP
MIX 33,444 1510 87.13
O-EN 32,576 1961 105.93
T-DE 28,935 2191 96.83
T-FR 30,609 1329 82.23
T-IT 29,633 1776 91.15
T-NL 29,221 2148 100.18
Italian to English translations
Orig. Lang. Unigrams OOV PP
MIX 33,353 462 90.71
O-EN 32,546 633 107.45
T-DE 28,835 628 100.46
T-FR 30,460 524 92.18
T-IT 29,466 470 80.57
T-NL 29,130 675 105.07
Dutch to English translations
Orig. Lang. Unigrams OOV PP
MIX 33,050 651 87.37
O-EN 32,064 771 100.75
T-DE 28,766 778 90.35
T-FR 30,502 775 96.38
T-IT 29,386 916 99.26
T-NL 29,178 560 78.25
Table 6: Fitness of various LMs to the reference set
other languages to L2.
A possible explanation for the different per-
plexity results between the LMs could be the
specific contents of the corpora used to com-
pile the LMs. To rule out this possibility and
to further emphasize that the corpora are in-
deed structurally different, we conduct more ex-
periments, in which we gradually abstract away
from the domain- and content-specific features
of the texts and emphasize their syntactic struc-
ture. We focus on German-to-English.
First, we remove all punctuation to eliminate
368
possible bias due to differences in punctuation
conventions. Then, we use the Stanford Named
Entity Recognizer (Finkel et al, 2005) to iden-
tify named entities, which we replace with a
unique token (?NE?). Next, we replace all nouns
with their POS tag; we use the Stanford POS
Tagger (Toutanova and Manning, 2000). Fi-
nally, for full lexical abstraction, we replace all
words with their POS tags.
At each step, we train six language models on
O- and T-texts and apply them to the reference
set (adapted to the same level of abstraction,
of course). As the abstraction of the text in-
creases, we also increase the order of the LMs:
From 4-grams for text without punctuation and
NE abstraction to 5-grams for noun abstraction
to 8-grams for full POS abstraction. The results,
which are depicted in Table 7, consistently show
that the T-based LM is a better fit to the ref-
erence set, albeit to a lesser extent. While we
do not show the details here, the same pattern
is persistent in all the other Europarl languages
we experiment with.
We repeat this experiment with the Hebrew-
to-English reference set. We train two 4-gram
LMs on the O-EN and T-HE corpora. We then
apply the two LMs to the reference set and com-
pute the perplexity. The results are presented
in Table 8. Although the T-based LM has more
OOVs, it is a better fit to the translated text
than the O-based LM: Its perplexity is lower
by 20.1%. Interestingly, the O-corpus LM has
more unique unigrams than the T-corpus LM,
supporting the claim of Al-Shabab (1996) that
translated texts have lower type-to-token ratio.
We also conduct the above-mentioned ab-
straction experiments. The results, which are
depicted in Table 9, consistently show that the
T-based LM is a better fit to the reference set.
Clearly, then, translated LMs better fit the
references than original ones, and the differences
can be traced back not just to (trivial) specific
lexical choice, but also to syntactic structure, as
evidenced by the POS abstraction experiments.
In fact, in order to retain the low perplexity level
of translated texts, a LM based on original texts
must be approximately ten times larger. We es-
tablish this by experimenting with the Hansard
No Punctuation
Orig. Lang. OOVs PP PP diff.
MIX 770 109.36 7.58%
O-EN 946 127.03 20.43%
T-DE 795 101.07 0.00%
T-FR 909 122.03 17.18%
T-IT 991 125.36 19.38%
T-NL 936 117.37 13.89%
NE Abstraction
Orig. Lang. OOVs PP PP diff.
MIX 643 99.13 6.99%
O-EN 772 114.19 19.26%
T-DE 661 92.20 0.00%
T-FR 752 110.22 16.35%
T-IT 823 112.72 18.21%
T-NL 771 105.81 12.86%
Noun Abstraction
Orig. Lang. OOVs PP PP diff.
MIX 400 38.48 4.71%
O-EN 459 42.06 12.80%
T-DE 405 36.67 0.00%
T-FR 472 40.96 10.47%
T-IT 489 41.39 11.39%
T-NL 440 39.54 7.26%
POS Abstraction
Orig. Lang. OOVs PP PP diff.
MIX 0 8.02 1.22%
O-EN 0 8.19 3.31%
T-DE 0 7.92 0.00%
T-FR 0 8.10 2.16%
T-IT 0 8.12 2.50%
T-NL 0 8.03 1.42%
Table 7: Fitness of O- vs. T-based LMs to the refer-
ence set (DE-EN), different abstraction levels
corpus. The results are persistent, but are omit-
ted for lack of space.
4.2 Original vs. Translated LMs for MT
The last hypothesis we test is whether a bet-
ter fitting language model yields a better ma-
chine translation system. In other words, we
expect the T-based LMs to outperform the O-
based LMs when used as part of an MT sys-
tem. We construct German-to-English, French-
to-English, Italian-to-English and Dutch-to-
369
Hebrew to English translations
Orig. Lang. Unigrams OOV PP
O-EN 74,305 2,955 282.75
T-HE 61,729 3,253 226.02
Table 8: Fitness of O- vs. T-based LMs to the refer-
ence set (HE-EN)
No Punctuation
Orig. Lang. OOVs PP PP diff.
O-EN 2,601 442.95 19.2%
T-HE 2,922 358.11 0.0%
NE Abstraction
Orig. Lang. OOVs PP PP diff.
O-EN 1,794 350.3 17.3%
T-HE 2,038 289.71 0.0%
Noun Abstraction
Orig. Lang. OOVs PP PP diff.
O-EN 679 93.31 12.4%
T-HE 802 81.72 0.0%
POS Abstraction
Orig. Lang. OOVs PP PP diff.
O-EN 0 11.47 6.2%
T-HE 0 10.76 0.0%
Table 9: Fitness of O- vs. T-based LMs to the refer-
ence set (HE-EN), different abstraction levels
English MT systems using the Moses phrase-
based SMT toolkit (Koehn et al, 2007). The
systems are trained on the parallel corpora de-
scribed in Section 3.3. We use the reference sets
(Section 3.4) as follows: 1,000 sentences are ran-
domly extracted for minimum error-rate tuning
(Och, 2003), and another set of 1,000 sentences
is randomly used for evaluation. Each system
is built and tuned with six different LMs: MIX,
O-based and four T-based (Section 3.2). We use
BLEU (Papineni et al, 2002) to evaluate trans-
lation quality. The results are listed in Table 10.
These results are consistent: the translated-
from-source systems outperform all other sys-
tems; mixture models come second; and systems
that use original English LMs always perform
worst. We test the statistical significance of dif-
ferences between various MT systems using the
bootstrap resampling method (Koehn, 2004). In
all experiments, the best system (translated-
from-source LM) is significantly better than all
DE to EN
LM BLEU
MIX 21.95
O-EN 21.35
T-DE 22.42
T-FR 21.47
T-IT 21.79
T-NL 21.59
FR to EN
LM BLEU
MIX 25.43
O-EN 24.85
T-DE 25.03
T-FR 25.91
T-IT 25.44
T-NL 25.17
IT to EN
LM BLEU
MIX 26.79
O-EN 25.69
T-DE 25.86
T-FR 26.56
T-IT 27.28
T-NL 25.77
NL to EN
LM BLEU
MIX 25.17
O-EN 24.46
T-DE 25.12
T-FR 24.79
T-IT 24.93
T-NL 25.73
Table 10: Machine translation with various LMs
other systems (p < 0.05); (even more) signifi-
cantly better than the O-EN system (p < 0.01);
and the mixture systems are significantly better
than the O-EN systems (p < 0.01).
We also construct a Hebrew-to-English MT
system using Moses? factored translation model
(Koehn and Hoang, 2007). Every token in the
training corpus is represented as two factors:
surface form and lemma. Moreover, the Hebrew
input is fully segmented. The system is built
and tuned with O- and T-based LMs. Table 11
depicts the performance of the systems. The
T-based LM yields a statistically better BLEU
score than the O-based system.
LM BLEU p-value
O-based LM 11.98 0.012
T-based LM 12.57
Table 11: Hebrew-to-English MT results
The LMs used in the above experiments are
small. We now want to assess whether the ben-
efits of using translated LMs carry over to sce-
narios where large original corpora exist. We
build yet another set of French-to-English MT
systems. We use the Hansard SMT transla-
tion model and Hansard LMs to train nine MT
systems, three with varying sizes of translated
texts and six with varying sizes of original texts.
370
We tune and evaluate on the Hansard reference
set. In another set of experiments we use the
Europarl French-to-English scenario (using Eu-
roparl corpora for the translation model and
for tuning and evaluation), but we use the nine
Hansard LMs to see whether our findings are
consistent also when LMs are trained on out-of-
domain (but similar genre) material.
Table 12 shows that the original English LMs
should be enlarged by a factor of ten to achieve
translation quality similar to that of translation-
based LMs. In other words, much smaller trans-
lated LMs perform better than much larger orig-
inal ones, and this is true for various LM sizes.
In-Domain
Original French
Size BLEU
1M 34.05
5M 35.12
10M 35.65
Original English
Size BLEU
1M 32.57
5M 33.37
10M 33.92
25M 34.71
50M 34.85
100M 35.36
Out-of-Domain
Original French
Size BLEU
1M 18.87
5M 23.90
10M 24.36
Original English
Size BLEU
1M 18.68
5M 23.02
10M 23.45
25M 23.82
50M 23.95
100M 24.16
Table 12: The effect of LM size on MT performance
5 Discussion
We use language models computed from dif-
ferent types of corpora to investigate whether
their fitness to a reference set of translated-
to-English sentences can differentiate between
them (and, hence, between the corpora on which
they are based). Our main findings are that LMs
compiled from manually translated corpora are
much better predictors of translated texts than
LMs compiled from original-language corpora of
the same size. The results are robust, and are
sustainable even when the corpora and the refer-
ence sentences are abstracted in ways that retain
their syntactic structure but ignore specific word
meanings. Furthermore, we show that trans-
lated LMs are better predictors of translated
sentences even when the LMs are compiled from
texts translated from languages other than the
source language. However, LMs based on texts
translated from the source language still outper-
form LMs translated from other languages.
We also show that MT systems based on
translated-from-source-language LMs outper-
form MT systems based on originals LMs or
LMs translated from other languages. Again,
these results are robust and the improvements
are statistically significant. This effect seems
to be amplified as translation quality improves.
Furthermore, our results show that original LMs
require ten times more data to exhibit the same
fitness to the reference set and the same trans-
lation quality as translated LMs.
More generally, this study confirms that in-
sights drawn from the field of theoretical trans-
lation studies, namely the dual claim according
to which (1) translations as such differ from orig-
inals, and (2) translations from different source
languages differ from each other, can be veri-
fied experimentally and contribute to the per-
formance of machine translation.
Future research is needed in order to un-
derstand why this is the case. One plausi-
ble hypothesis is that recurrent multiword ex-
pressions in the source language are frequently
solved by human translations and each of these
expressions converges to a set of high-quality
translation equivalents which are represented
in the LM. Another hypothesis is that since
translation-based LMs represent a simplified
mode of language use, the error potential is
smaller. We therefore expect translation-based
LMs to use more unmarked forms.
This work also bears on language typology:
we conjecture that LMs compiled from texts
translated not from the original language, but
from a closely related one, can be better than
texts translated from a more distant language.
Some of our results support this hypothesis, but
more research is needed in order to establish it.
Acknowledgements
This research was supported by the Israel Sci-
ence Foundation (grant No. 137/06). We are
grateful to Alon Lavie for his consistent help.
371
References
Omar S. Al-Shabab. Interpretation and the lan-
guage of translation: creativity and conven-
tions in translation. Janus, Edinburgh, 1996.
Lalit R. Bahl, Frederick Jelinek, and Robert L.
Mercer. A maximum likelihood approach to
continuous speech recognition. IEEE Trans-
actions on Pattern Analysis and Machine In-
telligence, 5(2):179?190, 1983.
Mona Baker. Corpus linguistics and transla-
tion studies: Implications and applications. In
Gill Francis Mona Baker and Elena Tognini-
Bonelli, editors, Text and technology: in hon-
our of John Sinclair, pages 233?252. John
Benjamins, Amsterdam, 1993.
Mona Baker. Corpora in translation studies: An
overview and some suggestions for future re-
search. Target, 7(2):223?243, September 1995.
Mona Baker. Corpus-based translation studies:
The challenges that lie ahead. In Gill Fran-
cis Mona Baker and Elena Tognini-Bonelli,
editors, Terminology, LSP and Translation.
Studies in language engineering in honour of
Juan C. Sager, pages 175?186. John Ben-
jamins, Amsterdam, 1996.
Marco Baroni and Silvia Bernardini. A new
approach to the study of Translationese:
Machine-learning the difference between orig-
inal and translated text. Literary and Lin-
guistic Computing, 21(3):259?274, September
2006. URL http://llc.oxfordjournals.
org/cgi/content/short/21/3/259?rss=1.
Thorsten Brants and Peng Xu. Distributed
language models. In Proceedings of Human
Language Technologies: The 2009 Annual
Conference of the North American Chapter
of the Association for Computational Lin-
guistics, Companion Volume: Tutorial Ab-
stracts, pages 3?4, Boulder, Colorado, May
2009. Association for Computational Lin-
guistics. URL http://www.aclweb.org/
anthology/N/N09/N09-4002.
Jenny Rose Finkel, Trond Grenager, and
Christopher Manning. Incorporating non-
local information into information extraction
systems by gibbs sampling. In ACL ?05: Pro-
ceedings of the 43rd Annual Meeting on Asso-
ciation for Computational Linguistics, pages
363?370, Morristown, NJ, USA, 2005. Asso-
ciation for Computational Linguistics. doi:
http://dx.doi.org/10.3115/1219840.1219885.
William Frawley. Prolegomenon to a theory
of translation. In William Frawley, editor,
Translation. Literary, Linguistic and Philo-
sophical Perspectives, pages 159?175. Univer-
sity of Delaware Press, Newark, 1984.
Martin Gellerstam. Translationese in Swedish
novels translated from English. In Lars
Wollin and Hans Lindquist, editors, Trans-
lation Studies in Scandinavia, pages 88?95.
CWK Gleerup, Lund, 1986.
Iustina Ilisei, Diana Inkpen, Gloria Corpas
Pastor, and Ruslan Mitkov. Identification
of translationese: A machine learning ap-
proach. In Alexander F. Gelbukh, editor,
Proceedings of CICLing-2010: 11th Interna-
tional Conference on Computational Linguis-
tics and Intelligent Text Processing, volume
6008 of Lecture Notes in Computer Science,
pages 503?511. Springer, 2010. ISBN 978-3-
642-12115-9. URL http://dx.doi.org/10.
1007/978-3-642-12116-6.
Frederick Jelinek, Robert L. Mercer, Lalit R.
Bahl, and J. K. Baker. Perplexity?a measure
of the difficulty of speech recognition tasks.
Journal of the Acoustical Society of America,
62:S63, November 1977. Supplement 1.
Daniel Jurafsky and James H. Martin. Speech
and Language Processing: An Introduction to
Natural Language Processing, Computational
Linguistics and Speech Recognition. Prentice
Hall, second edition, February 2008. ISBN
013122798X. URL http://www.worldcat.
org/isbn/013122798X.
Philipp Koehn. Statistical significance tests
for machine translation evaluation. In Pro-
ceedings of EMNLP 2004, pages 388?395,
Barcelona, Spain, July 2004. Association for
Computational Linguistics.
Philipp Koehn. Europarl: A parallel corpus for
372
statistical machine translation. MT Summit,
2005.
Philipp Koehn and Hieu Hoang. Factored trans-
lation models. In Proceedings of the 2007
Joint Conference on Empirical Methods in
Natural Language Processing and Computa-
tional Natural Language Learning (EMNLP-
CoNLL), pages 868?876, Prague, Czech Re-
public, June 2007. Association for Computa-
tional Linguistics. URL http://www.aclweb.
org/anthology/D/D07/D07-1091.
Philipp Koehn, Franz Josef Och, and Daniel
Marcu. Statistical phrase-based translation.
In NAACL ?03: Proceedings of the 2003 Con-
ference of the North American Chapter of
the Association for Computational Linguistics
on Human Language Technology, pages 48?
54. Association for Computational Linguis-
tics, 2003.
Philipp Koehn, Hieu Hoang, Alexandra Birch,
Chris Callison-Burch, Marcello Federico,
Nicola Bertoldi, Brooke Cowan, Wade Shen,
Christine Moran, Richard Zens, Chris Dyer,
Ondrej Bojar, Alexandra Constantin, and
Evan Herbst. Moses: Open source toolkit for
statistical machine translation. In Proceed-
ings of the 45th Annual Meeting of the Asso-
ciation for Computational Linguistics Com-
panion Volume Proceedings of the Demo
and Poster Sessions, pages 177?180, Prague,
Czech Republic, June 2007. Association for
Computational Linguistics. URL http://
www.aclweb.org/anthology/P07-2045.
David Kurokawa, Cyril Goutte, and Pierre Is-
abelle. Automatic detection of translated text
and its impact on machine translation. In Pro-
ceedings of MT-Summit XII, 2009.
Franz Josef Och. Minimum error rate train-
ing in statistical machine translation. In ACL
?03: Proceedings of the 41st Annual Meeting
on Association for Computational Linguis-
tics, pages 160?167, Morristown, NJ, USA,
2003. Association for Computational Linguis-
tics. doi: http://dx.doi.org/10.3115/1075096.
1075117.
Kishore Papineni, Salim Roukos, Todd Ward,
and Wei-Jing Zhu. BLEU: a method for auto-
matic evaluation of machine translation. In
ACL ?02: Proceedings of the 40th Annual
Meeting on Association for Computational
Linguistics, pages 311?318, Morristown, NJ,
USA, 2002. Association for Computational
Linguistics. doi: http://dx.doi.org/10.3115/
1073083.1073135.
Anthony Pym and Grzegorz Chrupa la. The
quantitative analysis of translation flows in
the age of an international language. In Al-
bert Branchadell and Lovell M. West, editors,
Less Translated Languages, pages 27?38. John
Benjamins, Amsterdam, 2005.
Diana Santos. On grammatical translationese.
In In Koskenniemi, Kimmo (comp.), Short
papers presented at the Tenth Scandina-
vian Conference on Computational Linguis-
tics (Helsinki, pages 29?30, 1995.
Andreas Stolcke. SRILM?an extensible lan-
guage modeling toolkit. In Procedings of
International Conference on Spoken Lan-
guage Processing, pages 901?904, 2002. URL
citeseer.ist.psu.edu/stolcke02srilm.
html.
Gideon Toury. In Search of a Theory of Trans-
lation. The Porter Institute for Poetics and
Semiotics, Tel Aviv University, Tel Aviv,
1980.
Gideon Toury. Descriptive Translation Studies
and beyond. John Benjamins, Amsterdam /
Philadelphia, 1995.
Kristina Toutanova and Christopher D. Man-
ning. Enriching the knowledge sources used
in a maximum entropy part-of-speech tag-
ger. In Proceedings of the 2000 Joint SIGDAT
conference on Empirical methods in natural
language processing and very large corpora,
pages 63?70, Morristown, NJ, USA, 2000. As-
sociation for Computational Linguistics. doi:
http://dx.doi.org/10.3115/1117794.1117802.
Yulia Tsvetkov and Shuly Wintner. Automatic
acquisition of parallel corpora from websites
with dynamic content. In Proceedings of
the Seventh conference on International Lan-
guage Resources and Evaluation (LREC?10),
373
pages 3389?3392. European Language Re-
sources Association (ELRA), May 2010. ISBN
2-9517408-6-7.
Hans van Halteren. Source language markers in
EUROPARL translations. In COLING ?08:
Proceedings of the 22nd International Con-
ference on Computational Linguistics, pages
937?944, Morristown, NJ, USA, 2008. Asso-
ciation for Computational Linguistics. ISBN
978-1-905593-44-6.
374
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 255?265,
Avignon, France, April 23 - 27 2012. c?2012 Association for Computational Linguistics
Adapting Translation Models to Translationese Improves SMT
Gennadi Lembersky
Dept. of Computer Science
University of Haifa
31905 Haifa, Israel
glembers@campus.haifa.ac.il
Noam Ordan
Dept. of Computer Science
University of Haifa
31905 Haifa, Israel
noam.ordan@gmail.com
Shuly Wintner
Dept. of Computer Science
University of Haifa
31905 Haifa, Israel
shuly@cs.haifa.ac.il
Abstract
Translation models used for statistical ma-
chine translation are compiled from par-
allel corpora; such corpora are manually
translated, but the direction of translation is
usually unknown, and is consequently ig-
nored. However, much research in Trans-
lation Studies indicates that the direction of
translation matters, as translated language
(translationese) has many unique proper-
ties. Specifically, phrase tables constructed
from parallel corpora translated in the same
direction as the translation task perform
better than ones constructed from corpora
translated in the opposite direction.
We reconfirm that this is indeed the case,
but emphasize the importance of using also
texts translated in the ?wrong? direction.
We take advantage of information pertain-
ing to the direction of translation in con-
structing phrase tables, by adapting the
translation model to the special proper-
ties of translationese. We define entropy-
based measures that estimate the correspon-
dence of target-language phrases to transla-
tionese, thereby eliminating the need to an-
notate the parallel corpus with information
pertaining to the direction of translation.
We show that incorporating these measures
as features in the phrase tables of statisti-
cal machine translation systems results in
consistent, statistically significant improve-
ment in the quality of the translation.
1 Introduction
Much research in Translation Studies indicates
that translated texts have unique characteristics
that set them apart from original texts (Toury,
1980; Gellerstam, 1986; Toury, 1995). Known
as translationese, translated texts (in any lan-
guage) constitute a genre, or a dialect, of the
target language, which reflects both artifacts of
the translation process and traces of the origi-
nal language from which the texts were trans-
lated. Among the better-known properties of
translationese are simplification and explicitation
(Baker, 1993, 1995, 1996): translated texts tend
to be shorter, to have lower type/token ratio, and
to use certain discourse markers more frequently
than original texts. Incidentally, translated texts
are so markedly different from original ones that
automatic classification can identify them with
very high accuracy (van Halteren, 2008; Baroni
and Bernardini, 2006; Ilisei et al 2010; Koppel
and Ordan, 2011).
Contemporary Statistical Machine Translation
(SMT) systems use parallel corpora to train trans-
lation models that reflect source- and target-
language phrase correspondences. Typically,
SMT systems ignore the direction of translation
used to produce those corpora. Given the unique
properties of translationese, however, it is reason-
able to assume that this direction may affect the
quality of the translation. Recently, Kurokawa
et al(2009) showed that this is indeed the case.
They train a system to translate between French
and English (and vice versa) using a French-
translated-to-English parallel corpus, and then an
English-translated-to-French one. They find that
in translating into French the latter parallel cor-
pus yields better results, whereas for translating
into English it is better to use the former.
Usually, of course, the translation direction of a
parallel corpus is unknown. Therefore, Kurokawa
et al(2009) train an SVM-based classifier to pre-
dict which side of a bi-text is the origin and which
one is the translation, and only use the subset
of the corpus that corresponds to the translation
direction of the task in training their translation
model.
255
We use these results as our departure point,
but improve them in two major ways. First,
we demonstrate that the other subset of the cor-
pus, reflecting translation in the ?wrong? direc-
tion, is also important for the translation task, and
must not be ignored; second, we show that ex-
plicit information on the direction of translation of
the parallel corpus, whether manually-annotated
or machine-learned, is not mandatory. This is
achieved by casting the problem in the framework
of domain adaptation: we use domain-adaptation
techniques to direct the SMT system toward pro-
ducing output that better reflects the properties
of translationese. We show that SMT systems
adapted to translationese produce better transla-
tions than vanilla systems trained on exactly the
same resources. We confirm these findings using
an automatic evaluation metric, BLEU (Papineni
et al 2002), as well as through a qualitative anal-
ysis of the results.
Our departure point is the results of Kurokawa
et al(2009), which we successfully replicate in
Section 3. First (Section 4), we explain why trans-
lation quality improves when the parallel corpus
is translated in the ?right? direction. We do so
by showing that the subset of the corpus that was
translated in the direction of the translation task
(the ?right? direction, henceforth source-to-target,
or S ? T ) yields phrase tables that are better
suited for translation of the original language than
the subset translated in the reverse direction (the
?wrong? direction, henceforth target-to-source, or
T ? S). We use several statistical measures that
indicate the better quality of the phrase tables in
the former case.
Then (Section 5), we explore ways to build a
translation model that is adapted to the unique
properties of translationese. We first show that
using the entire parallel corpus, including texts
that are translated both in the ?right? and in the
?wrong? direction, improves the quality of the re-
sults. Furthermore, we show that the direction of
translation used for producing the parallel corpus
can be approximated by defining several entropy-
based measures that correlate well with transla-
tionese, and, consequently, with the quality of the
translation.
Specifically, we use the entire corpus, create a
single, unified phrase table and then use the statis-
tical measures mentioned above, and in particular
cross-entropy, as a clue for selecting phrase pairs
from this table. The benefit of this method is that
not only does it yield the best results, but it also
eliminates the need to directly predict the direc-
tion of translation of the parallel corpus. The main
contribution of this work, therefore, is a method-
ology that improves the quality of SMT by build-
ing translation models that are adapted to the na-
ture of translationese.
2 Related Work
Kurokawa et al(2009) are the first to address
the direction of translation in the context of SMT.
Their main finding is that using the S ? T por-
tion of the parallel corpus results in mucqqh better
translation quality than when the T ? S portion
is used for training the translation model. We in-
deed replicate these results here (Section 3), and
view them as a baseline. Additionally, we show
that the T ? S portion is also important for ma-
chine translation and thus should not be discarded.
Using information-theory measures, and in par-
ticular cross-entropy, we gain statistically signif-
icant improvements in translation quality beyond
the results of Kurokawa et al(2009). Further-
more, we eliminate the need to (manually or au-
tomatically) detect the direction of translation of
the parallel corpus.
Lembersky et al(2011) also investigate the re-
lations between translationese and machine trans-
lation. Focusing on the language model (LM),
they show that LMs trained on translated texts
yield better translation quality than LMs compiled
from original texts. They also show that perplex-
ity is a good discriminator between original and
translated texts.
Our current work is closely related to research
in domain-adaptation. In a typical domain adap-
tation scenario, a system is trained on a large cor-
pus of ?general? (out-of-domain) training mate-
rial, with a small portion of in-domain training
texts. In our case, the translation model is trained
on a large parallel corpus, of which some (gener-
ally unknown) subset is ?in-domain? (S ? T ),
and some other subset is ?out-of-domain? (T ?
S). Most existing adaptation methods focus on
selecting in-domain data from a general domain
corpus. In particular, perplexity is used to score
the sentences in the general-domain corpus ac-
cording to an in-domain language model. Gao
et al(2002) and Moore and Lewis (2010) apply
this method to language modeling, while Foster
256
et al(2010) and Axelrod et al(2011) use it on
the translation model. Moore and Lewis (2010)
suggest a slightly different approach, using cross-
entropy difference as a ranking function.
Domain adaptation methods are usually applied
at the corpus level, while we focus on an adap-
tation of the phrase table used for SMT. In this
sense, our work follows Foster et al(2010), who
weigh out-of-domain phrase pairs according to
their relevance to the target domain. They use
multiple features that help distinguish between
phrase pairs in the general domain and those in
the specific domain. We rely on features that are
motivated by the findings of Translation Studies,
having established their relevance through a com-
parative analysis of the phrase tables. In particu-
lar, we use measures such as translation model en-
tropy, inspired by Koehn et al(2009). Addition-
ally, we apply the method suggested by Moore
and Lewis (2010) using perplexity ratio instead
of cross-entropy difference.
3 Experimental Setup
The tasks we focus on are translation between
French and English, in both directions. We
use the Hansard corpus, containing transcripts of
the Canadian parliament from 1996?2007, as the
source of all parallel data. The Hansard is a
bilingual French?English corpus comprising ap-
proximately 80% English-original texts and 20%
French-original texts. Crucially, each sentence
pair in the corpus is annotated with the direction
of translation. Both English and French are lower-
cased and tokenized using MOSES (Koehn et al
2007). Sentences longer than 80 words are dis-
carded.
To address the effect of the corpus size, we
compile six subsets of different sizes (250K,
500K, 750K, 1M, 1.25M and 1.5M parallel
sentences) from each portion (English-original
and French-original) of the corpus. Addition-
ally, we use the devtest section of the Hansard
corpus to randomly select French-original and
English-original sentences that are used for tun-
ing (1,000 sentences each) and evaluation (5,000
sentences each). French-to-English MT sys-
tems are tuned and tested on French-original sen-
tences and English-to-French systems on English-
original ones.
To replicate the results of Kurokawa et al
(2009) and set up a baseline, we train twelve
French-to-English and twelve English-to-French
phrase-based (PB-) SMT systems using the
MOSES toolkit (Koehn et al 2007), each trained
on a different subset of the corpus. We use
GIZA++ (Och and Ney, 2000) with grow-diag-
final alignment, and extract phrases of length up
to 10 words. We prune the resulting phrase tables
as in Johnson et al(2007), using at most 30 trans-
lations per source phrase and discarding singleton
phrase pairs.
We construct English and French 5-gram lan-
guage models from the English and French
subsections of the Europarl-V6 corpus (Koehn,
2005), using interpolated modified Kneser-Ney
discounting (Chen, 1998) and no cut-off on all
n-grams. Europarl consists of a large number
of subsets translated from various languages, and
is therefore unlikely to be biased towards a spe-
cific source language. The reordering model used
in all MT systems is trained on the union of
the 1.5M French-original and the 1.5M English-
original subsets, using msd-bidirectional-fe re-
ordering. We use the MERT algorithm (Och,
2003) for tuning and BLEU (Papineni et al 2002)
as our evaluation metric. We test the statistical
significance of the differences between the results
using the bootstrap resampling method (Koehn,
2004).
A word on notation: We use ?English-original?
(EO) and ?French-original? (FO) to refer to the
subsets of the corpus that are translated from En-
glish to French and from French to English, re-
spectively. The translation tasks are English-to-
French (E2F) and French-to-English (F2E). We
thus use ?S ? T ? when the FO corpus is used for
the F2E task or when the EO corpus is used for
the E2F task; and ?T ? S? when the FO corpus
is used for the E2F task or when the EO corpus is
used for the F2E task.
Table 1 depicts the BLEU scores of the baseline
systems. The data are consistent with the findings
of Kurokawa et al(2009): systems trained on
S ? T parallel texts outperform systems trained
on T ? S texts, even when the latter are much
larger. The difference in BLEU score can be as
high as 3 points.
4 Analysis of the Phrase Tables
The baseline results suggest that S ? T and
T ? S phrase tables differ substantially, presum-
ably due to the different characteristics of original
257
Task: French-to-English
Corpus subset S ? T T ? S
250K 34.35 31.33
500K 35.21 32.38
750K 36.12 32.90
1M 35.73 33.07
1.25M 36.24 33.23
1.5M 36.43 33.73
Task: English-to-French
Corpus subset S ? T T ? S
250K 27.74 26.58
500K 29.15 27.19
750K 29.43 27.63
1M 29.94 27.88
1.25M 30.63 27.84
1.5M 29.89 27.83
Table 1: BLEU scores of baseline systems
and translated texts. In this section we explain
the better translation quality in terms of the bet-
ter quality of the respective phrase tables, as de-
fined by a number of statistical measures. We first
relate these measures to the unique properties of
translationese.
Translated texts tend to be simpler than original
ones along a number of criteria. Generally, trans-
lated texts are not as rich and variable as origi-
nal ones, and in particular, their type/token ratio
is lower. Consequently, we expect S ? T phrase
tables (which are based on a parallel corpus whose
source is original texts, and whose target is trans-
lationese) to have more unique source phrases and
a lower number of translations per source phrase.
A large number of unique source phrases suggests
better coverage of the source text, while a small
number of translations per source phrase means a
lower phrase table entropy. Entropy-based mea-
sures are well-established tools to assess the qual-
ity of a phrase table. Phrase table entropy captures
the amount of uncertainty involved in choosing
candidate translation phrases (Koehn et al 2009).
Given a source phrase s and a phrase table T
with translations t of s whose probabilities are
p(t|s), the entropy H of s is:
H(s) = ?
?
t?T
p(t|s)? log2p(t|s) (1)
There are two major flavors of the phrase table
entropy metric: Lambert et al(2011) calculate
the average entropy over all translation options
for each source phrase (henceforth, phrase table
entropy or PtEnt), whereas Koehn et al(2009)
search through all possible segmentations of the
source sentence to find the optimal covering set of
test sentences that minimizes the average entropy
of the source phrases in the covering set (hence-
forth, covering set entropy or CovEnt).
We also propose a metric that assesses the qual-
ity of the source side of a phrase table. The met-
ric finds the minimal covering set of a given text
in the source language using source phrases from
a particular phrase table, and outputs the average
length of a phrase in the covering set (henceforth,
covering set average length or CovLen).
Lembersky et al(2011) show that perplexity
distinguishes well between translated and origi-
nal texts. Moreover, perplexity reflects the de-
gree of ?relatedness? of a given phrase to original
language or to translationese. Motivated by this
observation, we design two cross-entropy-based
measures to assess how well each phrase table fits
the genre of translationese. Since MT systems are
evaluated against human translations, we believe
that this factor may have a significant impact on
translation performance. The cross-entropy of a
text T = w1, w2, ? ? ?wN according to a language
model L is:
H(T, L) = ?
1
N
N?
i=1
log2L(wi) (2)
We build language models of translated texts
as follows. For English translationese, we
extract 170,000 French-original sentences from
the English portion of Europarl, and 3,000
English-translated-from-French sentences from
the Hansard corpus (disjoint from the training,
development and test sets, of course). We use
each corpus to train a trigram language model
with interpolated modified Kneser-Ney discount-
ing and no cut-off. All out-of-vocabulary words
are mapped to a special token, ?unk?. Then,
we interpolate the Hansard and Europarl language
models to minimize the perplexity of the target
side of the development set (? = 0.58). For
French translationese, we use 270,000 sentences
from Europarl and 3,000 sentences from Hansard,
? = 0.81. Finally, we compute the cross-entropy
of each target phrase in the phrase tables accord-
ing to these language models.
258
As with the entropy-based measures, we define
two cross-entropy metrics: phrase table cross-
entropy or PtCrEnt calculates the average cross-
entropy over weighted cross-entropies of all trans-
lation options for each source phrase, and cover-
ing set cross-entropy or CovCrEnt finds the opti-
mal covering set of test sentences that minimizes
the weighted cross-entropy of the source phrase
in the covering set. Given a phrase table T and a
language model L, the weighted cross-entropyW
for a source phrase s is:
W (s, L) = ?
?
t?T
H(t, L)? p(t|s) (3)
where H(t, L) is the cross-entropy of t according
to a language model L.
Table 2 depicts various statistical measures
computed on the phrase tables corresponding to
our 24 SMT systems.1 The data meet our pre-
liminary expectations: S ? T phrase tables have
more unique source phrases, but fewer translation
options per source phrase. They have lower en-
tropy and cross-entropy, but higher covering set
length.
In order to asses the correspondence of each
measure to translation quality, we compute the
correlation of BLEU scores from Table 1 with
each of the measures specified in Table 2; we
compute the correlation coefficientR2 (the square
of Pearson?s product-moment correlation coeffi-
cient) by fitting a simple linear regression model.
Table 3 lists the results. Only the covering set
cross-entropy measure shows stability over the
French-to-English and English-to-French transla-
tion tasks, with R2 equals to 0.56 and 0.54, re-
spectively. Other measures are sensitive to the
translation task: covering set entropy has the
highest correlation with BLEU (R2 = 0.94) when
translating French-to-English, but it drops to 0.46
for the reverse task. The covering set average
length measure shows similar behavior: R2 drops
from 0.75 in French-to-English to 0.56 in English-
to-French. Still, the correlation of these measures
with BLEU is high.
Consequently, we use the three best measures,
namely covering set entropy, cross-entropy and
average length, as indicators of better transla-
tions, more similar to translationese. Crucially,
1The phrase tables were pruned, retaining only phrases
that are included in the evaluation set.
Measure R2 (FR?EN) R2 (EN-FR)
AvgTran 0.06 0.22
PtEnt 0.03 0.19
CovEnt 0.94 0.46
PtCrEnt 0.33 0.44
CovCrEnt 0.56 0.54
CovLen 0.75 0.56
Table 3: Correlation of BLEU scores with phrase table
statistical measures
these measures are computed directly on the
phrase table, and do not require reference trans-
lations or meta-information pertaining to the di-
rection of translation of the parallel phrase.
5 Translation Model Adaptation
We have thus established the fact that S ? T
phrase tables have an advantage over T ? S ones
that stems directly from the different characteris-
tics of original and translated texts. We have also
identified three statistical measures that explain
most of the variability in translation quality. We
now explore ways for taking advantage of the en-
tire parallel corpus, including translations in both
directions, in light of the above findings. Our goal
is to establish the best method to address the is-
sue of different translation direction components
in the parallel corpus.
First, we simply take the union of the two sub-
sets of the parallel corpus. We create three dif-
ferent mixtures of FO and EO: 500K sentences
each of FO and EO (?MIX1?), 500K sentences
of FO and 1M sentences of EO (?MIX2?), and
1M sentences of FO and 500K sentences of EO
(?MIX3?). We use these corpora to train French-
to-English and English-to-French MT systems,
evaluating their quality on the evaluation sets de-
scribed in Section 3. We use the same Moses con-
figuration as well as the same language and re-
ordering models as in Section 3.
Table 4 reports the results, comparing them
to the results obtained for the baseline MT sys-
tems trained on individual French-original and
English-original bi-texts (see Section 3).2 Note
that the mixed corpus includes many more sen-
tences than each of the baseline models; this is a
2Recall that when translating from French to English,
S ? T means that the bi-text is French-original; when trans-
lating from English to French, S ? T means it is English-
original.
259
Task: French-to-English
Set Total Source AvgTran PtEnt CovEnt PtCrEnt CovCrEnt CovLen
S ? T
250K 231K 69K 3.35 0.86 0.36 3.94 1.64 2.44
500K 360K 86K 4.21 0.98 0.35 3.52 1.30 2.64
750K 461K 96K 4.81 1.05 0.35 3.24 1.10 2.77
1M 544K 103K 5.27 1.10 0.34 3.09 0.99 2.85
1.25M 619K 109K 5.66 1.14 0.34 2.98 0.91 2.92
1.5M 684K 114K 6.01 1.18 0.33 2.90 0.85 2.97
T ? S
250K 199K 55K 3.65 0.92 0.45 4.00 1.87 2.25
500K 317K 69K 4.56 1.05 0.43 3.57 1.52 2.42
750K 405K 78K 5.19 1.12 0.43 3.39 1.35 2.53
1M 479K 85K 5.66 1.16 0.42 3.21 1.21 2.61
1.25M 545K 90K 6.07 1.20 0.41 3.11 1.12 2.67
1.5M 602K 94K 6.43 1.24 0.41 3.04 1.07 2.71
Task: English-to-French
Set Total Source AvgTran PtEnt CovEnt PtCrEnt CovCrEnt CovLen
S ? T
250K 224K 49K 4.52 1.07 0.63 3.48 1.88 2.08
500K 346K 61K 5.64 1.21 0.59 3.08 1.49 2.25
750K 437K 68K 6.39 1.29 0.57 2.91 1.33 2.33
1M 513K 74K 6.95 1.34 0.55 2.75 1.18 2.41
1.25M 579K 78K 7.42 1.38 0.54 2.63 1.09 2.46
1.5M 635K 81K 7.83 1.41 0.53 2.58 1.03 2.50
T ? S
250K 220K 46K 4.75 1.12 0.63 3.62 2.09 2.02
500K 334K 57K 5.82 1.24 0.60 3.24 1.70 2.16
750K 421K 64K 6.54 1.31 0.58 2.97 1.48 2.25
1M 489K 69K 7.10 1.36 0.57 2.84 1.35 2.32
1.25M 550K 73K 7.56 1.40 0.55 2.74 1.25 2.37
1.5M 603K 76K 7.92 1.43 0.55 2.66 1.17 2.41
Table 2: Statistic measures computed on the phrase tables: total size, in tokens (?Total?); the number of unique
source phrases (?Source?); the average number of translations per source phrase (?AvgTran?); phrase table entropy
(?PtEnt?) and covering set entropy (?CovEnt?); phrase table cross-entropy (?PtCrEnt?) and covering set cross-
entropy (?CovCrEnt?); and the covering set average length (?CovLen?)
realistic scenario, in which one can opt either to
use the entire parallel corpus, or only its S ? T
subset. Even with a corpus several times as large,
however, the ?mixed? MT systems perform only
slightly better than the S ? T ones. On one
hand, this means that one can train MT systems
on S ? T data only, at the expense of only a mi-
nor loss in quality. On the other hand, it is obvi-
ous that the T ? S component also contributes to
translation quality. We now look at ways to better
utilize this portion.
We compute the measures established in the
previous section on phrase tables trained on the
MIX corpora, and compare them with the same
measures computed for phrase tables trained on
the relevant S ? T corpus for both translation
tasks. Table 5 displays the figures for the MIX1
corpus: Phrase tables trained on mixed corpora
have higher covering set average length, similar
covering set entropy, but significantly worse cov-
ering set cross-entropy. Consequently, improving
covering set cross-entropy has the greatest poten-
tial for improving translation quality. We there-
fore use this feature to ?encourage? the decoder to
260
Task: French-to-English
System MIX1 MIX2 MIX3
Union 35.27 35.36 35.94
S ? T 35.21 35.21 35.73
T ? S 32.38 33.07 32.38
Task: English-to-French
System MIX1 MIX2 MIX3
Union 29.27 30.01 29.44
S ? T 29.15 29.94 29.15
T ? S 27.19 27.19 27.88
Table 4: Evaluation of the MIX systems
select translation options that are more related to
the genre of translated texts.
French-to-English
Measure MIX1 S ? T
CovLen 2.78 2.64
CovEnt 0.37 0.35
CovCrEnt 1.58 1.10
English-to-French
Measure MIX1 S ? T
CovLen 2.40 2.25
CovEnt 0.55 0.58
CovCrEnt 2.09 1.48
Table 5: Statistical measures computed for mixed vs.
source-to-target phrase tables
We do so by adding to each phrase pair in the
phrase tables an additional factor, as a measure of
its fitness to the genre of translationese. We ex-
periment with two such factors. First, we use the
language models described in Section 4 to com-
pute the cross-entropy of each translation option
according to this model. We add cross-entropy
as an additional score of a translation pair that
can be tuned by MERT (we refer to this system
as CrEnt). Since cross-entropy is ?the lower the
better? metric, we adjust the range of values used
by MERT for this score to be negative. Sec-
ond, following Moore and Lewis (2010), we de-
fine an adapting feature that not only measures
how close phrases are to translated language, but
also how far they are from original language, and
use it as a factor in a phrase table (this system
is referred to as PplRatio). We build two addi-
tional language models of original texts as fol-
lows. For original English, we extract 135,000
English-original sentences from the English por-
tion of Europarl, and 2,700 English-original sen-
tences from the Hansard corpus. We train a tri-
gram language model with interpolated modified
Kneser-Ney discounting on each corpus and we
interpolate both models to minimize the perplex-
ity of the source side of the development set for
the English-to-French translation task (? = 0.49).
For original French, we use 110,000 sentences
from Europarl and 2,900 sentences from Hansard,
? = 0.61. Finally, for each target phrase t in the
phrase table we compute the ratio of the perplex-
ity of t according to the original language model
Lo and the perplexity of twith respect to the trans-
lated modelLt (see Section 4). In other words, the
factor F is computed as follows:
F (t) =
H(t, Lo)
H(t, Lt)
(4)
We apply these techniques to the French-to-
English and English-to-French phrase tables built
from the mixed corpora and use each phrase ta-
ble to train an SMT system. Table 6 summa-
rizes the performance of these systems. All sys-
tems outperform the corresponding Union sys-
tems. ?CrEnt? systems show significant improve-
ments (p < 0.05) on balanced scenarios (?MIX1?)
and on scenarios biased towards the S ? T com-
ponent (?MIX2? in the French-to-English task,
?MIX3? in English-to-French). ?PplRatio? sys-
tems exhibit more consistent behavior, showing
small, but statistically significant improvement
(p < 0.05) in all scenarios.
Task: French-to-English
System MIX1 MIX2 MIX3
Union 35.27 35.36 35.94
CrEnt 35.54 35.45 36.75
PplRatio 35.59 35.78 36.22
Task: English-to-French
System MIX1 MIX2 MIX3
Union 29.27 30.01 29.44
CrEnt 29.47 30.44 29.45
PplRatio 29.65 30.34 29.62
Table 6: Evaluation of MT Systems
Note again that all systems in the same column
are trained on exactly the same corpus and have
exactly the same phrase tables. The only differ-
ence is an additional factor in the phrase table that
?encourages? the decoder to select translation op-
261
tions that are closer to translated texts than to orig-
inal ones.
6 Analysis
In order to study the effect of the adaptation qual-
itatively, rather than quantitatively, we focus on
several concrete examples. We compare transla-
tions produced by the ?Union? (henceforth base-
line) and by the ?PplRatio? (henceforth adapted)
French-English SMT systems. We manually in-
spect 200 sentences of length between 15 and 25
from the French-English evaluation set.
In many cases, the adapted system produces
more fluent and accurate translations. In the fol-
lowing examples, the baseline system generates
common translations of French words that are ad-
equate for a wider context, whereas the adapted
system chooses less common, but more suitable
translations:
Source J?ai eu cette perception et j?e?tais assez
certain que c?a allait se faire.
Baseline I had that perception and I was enough
certain it was going do.
Adapted I had that perception and I was quite
certain it was going do.
Source J?attends donc que vous en demandiez la
permission, monsieur le Pre?sident.
Baseline I look so that you seek permission, mr.
chairman.
Adapted I await, then, that you seek permission,
mr. chairman.
In quite a few cases, the baseline system leaves
out important words from the source sentence,
producing ungrammatical, even illegible transla-
tions, whereas the adapted system generates good
translations. Careful traceback reveals that the
baseline system ?splits? the source sentence into
phrases differently (and less optimally) than the
adapted system. Apparently, when the decoder is
coerced to select translation options that are more
adapted to translationese, it tends to select source
phrases that are more related to original texts, re-
sulting in more successful coverage of the source
sentence:
Source Pourtant, lorsqu? on les avait pre?sente?s,
c?e?tait pour corriger les proble`mes lie?s au
PCSRA.
Baseline Yet when they had presented, it was to
correct the problems the CAIS program.
Adapted Yet when they had presented, it was to
correct the problems associated with CAIS.
Source Cependant, je pense qu?il est pre?mature?
de le faire actuellement, e?tant donne? que le
ministre a lance? cette tourne?e.
Baseline However, I think it is premature to the
right now, since the minister launched this
tour.
Adapted However, I think it is premature to do
so now, given that the minister has launched
this tour.
Finally, there are often cultural differences be-
tween languages, specifically the use of a 24-hour
clock (common in French) vs. a 12-hour clock
(common in English). The adapted system is
more consistent in translating the former to the
latter:
Source On avait de?cide? de poursuivre la se?ance
jusqu? a` 18 heures, mais on n?aura pas le
temps de faire un autre tour de table.
Baseline We had decided to continue the meeting
until 18 hours, but we will not have the time
to do another round.
Adapted We had decided to continue the meeting
until 6 p.m., but we won?t have the time to do
another round.
Source Vu qu?il est 17h 20, je suis d?accord
pour qu?on ne discute pas de ma motion
imme?diatement.
Baseline Seen that it is 17h 20, I agree that we are
not talking about my motion immediately.
Adapted Given that it is 5:20, I agree that we are
not talking about my motion immediately.
In (human) translation circles, translating out of
one?s mother tongue is considered unprofessional,
even unethical (Beeby, 2009). Many professional
associations in Europe urge translators to work
exclusively into their mother tongue (Pavlovic?,
2007). The two kinds of automatic systems built
in this paper reflect only partly the human sit-
uation, but they do so in a crucial way. The
S ? T systems learn examples from many hu-
man translators who follow the decree according
to which translation should be made into one?s na-
tive tongue. The T ? S systems are flipped di-
rections of humans? input and output. The S ? T
direction proved to be more fluent, accurate and
even more culturally sensitive. This has to do with
fact that the translators ?cover? the source texts
more fully, having a better ?translation model?.
262
7 Conclusion
Phrase tables trained on parallel corpora that were
translated in the same direction as the translation
task perform better than ones trained on corpora
translated in the opposite direction. Nonethe-
less, even ?wrong? phrase tables contribute to the
translation quality. We analyze both ?correct? and
?wrong? phrase tables, uncovering a great deal of
difference between them. We use insights from
Translation Studies to explain these differences;
we then adapt the translation model to the nature
of translationese.
We incorporate information-theoretic measures
that correlate well with translationese into phrase
tables as an additional score that can be tuned
by MERT, and show a statistically significant im-
provement in the translation quality over all base-
line systems. We also analyze the results qual-
itatively, showing that SMT systems adapted to
translationese tend to produce more coherent and
fluent outputs than the baseline systems. An addi-
tional advantage of our approach is that it does not
require an annotation of the translation direction
of the parallel corpus. It is completely generic
and can be applied to any language pair, domain
or corpus.
This work can be extended in various direc-
tions. We plan to further explore the use of two
phrase tables, one for each direction-determined
subset of the parallel corpus. Specifically, we will
interpolate the translation models as in Foster and
Kuhn (2007), including a maximum a posteriori
combination (Bacchiani et al 2006). We also
plan to upweight the S ? T subset of the parallel
corpus and train a single phrase table on the con-
catenated corpus. Finally, we intend to extend this
work by combining the translation-model adap-
tation we present here with the language-model
adaptation suggested by Lembersky et al(2011)
in a unified system that is more tuned to generat-
ing translationese.
Acknowledgments
We are grateful to Cyril Goutte, George Foster
and Pierre Isabelle for providing us with an anno-
tated version of the Hansard corpus. This research
was supported by the Israel Science Foundation
(grant No. 137/06) and by a grant from the Israeli
Ministry of Science and Technology.
References
Amittai Axelrod, Xiaodong He, and Jianfeng Gao.
Domain adaptation via pseudo in-domain data se-
lection. In Proceedings of the 2011 Conference
on Empirical Methods in Natural Language Pro-
cessing, pages 355?362. Association for Computa-
tional Linguistics, July 2011. URL http://www.
aclweb.org/anthology/D11-1033.
Michiel Bacchiani, Michael Riley, Brian Roark, and
Richard Sproat. MAP adaptation of stochastic
grammars. Computer Speech and Language, 20:41?
68, January 2006. ISSN 0885-2308. doi: 10.1016/
j.csl.2004.12.001. URL http://dl.acm.org/
citation.cfm?id=1648820.1648854.
Mona Baker. Corpus linguistics and translation stud-
ies: Implications and applications. In Gill Fran-
cis Mona Baker and Elena Tognini-Bonelli, editors,
Text and technology: in honour of John Sinclair,
pages 233?252. John Benjamins, Amsterdam, 1993.
Mona Baker. Corpora in translation studies: An
overview and some suggestions for future research.
Target, 7(2):223?243, September 1995.
Mona Baker. Corpus-based translation studies:
The challenges that lie ahead. In Gill Francis
Mona Baker and Elena Tognini-Bonelli, editors,
Terminology, LSP and Translation. Studies in lan-
guage engineering in honour of Juan C. Sager,
pages 175?186. John Benjamins, Amsterdam, 1996.
Marco Baroni and Silvia Bernardini. A new
approach to the study of Translationese: Machine-
learning the difference between original and
translated text. Literary and Linguistic Com-
puting, 21(3):259?274, September 2006. URL
http://llc.oxfordjournals.org/cgi/
content/short/21/3/259?rss=1.
Alison Beeby. Direction of translation (directional-
ity). In Mona Baker and Gabriela Saldanha, edi-
tors, Routledge Encyclopedia of Translation Stud-
ies, pages 84?88. Routledge (Taylor and Francis),
New York, 2nd edition, 2009.
Stanley F. Chen. An empirical study of smoothing
techniques for language modeling. Technical report
10-98, Computer Science Group, Harvard Univer-
sity, November 1998.
George Foster and Roland Kuhn. Mixture-model adap-
tation for SMT. In Proceedings of the Second
Workshop on Statistical Machine Translation, pages
128?135. Association for Computational Linguis-
tics, June 2007. URL http://www.aclweb.
org/anthology/W/W07/W07-0717.
George Foster, Cyril Goutte, and Roland Kuhn. Dis-
criminative instance weighting for domain adap-
tation in statistical machine translation. In
263
Proceedings of the 2010 Conference on Em-
pirical Methods in Natural Language Process-
ing, pages 451?459, Stroudsburg, PA, USA,
2010. Association for Computational Linguis-
tics. URL http://dl.acm.org/citation.
cfm?id=1870658.1870702.
Jianfeng Gao, Joshua Goodman, Mingjing Li, and Kai-
Fu Lee. Toward a unified approach to statistical lan-
guage modeling for Chinese. ACM Transactions
on Asian Language Information Processing, 1:3?
33, March 2002. ISSN 1530-0226. doi: http://doi.
acm.org/10.1145/595576.595578. URL http://
doi.acm.org/10.1145/595576.595578.
Martin Gellerstam. Translationese in Swedish novels
translated from English. In Lars Wollin and Hans
Lindquist, editors, Translation Studies in Scandi-
navia, pages 88?95. CWK Gleerup, Lund, 1986.
Iustina Ilisei, Diana Inkpen, Gloria Corpas Pastor,
and Ruslan Mitkov. Identification of translationese:
A machine learning approach. In Alexander F.
Gelbukh, editor, Proceedings of CICLing-2010:
11th International Conference on Computational
Linguistics and Intelligent Text Processing, vol-
ume 6008 of Lecture Notes in Computer Science,
pages 503?511. Springer, 2010. ISBN 978-3-
642-12115-9. URL http://dx.doi.org/10.
1007/978-3-642-12116-6.
Howard Johnson, Joel Martin, George Foster, and
Roland Kuhn. Improving translation quality by dis-
carding most of the phrasetable. In Proceedings of
the Joint Conference on Empirical Methods in Nat-
ural Language Processing and Computational Nat-
ural Language Learning (EMNLP-CoNLL), pages
967?975. Association for Computational Linguis-
tics, June 2007. URL http://www.aclweb.
org/anthology/D/D07/D07-1103.
Philipp Koehn. Statistical significance tests for ma-
chine translation evaluation. In Proceedings of
EMNLP 2004, pages 388?395, Barcelona, Spain,
July 2004. Association for Computational Linguis-
tics.
Philipp Koehn. Europarl: A Parallel Corpus
for Statistical Machine Translation. In Confer-
ence Proceedings: the tenth Machine Translation
Summit, pages 79?86, Phuket, Thailand, 2005.
AAMT. URL http://mt-archive.info/
MTS-2005-Koehn.pdf.
Philipp Koehn, Hieu Hoang, Alexandra Birch,
Chris Callison-Burch, Marcello Federico, Nicola
Bertoldi, Brooke Cowan, Wade Shen, Christine
Moran, Richard Zens, Chris Dyer, Ondrej Bojar,
Alexandra Constantin, and Evan Herbst. Moses:
Open source toolkit for statistical machine transla-
tion. In Proceedings of the 45th Annual Meeting
of the Association for Computational Linguistics
Companion Volume Proceedings of the Demo and
Poster Sessions, pages 177?180, Prague, Czech Re-
public, June 2007. Association for Computational
Linguistics. URL http://www.aclweb.org/
anthology/P07-2045.
Philipp Koehn, Alexandra Birch, and Ralf Steinberger.
462 machine translation systems for Europe. In Ma-
chine Translation Summit XII, 2009.
Moshe Koppel and Noam Ordan. Translationese
and its dialects. In Proceedings of the 49th An-
nual Meeting of the Association for Computa-
tional Linguistics: Human Language Technolo-
gies, pages 1318?1326, Portland, Oregon, USA,
June 2011. Association for Computational Lin-
guistics. URL http://www.aclweb.org/
anthology/P11-1132.
David Kurokawa, Cyril Goutte, and Pierre Isabelle.
Automatic detection of translated text and its im-
pact on machine translation. In Proceedings of MT-
Summit XII, 2009.
Patrik Lambert, Holger Schwenk, Christophe Ser-
van, and Sadaf Abdul-Rauf. Investigations on
translation model adaptation using monolingual
data. In Proceedings of the Sixth Workshop
on Statistical Machine Translation, pages 284?
293. Association for Computational Linguistics,
July 2011. URL http://www.aclweb.org/
anthology/W11-2132.
Gennadi Lembersky, Noam Ordan, and Shuly Wint-
ner. Language models for machine translation:
Original vs. translated texts. In Proceedings of the
2011 Conference on Empirical Methods in Natural
Language Processing, pages 363?374, Edinburgh,
Scotland, UK, July 2011. Association for Computa-
tional Linguistics. URL http://www.aclweb.
org/anthology/D11-1034.
Robert C. Moore and William Lewis. Intelligent
selection of language model training data. In
Proceedings of the ACL 2010 Conference, Short
Papers, pages 220?224, Stroudsburg, PA, USA,
2010. Association for Computational Linguis-
tics. URL http://dl.acm.org/citation.
cfm?id=1858842.1858883.
Franz Josef Och. Minimum error rate training in sta-
tistical machine translation. In ACL ?03: Proceed-
ings of the 41st Annual Meeting on Association for
Computational Linguistics, pages 160?167, Morris-
town, NJ, USA, 2003. Association for Computa-
tional Linguistics. doi: http://dx.doi.org/10.3115/
1075096.1075117.
Franz Josef Och and Hermann Ney. Improved statisti-
cal alignment models. In ACL ?00: Proceedings of
the 38th Annual Meeting on Association for Com-
putational Linguistics, pages 440?447, Morristown,
264
NJ, USA, 2000. Association for Computational Lin-
guistics. doi: http://dx.doi.org/10.3115/1075218.
1075274.
Kishore Papineni, Salim Roukos, Todd Ward, and
Wei-Jing Zhu. BLEU: a method for automatic eval-
uation of machine translation. In ACL ?02: Proceed-
ings of the 40th Annual Meeting on Association for
Computational Linguistics, pages 311?318, Morris-
town, NJ, USA, 2002. Association for Computa-
tional Linguistics. doi: http://dx.doi.org/10.3115/
1073083.1073135.
Natas?a Pavlovic?. Directionality in translation and in-
terpreting practice. Report on a questionnaire sur-
vey in Croatia. Forum, 5(2):79?99, 2007.
Gideon Toury. In Search of a Theory of Translation.
The Porter Institute for Poetics and Semiotics, Tel
Aviv University, Tel Aviv, 1980.
Gideon Toury. Descriptive Translation Studies and be-
yond. John Benjamins, Amsterdam / Philadelphia,
1995.
Hans van Halteren. Source language markers in EU-
ROPARL translations. In COLING ?08: Proceed-
ings of the 22nd International Conference on Com-
putational Linguistics, pages 937?944, Morristown,
NJ, USA, 2008. Association for Computational Lin-
guistics. ISBN 978-1-905593-44-6.
265
Language Models for Machine Translation:
Original vs. Translated Texts
Gennadi Lembersky?
University of Haifa
Noam Ordan?
University of Haifa
Shuly Wintner?
University of Haifa
We investigate the differences between language models compiled from original target-language
texts and those compiled from texts manually translated to the target language. Corroborating
established observations of Translation Studies, we demonstrate that the latter are significantly
better predictors of translated sentences than the former, and hence fit the reference set better.
Furthermore, translated texts yield better language models for statistical machine translation
than original texts.
1. Introduction
Statistical machine translation (MT) uses large target languagemodels (LMs) to improve
the fluency of generated texts, and it is commonly assumed that for constructing lan-
guage models, ?more data is better data? (Brants and Xu 2009). Not all data, however,
are created the same. In this work we explore the differences between language models
compiled from texts originally written in the target language (O) and language models
compiled from translated texts (T).
This work is motivated by much research in Translation Studies that suggests
that original texts are significantly different from translated ones in various aspects
(Gellerstam 1986). Recently, corpus-based computational analysis corroborated this
observation, and Kurokawa, Goutte, and Isabelle (2009) apply it to statistical machine
translation, showing that for an English-to-French MT system, a translation model
trained on an English-translated-to-French parallel corpus is better than one trained
on French-translated-to-English texts. The main research question we investigate here
is whether a language model compiled from translated texts may similarly improve the
results of machine translation.
We test this hypothesis on several translation tasks, including translation from
several languages to English, and two additional tasks where the target language is
? Department of Computer Science, University of Haifa, 31905 Haifa, Israel.
E-mails: glembers@campus.haifa.ac.il, noam.ordan@gmail.com, shuly@cs.haifa.ac.il.
Submission received: 22 August 2011; revised submission received: 25 December 2011; accepted for
publication: 31 January 2012
? 2012 Association for Computational Linguistics
Computational Linguistics Volume 38, Number 4
not English. For each language pair we build two language models from two types of
corpora: texts originally written in the target language, and human translations from
the source language into the target language. We show that for each language pair, the
latter language model better fits a set of reference translations in terms of perplexity. We
also demonstrate that the differences between the two LMs are not biased by content,
but rather reflect differences on abstract linguistic features.
Research in Translation Studies holds a dual view on translationese, the sub-
language of translated texts. On the one hand, there is a claim for so-called transla-
tion universals, traits of translationese which occur in any translated text irrespective
of the source language. Others hold, on the other hand, that each source language
?spills over? to the target text, and therefore creates a sub-translationese, the result
of a pair-specific encounter between two specific languages. If both these claims are
true then language models based on translations from the source language should
best fit target language reference sentences, and language models based on transla-
tions from other source languages should fit reference sentences to a lesser extent yet
outperform originally written texts. To test this hypothesis, we compile additional
English LMs, this time using texts translated to English from languages other than
the source. Again, we use perplexity to assess the fit of these LMs to reference sets
of translated-to-English sentences. We show that these LMs depend on the source
language and differ from each other. Whereas they outperform O-based LMs, LMs
compiled from texts that were translated from the source language still fit the reference
set best.
Finally, we train phrase-based MT systems (Koehn, Och, and Marcu 2003) for each
language pair. We use four types of LMs: original; translated from the source language;
translated from other languages; and a mixture of translations from several languages.
We show that the translated-from-source-language LMs provide a significant improve-
ment in the quality of the translation output over all other LMs, and that the mixture
LMs always outperform the original LMs. This improvement persists even when the
original LMs are up to ten times larger than the translated ones. In other words, one
has to collect ten times more original material in order to reach the same quality as is
provided with translated material.
It is important to emphasize that translated texts abound: in fact, Pym and Chrupa?a
(2005) show (quantitatively!) that the rate of translations into a language is inversely
proportional to the number of books published in that language: So whereas in English
only around 2% of texts published are translations, in languages such as Albanian,
Arabic, Danish, Finnish, or Hebrew translated texts constitute between 20% and
25% of the total publications. Furthermore, such data can be automatically identified
(see Section 2). The practical impact of our work on MT is therefore potentially
dramatic.
The main contributions of this work are thus a computational corroboration of the
following hypotheses:
1. Original and translated texts exhibit significant, measurable differences.
2. LMs compiled from translated texts better fit translated references than
LMs compiled from original texts of the same (and much larger) size (and,
to a lesser extent, LMs compiled from texts translated from languages
other than the source language).
3. MT systems that use LMs based on manually translated texts significantly
outperform LMs based on originally written texts.
800
Lembersky, Ordan, and Wintner Language Models for Machine Translation
This article1 is organized as follows: Section 2 provides background and describes
related work. We explain our experimental set-up, research methodology and resources
in Section 3 and detail our experiments and results in Section 4. Section 5 discusses the
results and their implications, and suggests directions for future research.
2. Background and Related Work
Numerous studies suggest that translated texts are different from original ones.
Gellerstam (1986) compares texts written originally in Swedish and texts translated
from English into Swedish. He notes that the differences between them do not indicate
poor translation but rather a statistical phenomenon, which he terms translationese. He
focuses mainly on lexical differences, for example, less colloquialism in the translations,
or foreign words used in the translations ?with new shades of meaning taken from
the English lexeme? (page 91). Only later studies consider grammatical differences
(see, e.g., Santos 1995). The features of translationese were theoretically organized
under the terms laws of translation and translation universals.
Toury (1980, 1995) distinguishes between two laws: the law of interference and the
law of growing standardization. The law of interference pertains to the fingerprints
of the source text that are left in the translation product. The law of standardization
pertains to the effort to standardize the translation product according to existing norms
in the target language (and culture). Interestingly, these two laws are in fact reflected in
the architecture of statistical machine translation: Interference in the translation model
and standardization in the language model.
The combined effect of these laws creates a hybrid text that partly corresponds to
the source text and partly to texts written originally in the target language, but in fact
belongs to neither (Frawley 1984). Baker (1993, 1995, 1996) suggests several candidates
for translation universals, which are claimed to appear in any translated text, regardless
of the source language. These include simplification, the tendency of translated texts to
simplify the language, the message or both; and explicitation, their tendency to spell
out implicit utterances that occur in the source text.
During the 1990s, corpora were used extensively to study translationese. For exam-
ple, Al-Shabab (1996) shows that translated texts exhibit lower lexical variety (type-to-
token ratio) and Laviosa (1998) shows that their mean sentence length is lower, as is
their lexical density (ratio of content to non-content words). These studies, although not
conclusive, provide some evidence for the simplification hypothesis.
Baroni and Bernardini (2006) use machine learning techniques to distinguish be-
tween original and translated Italian texts, reporting 86.7% accuracy. They manage to
abstract from content and perform the task using onlymorpho-syntactic cues. Ilisei et al
(2010) perform the same task for Spanish but enhance it theoretically in order to check
the simplification hypothesis. They first use a set of features which seem to capture
?general? characteristics of the text (ratio of grammatical words to content words); they
then add another set of features, each of which relates to the simplification hypothesis.
Finally, they remove each ?simplification feature? in turn and evaluate its contribution
to the classification task. The most informative features are lexical variety, sentence
length, and lexical density.
1 Preliminary results were published in Lembersky, Ordan, and Wintner (2011). This is an extended,
revised version of that paper, providing fuller data and reporting on more language pairs. Some
experiments (in particular, Section 4.2.3) are completely new, as is the bulk of the discussion in
Section 5, including the human evaluation.
801
Computational Linguistics Volume 38, Number 4
van Halteren (2008) focuses on six languages from Europarl (Koehn 2005): Dutch,
English, French, German, Italian, and Spanish. For each of these languages, a parallel
six-lingual subcorpus is extracted, including an original text and its translations into the
other five languages. The task is to identify the source language of translated texts, and
the reported results are excellent. This finding is crucial: as Baker (1996) states, transla-
tions do resemble each other; in accordance with the law of interference, however, the
study of van Halteren (2008) suggests that translation from different source languages
constitute different sublanguages. As we show in Section 4.2, LMs based on translations
from the source language outperform LMs compiled from non-source translations, in
terms of both fitness to the reference set and improving MT.
Kurokawa, Goutte, and Isabelle (2009) show that the direction of translation affects
the performance of statistical MT. They train systems to translate between French and
English (and vice versa) using a French-translated-to-English parallel corpus, and then
an English-translated-to-French one. They find that in translating into French it is
better to use the latter parallel corpus, and when translating into English it is better
to use the former. Whereas they address the translationmodel, we focus on the language
model in this work. We show that using a language model trained on a text translated
from the source language of the MT system does indeed improve the results of the
translation.
3. Methodology and Resources
3.1 Hypotheses
We investigate the following three hypotheses:
1. Translated texts differ from original texts.
2. Texts translated from one language differ from texts translated from other
languages.
3. LMs compiled from manually translated texts are better for MT than LMs
compiled from original texts.
We test our hypotheses by considering translations from several languages to
English, and from English to German and French. For each language pair we create a
reference set comprising several thousands of sentences written originally in the source
language and manually translated to the target language. Section 3.4 provides details
on the reference sets.
To investigate the first hypothesis, we train two LMs for each language pair, one
created from texts originally written in the language (O-based) and the other from texts
translated into the target language (T-based). Then, we check which LM better fits the
reference set.
Fitness of a language model to a set of sentences is measured in terms of perplexity
(Jelinek et al 1977; Bahl, Jelinek, and Mercer 1983). Given a language model and a test
(reference) set, perplexity measures the predictive power of the language model over
the test set, by looking at the average probability the model assigns to the test data.
Intuitively, a better model assigns higher probability to the test data, and consequently
has a lower perplexity; it is less surprised by the test data. Formally, the perplexity PP of
802
Lembersky, Ordan, and Wintner Language Models for Machine Translation
a language model L on a test setW = w1 w2 . . .wN is the probability ofW normalized by
the number of words N (Jurafsky and Martin 2008, page 96):
PP(L,W) = N
?
?
?
?
N
?
i=1
1
PL(wi|w1 . . .wi?1)
(1)
For the second hypothesis, we extend the experiment to LMs created from texts
translated from other languages. For example, we test how well an LM trained on
French-translated-to-English texts fits the German-translated-to-English reference set;
and how well an LM trained on German-translated-to-English texts fits the French-
translated-to-English reference set.
Finally, for the third hypothesis, we use these LMs for statistical MT (SMT). For
each language pair we build several SMT systems. All systems use a translation model
extracted from a parallel corpus which is oblivious to the direction of the translation;
and one of the above-mentioned LMs. Then, we compare the translation quality of these
systems in terms of the Bleu metric (Papineni et al 2002) (as we show in Section 5.1,
other automatic evaluation metrics reveal the same pattern).
3.2 Language Models
In all the experiments, we use SRILM (Stolcke 2002) with interpolated modified Kneser-
Ney discounting (Chen 1998) and no cut-off on all n-grams, to train n-gram language
models from various corpora. Unless mentioned otherwise, n = 4. We limit language
models to a fixed vocabulary and map out-of-vocabulary (OOV) tokens to a unique
symbol to better control the OOV rates among various corpora. We experimented with
two techniques for setting the vocabulary: Use all words that occur more than once
in the evaluation set (see Section 3.4); and use the intersection of all words occurring
in all corpora used to train the language model. Both techniques produce very similar
results, and for brevity we only report the results achieved with the former technique.
In addition, we tried various discounting schemes (e.g., Good-Turing smoothing [Chen
1998]), and also ran experiments with an open vocabulary. The results of all these
experiments are consistent with our findings, and therefore we do not elaborate on
them here.
Our main corpus is Europarl (Koehn 2005), specifically, portions collected over
the years 1996?1999 and 2001?2009. This is a large multilingual corpus, containing
sentences translated from several European languages. It is organized as a collection
of bilingual corpora rather than as a single multilingual one, however, and it is hard to
identify sentences that are translated into several languages.
We therefore treat each bilingual subcorpus in isolation; each such subcorpus con-
tains sentences translated to English from various languages. We rely on the language
attribute of the speaker tag to identify the source language of sentences in the English
part of the corpus. Because this tag is rarely used with English-language speakers,
we also exploit the ID attribute of the speaker tag, which we match against the list
of British members of the European parliament.2
2 We wrote a small script that determines the original language of Europarl utterances in this way. The
script is publicly available.
803
Computational Linguistics Volume 38, Number 4
Table 1
Europarl English-target corpus statistics, translation from Lang. to English.
German?English
Lang. Sentences Tokens Length
MIX 82,700 2,325,261 28.1
O-EN 91,100 2,324,745 25.5
T-DE 87,900 2,322,973 26.4
T-FR 77,550 2,325,183 30.0
T-IT 65,199 2,325,996 35.7
T-NL 94,000 2,323,646 24.7
French?English
Lang. Sentences Tokens Length
MIX 90,700 2,546,274 28.1
O-EN 99,300 2,545,891 25.6
T-DE 94,900 2,546,124 26.8
T-FR 85,750 2,546,085 29.7
T-IT 72,008 2,546,984 35.4
T-NL 103,350 2,545,645 24.6
Italian?English
Lang. Sentences Tokens Length
MIX 87,040 2,534,793 29.1
O-EN 93,520 2,534,892 27.1
T-DE 90,550 2,534,867 28.0
T-FR 82,930 2,534,930 30.6
T-IT 69,270 2,535,225 36.6
T-NL 96,850 2,535,053 26.2
Dutch?English
Lang. Sentences Tokens Length
MIX 90,500 2,508,265 27.7
O-EN 97,000 2,475,652 25.5
T-DE 94,200 2,503,354 26.6
T-FR 86,600 2,523,055 29.1
T-IT 73,541 2,518,196 34.2
T-NL 101,950 2,513,769 24.7
We focus on the following languages: German (DE), French (FR), Italian (IT), and
Dutch (NL). For each of these languages, L, we consider the L-English Europarl subcor-
pus. In each subcorpus, we extract chunks of approximately 2.5 million English tokens
translated from each of these source languages (T-DE, T-FR, T-IT, and T-NL), as well
as sentences written originally in English (O-EN). The mixture corpus (MIX), which
is designed to represent ?general? translated language, is constructed by randomly
selecting sentences translated from any language (excluding original sentences). For
English-to-German and English-to-French, we use the German?English and French?
English Europarl sub-corpora. We extract German (and French) sentences translated
from English, French (or German), Italian, and Dutch, as well as sentences originally
written in German (or French).
Table 1 lists the number of sentences, number of tokens, and average sentence
length for each English subcorpus and each original language. Table 2 lists the statistics
for German and French corpora.
Table 2
Europarl corpus statistics, translation from Lang. to German and French.
English?German
Lang. Sentences Tokens Length
MIX 81,447 2,215,044 27.2
O-DE 89,739 2,215,036 24.7
T-EN 88,081 2,215,040 25.2
T-FR 77,555 2,215,021 28.6
T-IT 64,374 2,215,030 34.4
T-NL 94,289 2,215,033 23.5
English?French
Lang. Sentences Tokens Length
MIX 89,660 2,845,071 31.7
O-FR 89,875 2,844,265 31.6
T-EN 96,057 2,847,238 29.6
T-DE 93,468 2,843,730 30.4
T-IT 73,257 2,848,931 38.9
T-NL 102,498 2,835,006 27.7
804
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Table 3
Hansard corpus statistics.
Original French
Size Sentences Tokens Length
1M 54,851 1,000,076 18.2
5M 276,187 5,009,157 18.1
10M 551,867 10,001,716 18.1
Original English
Size Sentences Tokens Length
1M 54,216 1,006,275 18.6
5M 268,806 5,006,482 18.6
10M 537,574 10,004,191 18.6
25M 1,344,580 25,001,555 18.6
50M 2,689,332 50,009,861 18.6
100M 5,376,886 100,016,704 18.6
In another set of experiments we address the size of language models, to assess
how much more original material is needed compared with translated material (Sec-
tion 4.2.2). Because Europarl does not have enough trainingmaterial for this task, we use
the Hansard corpus, containing transcripts of the Canadian parliament from 1996?2007.
This is a bilingual French?English corpus comprising about 80% original English texts
(EO) and about 20% texts translated from French (FO). We first separate original English
texts from texts translated from French and then, for each subcorpus, we randomly
extract portions of texts of different sizes: 1M, 5M, and 10M tokens from the FO corpus
and 1M, 5M, 10M, 25M, 50M, and 100M tokens from the EO corpus; see Table 3. For even
larger amounts of data, we use the English Gigaword corpus (Graff and Cieri 2007),
fromwhich we randomly extract portions of up to 1G tokens; see Table 4. Unfortunately,
we do not know how much of this corpus is original; because it includes data from the
Xinhua news agency, we suspect that parts of it are indeed translated.
To experiment with a non-European language (and a different genre) we choose
Hebrew (HE). We use two English corpora: The original (O-EN) corpus comprises
articles from the International Herald Tribune, downloaded over a period of sevenmonths
(from January to July 2009). The articles cover four topics: news (53.4%), business
(20.9%), opinion (17.6%), and arts (8.1%). The translated (T-HE) corpus consists of
articles collected from the Israeli newspaper HaAretz over the same period of time.
HaAretz is published in Hebrew, but portions of it are translated to English. The
O-corpus was downsized in order for both subcorpora to have approximately the same
number of tokens in each topic. Table 5 lists basic statistics for this corpus.
3.3 SMT Training Data
To focus on the effect of the language model on translation quality, we design SMT
training corpora to be oblivious to the direction of translation. Again, we use Europarl
Table 4
Gigaword corpus statistics.
English, various sources
Size Sentences Tokens Length
100M 4,448,260 107,483,194 24.2
500M 20,797,060 502,380,054 24.2
1,000M 41,517,095 1,002,919,581 24.2
805
Computational Linguistics Volume 38, Number 4
Table 5
Hebrew-to-English corpus statistics.
Hebrew?English
Orig. Lang. Sentences Tokens Length
O-EN 135,228 3,561,559 26.3
T-HE 147,227 3,561,556 24.2
(January 2000 to September 2000) as the main source of our parallel corpora. We also use
the Hansard corpus: We randomly extract 50,000 sentences from the French-translated-
to-English subcorpus and another 50,000 sentences from the original English sub-
corpus. For Hebrew we use the Hebrew?English parallel corpus (Tsvetkov andWintner
2010) that contains sentences translated fromHebrew to English (54%) and from English
to Hebrew (46%). The English-to-Hebrew part comprises many short sentences (ap-
proximately six tokens per sentence) taken from amovie subtitle database. This explains
the low average sentence length of this particular corpus. Table 6 lists some details on
those corpora.
3.4 Reference Sets
The reference sets have two uses. First, they are used as the test sets in the experiments
that measure the perplexity of the language models. Second, in the MT experiments we
use them to randomly extract 1,000 sentences for tuning and 1,000 (different) sentences
for evaluation. All references are of course disjoint from the LM and training materials.
For each language L we use the L-English subcorpus of Europarl (over the period
of October 2000 to December 2000). For L-to-English translation tasks we only use
sentences originally produced in L, and for English-to-L tasks we use sentences orig-
inally written in English. The Hansard reference set comprises only French-translated-
to-English sentences. The Hebrew-to-English reference set is an independent (disjoint)
part of the Hebrew-to-English parallel corpus. This set mostly comprises literary data
(88.6%) and a small portion of news (11.4%). All sentences are originally written in
Hebrew and are manually translated to English. See Table 7 for the figures.
Table 6
Parallel corpora used for SMT training.
Language pair Side Sentences Tokens Length
DE-EN
DE 92,901 2,439,370 26.3
EN 92,901 2,602,376 28.0
FR-EN
FR 93,162 2,610,551 28.0
EN 93,162 2,869,328 30.8
IT-EN
IT 85,485 2,531,925 29.6
EN 85,485 2,517,128 29.5
NL-EN
NL 84,811 2,327,601 27.4
EN 84,811 2,303,846 27.2
Hansard
FR 100,000 2,167,546 21.7
EN 100,000 1,844,415 18.4
HE-EN
HE 95,912 726,512 7.6
EN 95,912 856,830 8.9
806
Lembersky, Ordan, and Wintner Language Models for Machine Translation
4. Experiments and Results
We detail in this section the experiments performed to test the three hypotheses: that
translated texts can be distinguished from original ones, and provide better language
models for other translated texts; that texts translated from other languages than the
source are still better predictors of translations than original texts (Section 4.1); and that
these differences are important for SMT (Section 4.2).
4.1 Translated vs. Original texts
4.1.1 Adequacy of O-based and T-based LMs.We begin with English as the target language.
We train 1-, 2-, 3-, and 4-gram language models for each Europarl subcorpus, based
on the corpora described in Section 3.2. For each language L, we compile a LM from
texts translated (into English) from L; from texts translated from languages other than
L (including a mixture of such languages, MIX); and from texts originally written in
English. The LMs are applied to the reference set of texts translated from L, and we
compute the perplexity: the fitness of the LM to the reference set. Table 8 details the
results. The lowest perplexity (reflecting the best fit) in each subcorpus is typeset in
boldface, and the highest (worst fit) is italicized.
These results overwhelmingly support our hypothesis. For each language L, the
perplexity of the language model that was created from L translations is lowest, fol-
lowed immediately by the MIX LM. Furthermore, the perplexity of the LM created
from originally-English texts is highest in all experiments (except the Dutch-to-English
translation task, where the perplexity of the 2-gram LM created from texts translated
from Italian is slightly higher). The perplexity of LMs constructed from texts translated
from languages other than L always lies between these two extremes: It is a better fit
of the reference set than original texts, but not as good as texts translated from L (or
mixture translations). This gives rise to yet another hypothesis, namely, that translations
from typologically related languages form a similar ?translationese dialect,? whereas
translations from more distant source languages form two different ?dialects? in the
target language (see Koppel and Ordan 2011).
Table 7
Reference sets.
Language pair Side Sentences Tokens Length
DE-EN
DE 6,675 161,889 24.3
EN 6,675 178,984 26.8
FR-EN
FR 8,494 260,198 30.6
EN 8,494 271,536 32.0
IT-EN
IT 2,269 82,261 36.3
EN 2,269 78,258 34.5
NL-EN
NL 4,593 114,272 24.9
EN 4,593 105,083 22.9
EN-DE
EN 8,358 215,325 25.8
DE 8,358 214,306 25.6
EN-FR
EN 4,284 108,428 25.3
FR 4,284 125,590 29.3
Hansard
FR 8,926 193,840 21.7
EN 8,926 163,448 18.3
HE-EN
HE 7,546 102,085 13.5
EN 7,546 126,183 16.7
807
Computational Linguistics Volume 38, Number 4
Table 8
Fitness of various LMs to the reference set.
German to English translations
Orig. Lang. 1-gram PPL 2-gram PPL 3-gram PPL 4-gram PPL
Mix 451.50 93.00 69.36 66.47
O-EN 468.09 103.74 79.57 76.79
T-DE 443.14 88.48 64.99 62.07
T-FR 460.98 99.90 76.23 73.38
T-IT 465.89 102.31 78.50 75.67
T-NL 457.02 97.34 73.54 70.56
French to English translations
Orig. Lang. 1-gram PPL 2-gram PPL 3-gram PPL 4-gram PPL
Mix 472.05 99.04 75.60 72.68
O-EN 500.56 115.48 91.14 88.31
T-DE 486.78 108.50 84.39 81.41
T-FR 463.58 94.59 71.24 68.37
T-IT 476.05 102.69 79.23 76.36
T-NL 490.09 110.67 86.61 83.55
Italian to English translations
Orig. Lang. 1-gram PPL 2-gram PPL 3-gram PPL 4-gram PPL
Mix 395.99 88.46 67.35 64.40
O-EN 415.47 99.92 79.27 76.34
T-DE 404.64 95.22 73.73 70.85
T-FR 395.99 89.44 68.38 65.54
T-IT 384.55 81.90 60.85 57.91
T-NL 411.58 98.78 76.98 73.94
Dutch to English translations
Orig. Lang. 1-gram PPL 2-gram PPL 3-gram PPL 4-gram PPL
Mix 434.89 90.73 69.05 66.08
O-EN 448.11 100.17 78.23 75.46
T-DE 437.68 93.67 71.54 68.57
T-FR 445.00 97.32 75.59 72.55
T-IT 448.11 100.19 78.06 75.19
T-NL 423.13 83.99 62.17 59.09
Boldface = best fit; italics = worst fit.
4.1.2 Linguistic Abstraction. A possible explanation for the different perplexity results
among the LMs could be the specific content of the corpora used to compile the LMs.
For example, onewould expect texts translated fromDutch to exhibit higher frequencies
of words such asAmsterdam or even canal. This, indeed, is reflected by the lower (usually
lowest) number of OOV items in language models compiled from texts translated from
the source language.
As a specific example, the top five words that occur in the T-FR corpus and the
evaluation set, but are absent from the O-EN corpus, are: biarritz, meat-and-bone,
808
Lembersky, Ordan, and Wintner Language Models for Machine Translation
armenian, ievoli, and ivorian. The top five words that occur in the O-EN corpus, but
are absent from the T-FR corpus, are: duhamel, paciotti, ivoirian, coke, and spds. Of
those, biarritz seems to be French-specific, but the other items seem more arbitrary.
To rule out the possibility that the perplexity results are due to specific content
phenomena, and to further emphasize that the corpora are indeed structurally different,
we conduct more experiments, in which we gradually abstract away from the domain-
and content-specific features of the texts and emphasize their syntactic structure. We
focus on French-to-English, but the results are robust and consistent (we repeated the
same experiments for all language pairs, with very similar outcomes).
First, we remove all punctuation to eliminate possible bias due to differences in
punctuation conventions.3 Then, we use the Stanford Named Entity Recognizer (Finkel,
Grenager, and Manning 2005) to identify named entities, which we replace with a
unique token (?NE?). Next, we replace all nouns with their part-of-speech (POS) tag;
we use the Stanford POS Tagger (Toutanova and Manning 2000). Finally, for full lexical
abstraction, we replace all words with their POS tags, retaining only abstract syntactic
structures devoid of lexical content.
At each step, we train six language models on O- and T-texts and apply them
to the reference set (which is adapted to the same level of abstraction, of course).
As the abstraction of the text increases, we also increase the order of the LMs: From
4-grams for text without punctuation and NE abstraction, to 5-grams for noun abstrac-
tion, to 8-grams for full POS abstraction. In all cases we fix the LM vocabulary to only
contain tokens that appear more than once in the ?abstracted? reference set. The results,
depicted in Table 9, consistently show that the T-based LM is a better fit to the reference
set, albeit to a lesser extent. The rightmost column specifies the improvement, in terms
of perplexity, of each language model, compared with the worst-performing model.
Although we do not show the details here, the same pattern is persistent in all the other
Europarl languages we experiment with.
4.1.3 More Language Pairs. To further test the robustness of these phenomena, we repeat
these experiments with the Hebrew-to-English corpus and reference set, reflecting a
different language family, a smaller corpus, and a different domain. We train two
4-gram language models on the O-EN and T-HE corpora. We then apply the two LMs
to the reference set and compute the perplexity. The results are presented in Table 10.
Again, the T-based LM is a better fit to the translated text than the O-based LM: Its
perplexity is lower by 12.8%. We also repeat the abstraction experiments on the Hebrew
scenario. The results, depicted in Table 11, consistently show that the T-based LM is a
better fit to the reference set.
Clearly, then, translated LMs better fit the references than original ones, and the
differences can be traced back not just to (trivial) specific lexical choice, but also to
syntactic structure, as evidenced by the POS abstraction experiments.
We further test our findings on other target languages, specifically English?German
and English?French. We train several 4-gram language models on the corpora specified
in Table 2. We then compute the perplexity of the German-translated-from-English and
French-translated-from-English reference sets (see Section 3.4) with respect to these
language models. Table 12 depicts the results; they are in complete agreement with our
hypothesis.
3 In fact, there is reason to assume that punctuation constitutes part of the translationese effect. Removing
punctuation therefore harms our cause of identifying this effect.
809
Computational Linguistics Volume 38, Number 4
Table 9
Fitness of O- vs. T-based LMs to the reference set (FR-EN), reflecting different abstraction levels.
No Punctuation
Orig. Lang. Perplexity Improvement (%)
MIX 105.91 19.73
O-EN 131.94
T-DE 122.50 7.16
T-FR 99.52 24.58
T-IT 112.71 14.58
T-NL 126.44 4.17
NE Abstraction
Orig. Lang. Perplexity Improvement (%)
MIX 93.88 18.51
O-EN 115.20
T-DE 107.48 6.70
T-FR 88.96 22.77
T-IT 99.17 13.91
T-NL 110.72 3.89
Noun Abstraction
Orig. Lang. Perplexity Improvement (%)
MIX 36.02 11.34
O-EN 40.62
T-DE 38.67 4.81
T-FR 34.75 14.46
T-IT 36.85 9.30
T-NL 39.44 2.91
POS Abstraction
Orig. Lang. Perplexity Improvement (%)
MIX 7.99 2.66
O-EN 8.20
T-DE 8.08 1.47
T-FR 7.89 3.77
T-IT 8.00 2.47
T-NL 8.11 1.11
Boldface = best fit; italics = worst fit.
4.1.4 Larger Language Models. Can these phenomena be attributed to the relatively small
size of the corpora we use?Will the perplexity of O texts converge to that of T texts when
more data become available, or will the differences persist? To address these questions,
we use the (much larger) Hansard corpus and the (even larger) Gigaword corpus. We
train 4-gram language models for each Hansard and Gigaword subcorpus described in
Section 3.2. We apply the LMs to the Hansard reference set, but also to the Europarl
reference set, to examine the effect on out-of-domain (but similar genre) texts. In both
cases we report perplexity (Table 13).
810
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Table 10
Fitness of O- vs. T-based LMs to the reference set (HE-EN).
Hebrew to English translations
Orig. Lang. Perplexity Improvement (%)
O-EN 187.26
T-HE 163.23 12.83
The results are fully consistent with our previous findings: In the case of the
Hansard reference set, a language model based on original texts must be up to ten
times larger to retain the low perplexity level of translated texts. For example, whereas a
languagemodel compiled from 10million English-translated-from-French tokens yields
a perplexity of 42.70 on the Hansard reference set, a LM compiled from original English
texts requires 100 million words to yield a similar perplexity of 43.70 on the same
reference set. The Gigaword LMs, which are trained on texts representing completely
different domains and genres, produce much higher (i.e., worse) perplexity in this
scenario. In the case of the Europarl reference set, a language model based on original
texts must be approximately five times larger (and a Gigaword language model approxi-
mately twenty times larger) than a language model based on original texts to yield similar
perplexity.
Table 11
Fitness of O- vs. T-based LMs to the reference set (HE-EN), reflecting different abstraction levels.
No Punctuation
Orig. Lang. Perplexity Improvement (%)
O-EN 401.44
T-HE 335.30 16.48
NE Abstraction
Orig. Lang. Perplexity Improvement (%)
O-EN 298.16
T-HE 251.39 15.69
Noun Abstraction
Orig. Lang. Perplexity Improvement (%)
O-EN 81.92
T-HE 72.34 11.70
POS Abstraction
Orig. Lang. Perplexity Improvement (%)
O-EN 11.47
T-HE 10.76 6.20
811
Computational Linguistics Volume 38, Number 4
4.2 Original vs. Translated LMs for Machine Translation
4.2.1 SMT Experiments. The last hypothesis we test is whether a better fitting lan-
guage model yields a better machine translation system. In other words, we expect
the T-based LMs to outperform the O-based LMs when used as part of machine
translation systems. We construct German-to-English, English-to-German, French-to-
English, French-to-German, Italian-to-English, and Dutch-to-English MT systems using
the Moses phrase-based SMT toolkit (Koehn et al 2007). The systems are trained on
the parallel corpora described in Section 3.3. We use the reference sets (Section 3.4) as
follows: 1,000 sentences are randomly extracted for minimum error-rate training (Och
2003), and another, disjoint set of 1,000 randomly selected sentences is used for evalu-
ation. Each system is built and tuned with six different LMs: MIX, O-based, and four
T-based models (Section 3.2). We use Bleu (Papineni et al 2002) to evaluate translation
quality. The results are listed in Tables 14 and 15.
The results are consistent and fully confirm our hypothesis. Across all language
pairs, MT systems using LMs compiled from translated-from-source texts consistently
outperform all other systems. Systems that use LMs compiled from texts originally
written in the target language always perform worst or second worst. We test the statis-
tical significance of the differences between the results using the bootstrap resampling
method (Koehn 2004). In all experiments, the best system (translated-from-source LM)
is significantly better than the system that uses the O-based LM (p < 0.01).
We now repeat the experiment with Hebrew to English translation. We construct a
Hebrew-to-English MT system with Moses, using a factored translation model (Koehn
and Hoang 2007). Every token in the training corpus is represented as two factors:
surface form and lemma. The Hebrew input is fully segmented (Itai and Wintner 2008).
The system is built and tuned with O- and T-based LMs. The O-based LM yields a
Bleu score of 11.94, whereas using the T-based LM results in somewhat higher Bleu
Table 12
Fitness of O- vs. T-based LMs to the reference set (EN-DE and EN-FR).
English to German translations
Orig. Lang. Perplexity Improvement (%)
Mix 106.37 20.24
O-DE 133.37
T-EN 99.39 25.47
T-FR 119.21 10.61
T-IT 123.35 7.51
T-NL 119.99 10.03
English to French translations
Orig. Lang. Perplexity Improvement (%)
Mix 58.71 3.20
O-FR 60.65
T-EN 49.44 18.47
T-DE 55.41 8.63
T-IT 57.75 4.77
T-NL 54.23 10.57
812
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Table 13
The effect of LM training corpus size on the fitness of LMs to the reference sets.
Hansard Reference Set
Hansard T-FR
Size Perplexity
1M 64.68
5M 47.63
10M 42.70
Hansard O-EN
Size Perplexity
1M 91.40
5M 66.95
10M 59.19
25M 51.59
50M 47.02
100M 43.70
Gigaword
Size Perplexity
100M 165.03
500M 151.00
1000M 145.88
Europarl Reference Set
Hansard T-FR
Size Perplexity
1M 169.66
5M 137.72
10M 128.65
Hansard O-EN
Size Perplexity
1M 198.93
5M 162.08
10M 150.05
25M 137.31
50M 129.43
100M 123.10
Gigaword
Size Perplexity
100M 136.72
500M 121.88
1000M 116.55
score, 12.07, but the difference is not statistically significant (p = 0.18). Presumably, the
low quality of both systems prevents the better LM frommaking a significant difference.
4.2.2 Larger Language Models. Again, the LMs used in the MT experiments reported here
are relatively small. To assess whether the benefits of using translated LMs carry over
to scenarios where larger original corpora exist, we build yet another set of French-to-
English MT systems. We use the Hansard SMT translation model and Hansard LMs
to train nine MT systems, three with varying sizes of translated texts and six with
varying sizes of original texts. We train additional MT systems with several subsets
of the Gigaword LM. We tune and evaluate on the Hansard reference set. In another
set of experiments we use the Europarl French-to-English scenario (using Europarl
813
Computational Linguistics Volume 38, Number 4
Table 14
Machine translation with various LMs; English target language.
DE to EN
LM Bleu
MIX 21.43
O-EN 21.10
T-DE 21.90
T-FR 21.16
T-IT 21.29
T-NL 21.20
FR to EN
LM Bleu
MIX 28.67
O-EN 27.98
T-DE 28.01
T-FR 29.14
T-IT 28.75
T-NL 28.11
IT to EN
LM Bleu
MIX 25.41
O-EN 24.69
T-DE 24.62
T-FR 25.37
T-IT 25.96
T-NL 24.77
NL to EN
LM Bleu
MIX 24.20
O-EN 23.40
T-DE 24.26
T-FR 23.56
T-IT 23.87
T-NL 24.52
Table 15
Machine translation with various LMs; non-English target language.
EN to DE
LM Bleu
MIX 13.00
O-DE 12.47
T-EN 13.10
T-FR 12.46
T-IT 12.65
T-NL 12.86
EN to FR
LM Bleu
MIX 24.83
O-FR 24.70
T-EN 25.31
T-DE 24.58
T-IT 24.89
T-NL 25.20
corpora for the translation model as well as for tuning and evaluation), but we use
the Hansard and Gigaword LMs to see whether our findings are consistent also when
LMs are trained on out-of-domain material.
Table 16 again demonstrates that language models compiled from original texts
must be up to ten times larger in order to yield translation quality similar to that of
LMs compiled from translated texts.4 In other words, much smaller translated LMs
perform better than much larger original ones, and this holds for various LM sizes,
both in-domain and out-of-domain. For example, on the Hansard corpus, a 10-million-
token T-FR language model yields a Bleu score of 34.67, whereas an O-EN language
model of 100 million tokens is required in order to yield a similar Bleu score of 34.44.
The systems that use the Gigaword LMs perform much worse in-domain, even with a
language model compiled from 1000M tokens. Out-of-domain, the Gigaword systems
are better than O-EN, but they require approximately five times more data to match the
performance of T-FR systems.
4.2.3 Enjoying Both Worlds. The previous section established the fact that language mod-
els compiled from translated texts are better for MT than ones compiled from original
texts, even when the original LMs are much larger. In many real-world scenarios,
however, one has access to texts of both types. Our results do not imply that original
4 The table only specifies three subsets of the Gigaword corpus, but the graphs show more data points.
Note that the x-axis is logarithmic. Incidentally, the graphs show that increases in (Gigaword) corpus size
do not monotonically translate to better MT quality.
814
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Table 16
The effect of LM size on MT performance.
Hansard TM and Test
Hansard T-FR
Size Bleu
1M 33.03
5M 34.25
10M 34.67
Hansard O-EN
Size Bleu
1M 31.91
5M 33.27
10M 33.43
25M 33.49
50M 34.29
100M 34.44
Gigaword
Size Bleu
100M 31.77
500M 32.31
1000M 32.51
Europarl TM and Test
Hansard T-FR
Size Bleu
1M 26.36
5M 27.06
10M 27.22
Hansard O-EN
Size Bleu
1M 26.06
5M 26.03
10M 26.72
25M 26.72
50M 27.01
100M 27.04
Gigaword
Size Bleu
100M 27.47
500M 27.71
1000M 27.69
texts are useless, and that only translated ones should be used. In this section we explore
various ways to combine original and translated texts, thereby yielding even better
language models.
For these experiments we use 10 million English-translated-from-French tokens
from the Hansard corpus (T-FR) and another 100 million original-English tokens from
the same source (O-EN). We combine them in five different ways: straightforward
concatenation of the corpora; a concatenation of the original-English corpus with the
translated corpus, upweighted by a factor of 10 and then of 20; log-linear modeling;
and an interpolated language model. In each experiment we report both the fitness of
the LM to the reference set, in terms of perplexity, and the quality of machine translation
815
Computational Linguistics Volume 38, Number 4
Table 17
Various combinations of original and translated texts and their effect on perplexity (PPL) and
translation quality (Bleu).
Hansard TM, LM and Test
Combination PPL Bleu
O-EN 43.70 34.44
T-FR 42.70 34.67
Concatenation 38.43 34.62
Concatenation x10 41.15 35.09
Concatenation x20 45.07 34.67
Log-Linear LM ? 35.26
Interpolated LM 36.69 35.35
Europarl TM and Test; Hansard LM
Combination PPL Bleu
O-EN 123.10 27.04
T-FR 128.65 27.22
Concatenation 116.71 27.14
Concatenation x10 135.09 27.29
Concatenation x20 152.02 27.09
Log-Linear LM ? 27.30
Interpolated LM 107.82 27.48
that uses this LM, in terms of Bleu.5 We execute each experiment twice, once (in-domain)
with the Hansard reference set and once (out-of-domain) where the translation model,
tuning corpus, and reference set al come from the Europarl FR-EN subcorpus, as above.
The results are listed in Table 17; we now provide a detailed explanation of these
experiments.
Concatenation of O and T texts. We train three language models by concatenating the
T-FR and O-EN corpora. First, we simply concatenate the corpora obtaining 110 million
tokens. Second, we upweight the T-FR corpus by a factor of 10 before the concatenation;
and finally, we upweight the T-FR corpus by a factor of 20 before the concatenation.
In the ?in-domain? scenario, the LM trained on a simple concatenation of the corpora
reduces the perplexity by more than 10%. The best translation quality is obtained
when the T-FR corpus is upweighted by a factor of 10. It improves by 0.42 Bleu points
compared to the MT system that uses T-FR (p = 0.074), and, more significantly, by 0.65
Bleu points compared to O-EN (p < 0.05). In the ?out-of-domain? scenario, there is a
small reduction in perplexity (about 5%) with a language model that is trained on a
simple concatenation of the corpora. There is also a very small improvement in the
translation quality (0.07 Bleu points compared to the T-FR system and 0.25 Bleu points
compared to O-EN).
Log-Linear combination of language models. The MOSES decoder uses log-linear model-
ing (Och and Ney 2001) to discriminate between better and worse hypotheses dur-
ing decoding. A log-linear model is defined as a combination of N feature functions
hi(t, s), 1 ? i ? N, that map input (s), output (t), or a pair of input and output strings
to a numeric value. Each feature function is associated with a model parameter ?i, its
feature weight, which determines the contribution of the feature to the overall value of
P(t|s). Formally, decoding based on a log-linear model is defined by:
t? = argmax
t
P(t|s) = argmax
t
{
N
?
i=1
?ihi(t, s)
}
(2)
5 Except log-linear models, for which we only report the quality of machine translation, because there are
two language models in this case and perplexity is harder to compute.
816
Lembersky, Ordan, and Wintner Language Models for Machine Translation
We train two language models, based on T-FR and O-EN. Then, we combine these
models by including them as different feature functions. The feature weight of each LM
is set by minimum error-rate tuning, optimizing the translation quality; this is the same
technique that Koehn and Schroeder (2007) employ for domain adaptation. In-domain,
this combination is better by 0.82 Bleu points compared with an MT system that uses
O-EN (p < 0.001), 0.59 Bleu points compared with the one that uses T-FR (p < 0.05).
Out of domain, this combination is again not significantly better than using T-FR only
(improvement of 0.08 Bleu points, p = 0.255).
Interpolated language models. In the interpolated scenario, two language models are
mixed on a fixed proportion ?, according to the following equation (Weintraub et al
1996):
p(w|h) = (1? ?) ? p(w|h;LM1)+ ? ? p(w|h;LM2) (3)
where w is a word, h is its ?history,? and ? is the fixed interpolation weight. We use
SRILM to train an interpolated language model from LM1 = O-EN and LM2 = T-FR.
The interpolation weight is tuned to minimize the perplexity of the combined model
with respect to the tuning set; we use the EM algorithm provided as part of the SRILM
toolkit to establish the optimized weights. In the in-domain scenario ? = 0.46 and in the
out-of-domain scenario ? = 0.49. The interpolated language model yields additional
improvement in perplexity and translation quality compared to all other models. It is
significantly better (p < 0.05) than the T-FR system on the in-domain scenarios, but the
improvement is less significant (p = 0.075) out of domain.
In summary, LMs compiled from source-translated-to-target texts are almost as
good asmuch larger LMs that also include large corpora of texts originally written in the
target language. Clearly, ignoring the status (original or translated) of monolingual texts
and creating a single languagemodel from all of them (the concatenation scenario) is not
much better than using only translated texts. In order to benefit from (often much larger)
original texts, one must consider more creative ways of combining the two subcorpora.
Of the methods we explored here, interpolated LMs provide the greatest advantage.
More research is needed in order to find an optimal combination.
5. Discussion
We use language models computed from different types of corpora to investigate
whether their fitness to a reference set of translated sentences can differentiate between
them (and, hence, between the corpora on which they are based). Our main findings
are that LMs compiled from manually translated corpora are much better predictors of
translated texts than LMs compiled from original-language corpora of the same size.
The results are robust, and are sustainable even when the corpora and the reference
sentences are abstracted in ways that retain their syntactic structure but ignore spe-
cific word meanings. Furthermore, we show that translated LMs are better predictors
of translated sentences even when the LMs are compiled from texts translated from
languages other than the source language. LMs based on texts translated from the source
language still outperform LMs translated from other languages, however.
We also show that MT systems based on translated-from-source-language LMs out-
perform MT systems based on originals LMs or LMs translated from other languages.
Again, these results are robust and the improvements are statistically significant. This
effect seems to be amplified as translation quality improves. Furthermore, our results
817
Computational Linguistics Volume 38, Number 4
Table 18
MT system performance as measured by METEOR and TER.
DE to EN
Orig. Lang. METEOR TER
O-EN 28.26 64.56
T-DE 28.64 63.57
FR to EN
Orig. Lang. METEOR TER
O-EN 33.05 54.45
T-FR 33.30 53.65
IT to EN
Orig. Lang. METEOR TER
O-EN 31.03 58.30
T-IT 31.16 57.63
NL to EN
Orig. Lang. METEOR TER
O-EN 29.97 60.29
T-NL 30.40 59.63
show that original LMs require five to ten times more data to exhibit the same fitness
to the reference set and the same translation quality as translated LMs.
More generally, this study confirms that insights drawn from the field of theoretical
Translation Studies, namely, the dual claim according to which translations as such
differ from originals, and translations from different source languages differ from each
other, can be verified experimentally and contribute to the performance of machine
translation.
One question, however, requires further investigation: Do MT systems based on
translated-from-source-language LMs produce better translations, or do they merely
generate sentences that are directly adapted to the reference set, thereby only improving
a specific evaluation metric, such as Bleu? We address this issue in three ways, showing
that the former is indeed the case. First, we use two automated evaluation metrics other
than Bleu, and show that the T-based LMs yield better MT systems even with different
metrics. Second, we perform a manual evaluation of a portion of the evaluation set. The
results show that human evaluators prefer translations produced by an MT system that
uses a T-based LM over translations produced by a system built with an O-based LM.
Finally, we provide a detailed analysis of the differences between O- and T-based LMs,
explaining these differences in terms of insights from Translation Studies.
5.1 Automatic Evaluation
First, we use two alternative automatic evaluation metrics, METEOR6 (Denkowski and
Lavie 2011) and TER (Snover et al 2006), to assess the quality of the MT systems
described in Section 4.2. We focus on four translation tasks: From German, French,
Italian, and Dutch to English.7 For each task we report the performance of two MT
systems: One that uses a language model compiled from original-English texts, and one
that uses a language model trained on texts translated from the source language. The
results, which are reported in Table 18, fully support our previous findings (recall that
lower TER is better): MT systems that use T-based LMs significantly outperform systems
that use O-based LMs.
6 More precisely, we use METEOR-RANK, the configuration used for WMT-2011.
7 All MT systems were tuned using Bleu.
818
Lembersky, Ordan, and Wintner Language Models for Machine Translation
5.2 Human Evaluation
To further establish the qualitative difference between translations produced with an
English-original language model and translations produced with a LM created from
French-translated-to-English texts, we conducted a human evaluation campaign, using
Amazon?sMechanical Turk as an inexpensive, reliable, and accessible pool of annotators
(Callison-Burch and Dredze 2010). We created a small evaluation corpus of 100 sen-
tences, selected randomly among all (Europarl) reference sentences whose length is
between 15 and 25 words. Each instance of the evaluation task includes two English
sentences, obtained from the two MT systems that use the O-EN and the T-FR language
models, respectively. Annotators are presented with these two translations, and are re-
quested to determine which one is better. The definition given to annotators is: ?A better
translation is more fluent, reflecting better use of English.? Observe that because the
only variable that distinguishes between the two MT systems is the different language
model, we only have to evaluate the fluency of the target sentence, not its faithfulness
to the source. Consequently, we do not present the source or the reference translation
to the annotators. All annotators were located in the United States (and, therefore, are
presumably English speakers).
As a control set, we added a set of 10 sentences produced with the O-based LM,
which were paired with their (manually created) reference translations, and 10 sen-
tences produced with the T-based LM, again paired with their references. Each of the
120 evaluation instances was assigned to 10 different Mechanical Turk annotators. We
report two evaluation metrics: score and majority. The score of a given sentence pair
?e1, e2? is i/j, where i is the number of annotators who preferred e1 over e2, and j = 10? i
is the number of annotators preferring e2. For such a sentence pair, the majority is e1 if
i > j, e2 if i < j, and undefined otherwise.
The average score of the 10 sentences in the O-vs.-reference control set is 22/78,
and the majority is the reference translation in all but one of the instances. As for the
T-vs.-reference control set, the average score is 18/82, and the majority is the reference
in all of the instances. This indicates that the annotators are reliable, and also that it
is unrealistic to expect a clear-cut distinction even between human translations and
machine-generated output.
As for the actual evaluation set, the average score of O-EN vs. T-FR is 38/62, and
the majority is T-FR in 75% of the cases, O-EN in only 25% of the sentence pairs. We
take these results as a very strong indication that English sentences generated by an
MT system whose language model is compiled from translated texts are perceived
by humans as more fluent than ones generated by a system built with an O-based
language model. Not only is the improvement reflected in significantly higher Bleu
(and METEOR, TER) scores, but it is undoubtedly also perceived as such by human
annotators.
5.3 Analysis
In order to look into the differences between T and O qualitatively, rather than quantita-
tively, we turn now to study several concrete examples. To do so, we extracted approx-
imately 200 sentences from the French?English Europarl evaluation set; we chose all
sentences of length between 15 and 25. In addition, we extracted the 100 most frequent
n-grams, for 1 ? n ? 5, from both English-original and English-translated-from-French
Europarl corpora. As both corpora include approximately the same number of tokens,
we report counts in the following rather than frequencies.
819
Computational Linguistics Volume 38, Number 4
The differences between O and T texts are consistent with well-established observa-
tions of translation scholars. Consider the explicitation hypothesis (Blum-Kulka 1986),
which Se?guinot (1998, page 108) spells out thus:
1. ?something which was implied or understood through presupposition in
the source text is overtly expressed in the translation?
2. ?something is expressed in the translation which was not in the original?
3. ?an element in the source text is given greater importance in the
translation through focus, emphasis, or lexical choice?
Blum-Kulka (1986) uses the term cohesive markers to refer to items that are utilized by
the translator which cannot be found overtly in the source text. One would expect such
markers to be much more prevalent in translationese.
An immediate example of (1) is the case of acronyms: these tend to be spelled out
in translated texts. Indeed, the acronym EU is ranked 77 among the O-EN bigrams,
whereas in T-FR it does not appear in the top 100. On the other hand, the explicit trigram
The European Union occurs more frequently in T than in O.
Similarly, an instance of (2) is the cohesivemarker because in the following example,
which appears in T but neither appears in O nor can it be traced back to the original
source sentence:
Source Enfin, ce qui est grave dans le rapport de M. Olivier Tautologie, c?est qu?il
propose une constitution tripotage.
O Finally, which is serious in the report of Mr Olivier Tautologie, is that it proposes a
constitution tripotage.
T Finally, and this is serious in the report by Mr olivier Tautologie, it is because it
proposes a constitution tripotage.
Another cohesive marker, nevertheless, is correctly generated only in the T-based trans-
lation in the following example:
Source C?est quand me?me quelque chose de pre?cieux qui a e?te? souligne? par tous les
membres du conseil europe?en.
O Even when it is something of valuable which has been pointed out by all the mem-
bers of the European Council.
T It is nevertheless something of a valuable which has been pointed out by all the
members of the European Council.
Other cohesive markers discussed by Blum-Kulka (1986) are over-represented in
T compared with O. These include: therefore (3,187 occurrences in T, 1,983 in O); for
example (863 occurrences in T, 701 in O); in particular (1336 vs. 1068); first of all (601 vs.
266); in fact (1014 vs. 441); in other words (553 vs. 87); with regard to (1137 vs. 310); in
order to (2,016 vs. 603); in this respect (363 vs. 94); on the one hand (288 vs. 72); on the
other hand (428 vs. 76); and with a view to (213 vs. 51). A similar list of markers have
been shown to be excellent discriminating features between original and translated texts
(from several European languages, including French) in an independent study (Koppel
and Ordan 2011).
820
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Another phenomenon we notice is that the T-based language model does a much
better job translating verbs than the O-based language model. In two very large corpora
of French and English (Ferraresi et al 2008), verbs are much more frequent in French
than in English (0.124 vs. 0.091). Human translations from French to English, therefore,
provide many more examples of verbs from which to model. Indeed, we encounter
several examples in which the O-based translation system fails to use a verb at all, or to
use one correctly, compared with the T-based system:
Source Une telle Europe serait un gage de paix et marquerait le refus de tout national-
isme ethnique.
O Such a Europe would be a show of peace and would the rejection of any ethnic
nationalism.
T Such a Europe would be a show of peace and would mark the refusal of all ethnic
nationalism.
Source Votre rapport, madame Sudre, met l?accent, a` juste titre, sur la ne?cessite? d?agir
dans la dure?e.
O Your report, Mrs Sudre, its emphasis, quite rightly, on the need to act in the long
term.
T Your report, Mrs Sudre, places the emphasis, quite rightly, on the need to act in the
long term.
Source Cette proposition, si elle constitue un pas dans la bonne direction n?en comporte
pas moins de nombreuses lacunes auxquelles le rapport evans reme?die.
O This proposal, if it is a step in the right direction do not least in contains many
shortcomings which the evans report resolve.
T This proposal, if it is a step in the right direction it contains no less many shortcom-
ings which the evans report resolve.
Last, there are several cases of interference, which Toury (1995, page 275) defines as
follows: ?Phenomena pertaining to the make-up of the source text tend to be transferred
to the target text.? In the following example, do not say nothing more is a literal
translation of the French construction On ne dit rien non plus. The T-based translation
is much more fluent:
Source On ne dit rien non plus sur la responsabilite? des fabricants, notamment en
grande-bretagne, qui ont e?te? les premiers responsables.
O We do not say nothing more on the responsibility of the manufacturers, particularly
in Britain, which were the first responsible.
T We do not say anything either on the responsibility of the manufacturers, particu-
larly in great Britain, who were the first responsible.
Incidentally, there are also some cultural differences between O and T that we
deem less important, because they are not part of the ?translationese dialect? but rather
indicate differences pertaining to the culture from which the speaker arrives. Most
notable is the form ladies and gentlemen, which is the tenth most frequent trigram in T,
but does not even rank among the top 100 in O. This is already noted by van Halteren
821
Computational Linguistics Volume 38, Number 4
(2008), according to whom this form is significantly more frequent in translations from
five European languages as opposed to original English.
In terms of (shallow) syntactic structure, we observe that part-of-speech n-grams
are distributed somewhat differently in O and in T (we use the POS-tagged Europarl
corpus of Section 4.1.2 for the following analysis). For example, proper nouns are more
frequent in O (ranking 7 among all POS 1-grams) than in T (rank 9). This has influence
on longer n-grams: For example, the 3-gram PRP MD VB is 20% more frequent in O
than in T. The sequence <S> PRP VBP is almost twice as frequent in O. The 4-gram IN
DT NN </S> is 25% more frequent in O. In contrast, the 4-gram IN DT NNS IN is 15%
more frequent in T than in O. A full analysis of such patterns is beyond the scope of
this article.
Summing up, T-based language models are more fluent and therefore yield better
translation results for the following reasons: They are more cohesive, less influenced by
structural differences between the languages, such as the under-representation of verbs
in original English texts, and less prone to interference (i.e., they can break away from
the original towards a more coherent model of the target language).
5.4 Future Research
This work is among the first to use insights from Translation Studies in order to improve
machine translation, and to use computational linguistic methodologies to corroborate
Translation Studies hypotheses. We believe that there are still vast opportunities for
fertile cross-disciplinary research in these directions. First, we only address the language
model in the present work. Kurokawa, Goutte, and Isabelle (2009) investigate the rela-
tions between the direction of translation and the quality of the translation model used
by SMT systems. There are various ways in which the two approaches can be extended
and combined, and we are actively pursuing such research directions now (Lembersky,
Ordan, and Wintner 2012).
This work also bears on language typology: We conjecture that LMs compiled
from texts translated not from the original language, but from a closely related one,
can be better than LMs compiled from texts translated from a more distant language.
Some of our results support this hypothesis, but more research is needed in order to
establish it.
The fact that translations seem to make do with fewer words (cf. also Laviosa 2008)
call into question certain norms in comparing corpora in the field of machine transla-
tion. Translated and original texts can be expected to either have the same number of
sentences or the same number of tokens, but not both. Similarly, theymay have the same
number of tokens or the same number of types, but not both.
Another interesting question that arises from this study is whether the perplexity
of a language model on a reference set is a good predictor of a translation quality
measure, such as Bleu. Although our results show a certain correlation between the
perplexity and Bleu, we acknowledge the fact that these results need further corrob-
oration. Chen, Beeferman, and Rosenfeld (1998) examine the ability of perplexity to
estimate the performance of speech recognition. They find that perplexity often does
not correlate well with word-error rates. As it is extremely important to have a reliable
measure capable of estimating the effect of language model improvements on transla-
tion quality without requiring expensive decoding resources, we believe that finding
correspondences between perplexity and the quality of MT is a valuable topic for future
research.
822
Lembersky, Ordan, and Wintner Language Models for Machine Translation
Acknowledgments
We are grateful to Cyril Goutte, George
Foster, and Pierre Isabelle for providing us
with an annotated version of the Hansard
corpus. Alon Lavie has been instrumental
in stimulating some of the ideas reported
in this article, as well as in his long-term
support and advice. We benefitted greatly
from several constructive suggestions by the
three anonymous Computational Linguistics
referees. This research was supported by the
Israel Science Foundation (grant no. 137/06)
and by a grant from the Israeli Ministry of
Science and Technology.
References
Al-Shabab, Omar S. 1996. Interpretation and
the Language of Translation: Creativity and
Conventions in Translation. Janus,
Edinburgh.
Bahl, Lalit R., Frederick Jelinek, and
Robert L. Mercer. 1983. A maximum
likelihood approach to continuous speech
recognition. IEEE Transactions on Pattern
Analysis and Machine Intelligence,
5(2):179?190.
Baker, Mona. 1993. Corpus linguistics and
translation studies: Implications and
applications. In Gill Francis Mona Baker
and Elena Tognini-Bonelli, editors, Text and
Technology: In Honour of John Sinclair. John
Benjamins, Amsterdam, pages 233?252.
Baker, Mona. 1995. Corpora in translation
studies: An overview and some
suggestions for future research.
Target, 7(2):223?243.
Baker, Mona. 1996. Corpus-based translation
studies: The challenges that lie ahead.
In Gill Francis Mona Baker and Elena
Tognini-Bonelli, editors, Terminology,
LSP and Translation. Studies in Language
Engineering in Honour of Juan C. Sager. John
Benjamins, Amsterdam, pages 175?186.
Baroni, Marco and Silvia Bernardini.
2006. A new approach to the study of
Translationese: Machine-learning the
difference between original and translated
text. Literary and Linguistic Computing,
21(3):259?274.
Blum-Kulka, Shoshana. 1986. Shifts of
cohesion and coherence in translation.
In Juliane House and Shoshana Editors
Blum-Kulka, editors, Interlingual and
Intercultural Communication Discourse and
Cognition in Translation and Second Language
Acquisition Studies, volume 35. Gunter
Narr Verlag, Berlin, pages 17?35.
Brants, Thorsten and Peng Xu. 2009.
Distributed language models. In
Proceedings of Human Language Technologies:
The 2009 Annual Conference of the North
American Chapter of the Association for
Computational Linguistics, Companion
Volume: Tutorial Abstracts, pages 3?4,
Boulder, CO.
Callison-Burch, Chris and Mark Dredze.
2010. Creating speech and language
data with Amazon?s Mechanical Turk.
In Proceedings of the NAACL HLT 2010
Workshop on Creating Speech and Language
Data with Amazon?s Mechanical Turk,
pages 1?12, Los Angeles, CA.
Chen, Stanley, Douglas Beeferman, and
Ronald Rosenfeld. 1998. Evaluation
metrics for language models. In
Proceedings of the DARPA Broadcast
News Transcription and Understanding
Workshop (BNTUW), Landsdowne, PA.
Chen, Stanley F. 1998. An empirical study
of smoothing techniques for language
modeling. Technical report 10-98,
Computer Science Group, Harvard
University, Cambridge, MA.
Denkowski, Michael and Alon Lavie. 2011.
Meteor 1.3: Automatic metric for reliable
optimization and evaluation of machine
translation systems. In Proceedings of the
Sixth Workshop on Statistical Machine
Translation, pages 85?91, Edinburgh.
Ferraresi, Adriano, Silvia Bernardini, Picci
Giovanni, and Marco Baroni. 2008. Web
corpora for bilingual lexicography. a pilot
study of English/French collocation
extraction and translation. In Proceedings
of The International Symposium on Using
Corpora in Contrastive and Translation
Studies, Hangzhou.
Finkel, Jenny Rose, Trond Grenager, and
Christopher Manning. 2005. Incorporating
non-local information into information
extraction systems by Gibbs sampling.
In ACL ?05: Proceedings of the 43rd Annual
Meeting of the Association for Computational
Linguistics, pages 363?370, Morristown, NJ.
Frawley, William. 1984. Prolegomenon to a
theory of translation. In William Frawley,
editor, Translation. Literary, Linguistic and
Philosophical Perspectives. University of
Delaware Press, Newark, pages 159?175.
Gellerstam, Martin. 1986. Translationese in
Swedish novels translated from English.
In Lars Wollin and Hans Lindquist,
editors, Translation Studies in Scandinavia.
CWK Gleerup, Lund, pages 88?95.
Graff, David and Christopher Cieri. 2007.
English Gigaword. Linguistic Data
823
Computational Linguistics Volume 38, Number 4
Consortium, Philadelphia, PA, third
edition. LDC Catalog No. LDC2007T07.
Ilisei, Iustina, Diana Inkpen, Gloria
Corpas Pastor, and Ruslan Mitkov. 2010.
Identification of translationese:
A machine learning approach.
In Alexander F. Gelbukh, editor,
Proceedings of CICLing-2010: 11th
International Conference on Computational
Linguistics and Intelligent Text Processing,
volume 6008 of Lecture Notes in Computer
Science, pages 503?511. Springer, Berlin.
Itai, Alon and Shuly Wintner. 2008.
Language resources for Hebrew. Language
Resources and Evaluation, 42(1):75?98.
Jelinek, Frederick, Robert L. Mercer, Lalit R.
Bahl, and J. K. Baker. 1977. Perplexity?
A measure of the difficulty of speech
recognition tasks. Journal of the Acoustical
Society of America, 62:S63.
Jurafsky, Daniel and James H. Martin.
2008. Speech and Language Processing:
An Introduction to Natural Language
Processing, Computational Linguistics
and Speech Recognition. Prentice Hall,
Upper Saddle River, NJ.
Koehn, Philipp. 2004. Statistical significance
tests for machine translation evaluation.
In Proceedings of EMNLP 2004,
pages 388?395, Barcelona.
Koehn, Philipp. 2005. Europarl: A parallel
corpus for statistical machine translation.
In Proceedings of the MT Summit X,
pages 79?86, Phuket.
Koehn, Philipp and Hieu Hoang. 2007.
Factored translation models. In Proceedings
of the 2007 Joint Conference on Empirical
Methods in Natural Language Processing and
Computational Natural Language Learning
(EMNLP-CoNLL), pages 868?876, Prague.
Koehn, Philipp, Hieu Hoang, Alexandra
Birch, Chris Callison-Burch, Marcello
Federico, Nicola Bertoldi, Brooke Cowan,
Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondrej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses:
Open source toolkit for statistical machine
translation. In Proceedings of the 45th
Annual Meeting of the Association for
Computational Linguistics Companion
Volume Proceedings of the Demo and
Poster Sessions, pages 177?180, Prague.
Koehn, Philipp, Franz Josef Och, and
Daniel Marcu. 2003. Statistical
phrase-based translation. In NAACL ?03:
Proceedings of the 2003 Conference of the
North American Chapter of the Association for
Computational Linguistics on Human Language
Technology, pages 48?54, Edmonton.
Koehn, Philipp and Josh Schroeder. 2007.
Experiments in domain adaptation
for statistical machine translation. In
Proceedings of the Second Workshop on
Statistical Machine Translation, StatMT ?07,
pages 224?227, Stroudsburg, PA.
Koppel, Moshe and Noam Ordan.
2011. Translationese and its dialects.
In Proceedings of the 49th Annual Meeting
of the Association for Computational
Linguistics: Human Language Technologies,
pages 1318?1326, Portland, OR.
Kurokawa, David, Cyril Goutte, and Pierre
Isabelle. 2009. Automatic detection of
translated text and its impact on machine
translation. In Proceedings of MT-Summit
XII, Kurokawa.
Laviosa, Sara. 1998. Core patterns of lexical
use in a comparable corpus of English
lexical prose.Meta, 43(4):557?570.
Laviosa, Sara. 2008. Universals. In Mona
Baker and Gabriela Saldanha, editors,
Routledge Encyclopedia of Translation
Studies, 2nd Edition. Routledge (Taylor
and Francis), New York, pages 288?292.
Lembersky, Gennadi, Noam Ordan, and
Shuly Wintner. 2011. Language models
for machine translation: Original vs.
translated texts. In Proceedings of EMNLP,
pages 363?374, Edinburgh.
Lembersky, Gennadi, Noam Ordan, and
Shuly Wintner. 2012. Adapting translation
models to translationese improves SMT.
In Proceedings of the 13th Conference of the
European Chapter of the Association for
Computational Linguistics (EACL-2012),
Avignon.
Och, Franz Josef. 2003. Minimum error rate
training in statistical machine translation.
In ACL ?03: Proceedings of the 41st Annual
Meeting of the Association for Computational
Linguistics, pages 160?167, Morristown, NJ.
Och, Franz Josef and Hermann Ney. 2001.
Discriminative training and maximum
entropy models for statistical machine
translation. In ACL ?02: Proceedings of the
40th Annual Meeting of the Association for
Computational Linguistics, pages 295?302,
Morristown, NJ.
Papineni, Kishore, Salim Roukos, Todd
Ward, and Wei-Jing Zhu. 2002. BLEU:
A method for automatic evaluation
of machine translation. In ACL ?02:
Proceedings of the 40th Annual Meeting of the
Association for Computational Linguistics,
pages 311?318, Morristown, NJ.
Pym, Anthony and Grzegorz Chrupa?a. 2005.
The quantitative analysis of translation
flows in the age of an international
824
Lembersky, Ordan, and Wintner Language Models for Machine Translation
language. In Albert Branchadell and
Lovell M. West, editors, Less Translated
Languages. John Benjamins, Amsterdam,
pages 27?38.
Santos, Diana. 1995. On grammatical
translationese. In Koskenniemi, Kimmo
(comp.), Short papers presented at the Tenth
Scandinavian Conference on Computational
Linguistics (Helsinki), University of
Helsinki, pages 29?30.
Se?guinot, Candice. 1998. Pragmatics
and the explicitation hypothesis. TTR:
Traduction, Terminologie, Re?daction,
11(2):106?114.
Snover, Matthew, Bonnie Dorr, Richard
Schwartz, Linnea Micciulla, and
John Makhoul. 2006. A study of
translation edit rate with targeted
human annotation. In Proceedings of the
Association for Machine Translation in the
Americas (AMTA-2006), pages 223?231,
Cambridge, MA.
Stolcke, Andreas. 2002. SRILM?An
extensible language modeling toolkit.
In Procedings of International Conference
on Spoken Language Processing,
pages 901?904, Denver, CO.
Toury, Gideon. 1980. In Search of a Theory of
Translation. The Porter Institute for Poetics
and Semiotics, Tel Aviv University,
Tel Aviv.
Toury, Gideon. 1995. Descriptive Translation
Studies and Beyond. John Benjamins,
Amsterdam / Philadelphia.
Toutanova, Kristina and Christopher D.
Manning. 2000. Enriching the knowledge
sources used in a maximum entropy
part-of-speech tagger. In Proceedings of the
2000 Joint SIGDAT Conference on Empirical
Methods in Natural Language Processing
and Very Large Corpora, pages 63?70,
Morristown, NJ.
Tsvetkov, Yulia and Shuly Wintner. 2010.
Automatic acquisition of parallel corpora
fromWebsites with dynamic content.
In Proceedings of the Seventh Conference
on International Language Resources and
Evaluation (LREC?10), pages 3389?3392,
Valleta.
van Halteren, Hans. 2008. Source language
markers in EUROPARL translations.
In COLING ?08: Proceedings of the 22nd
International Conference on Computational
Linguistics, pages 937?944, Morristown, NJ.
Weintraub, Mitch, Yaman Aksu, Satya
Dharanipragada, Sanjeev Khudanpur,
Herman Ney, John Prange, Andreas
Stolcke, Fred Jelinek, and Liz Shriberg.
1996. Fast training and portability. LM95
project report, Center for Language and
Speech Processing, Johns Hopkins
University, Baltimore, MD.
825

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1318?1326,
Portland, Oregon, June 19-24, 2011. c?2011 Association for Computational Linguistics
Translationese and Its Dialects 
Moshe Koppel Noam Ordan 
Department of Computer Science Department of Computer Science 
Bar Ilan University University of Haifa 
Ramat-Gan, Israel 52900 Haifa, Israel 31905 
moishk@gmail.com noam.ordan@gmail.com 
 
 
 
Abstract 
While it is has often been observed that the 
product of translation is somehow different 
than non-translated text, scholars have empha-
sized two distinct bases for such differences. 
Some have noted interference from the source 
language spilling over into translation in a 
source-language-specific way, while others 
have noted general effects of the process of 
translation that are independent of source lan-
guage. Using a series of text categorization 
experiments, we show that both these effects 
exist and that, moreover, there is a continuum 
between them. There are many effects of 
translation that are consistent among texts 
translated from a given source language, some 
of which are consistent even among texts 
translated from families of source languages. 
Significantly, we find that even for widely 
unrelated source languages and multiple ge-
nres, differences between translated texts and 
non-translated texts are sufficient for a learned 
classifier to accurately determine if a given 
text is translated or original. 
1 Introduction 
The products of translation (written or oral) are 
generally assumed to be ontologically different 
from non-translated texts. Researchers have em-
phasized two aspects of this difference. Some 
(Baker 1993) have emphasized general effects of 
the process of translation that are independent of 
source language and regard the collective product 
of this process in a given target language as an ?in-
terlanguage? (Selinker, 1972), ?third code? (Fraw-
ley, 1984) or ?translationese? (Gellerstam, 1986). 
Others (Toury, 1995) have emphasized the effects 
of interference, the process by which a specific 
source language leaves distinct marks or finger-
prints in the target language, so that translations 
from different source languages into the same tar-
get language may be regarded as distinct dialects 
of translationese.  
We wish to use text categorization methods to 
set both of these claims on a firm empirical foun-
dation. We will begin by bringing evidence for two 
claims: 
(1) Translations from different source languages 
into the same target language are sufficiently dif-
ferent from each other for a learned classifier to 
accurately identify the source language of a given 
translated text;  
(2) Translations from a mix of source languages 
are sufficiently distinct from texts originally writ-
ten in the target language for a learned classifier to 
accurately determine if a given text is translated or 
original. 
Each of these claims has been made before, but 
our results will strengthen them in a number of 
ways. Furthermore, we will show that the degree of 
difference between translations from two source 
languages reflects the degree of difference between 
the source languages themselves. Translations 
from cognate languages differ from non-translated 
texts in similar ways, while translations from unre-
lated languages differ from non-translated texts in 
distinct ways. The same result holds for families of 
languages. 
The outline of the paper is as follows. In the fol-
lowing section, we show that translations from dif-
ferent source languages can be distinguished from 
each other and that closely related source languag-
es manifest similar forms of interference. In sec-
tion 3, we show that, in a corpus involving five 
European languages, we can distinguish translatio-
nese from non-translated text and we consider 
some salient markers of translationese. In section 
1318
4, we consider the extent to which markers of 
translationese cross over into non-European lan-
guages as well as into different genres. Finally, we 
consider possible applications and implications for 
future studies.  
2 Interference Effects in Translationese 
In this section, we perform several text categoriza-
tion experiments designed to show the extent to 
which interference affects (both positively and ne-
gatively) our ability to classify documents. 
2.1 The Europarl Corpus 
The main corpus we will use throughout this paper 
is Europarl (Koehn, 2005), which consists of tran-
scripts of addresses given in the European Parlia-
ment. The full corpus consists of texts translated 
into English from 11 different languages (and vice 
versa), as well as texts originally produced in Eng-
lish. For our purposes, it will be sufficient to use 
translations from five languages (Finnish, French, 
German, Italian and Spanish), as well as original 
English. We note that this corpus constitutes a 
comparable corpus (Laviosa, 1997), since it con-
tains (1) texts written originally in a certain lan-
guage (English), as well as (2) texts translated into 
that same language, matched for genre, domain, 
publication timeframe, etc. Each of the five trans-
lated components is a text file containing just un-
der 500,000 words; the original English component 
is a file of the same size as the aggregate of the 
other five. 
The five source languages we use were selected 
by first eliminating several source languages for 
which the available text was limited and then 
choosing from among the remaining languages, 
those of varying degrees of pairwise similarity. 
Thus, we select three cognate (Romance) languag-
es (French, Italian and Spanish), a fourth less re-
lated language (German), and a fifth even further 
removed (Finnish). As will become clear, the mo-
tivation is to see whether the distance between the 
languages impacts the distinctiveness of the trans-
lation product. 
We divide each of the translated corpora into 
250 equal chunks, paying no attention to natural 
units within the corpus. Similarly, we divide the 
original English corpus into 1250 equal chunks. 
We set aside 50 chunks from each of the translated 
corpora and 250 chunks from the original English 
corpus for development purposes (as will be ex-
plained below). The experiments described below 
use the remaining 1000 translated chunks and 1000 
original English chunks.   
2.2 Identifying source language 
Our objective in this section is to measure the ex-
tent to which translations are affected by source 
language. Our first experiment will be to use text 
categorization methods to learn a classifier that 
categorizes translations according to source lan-
guage. We will check the accuracy of such clas-
sifiers on out-of-sample texts. High accuracy 
would reflect that there are exploitable differences 
among translations of otherwise comparable texts 
that differ only in terms of source language. 
The details of the experiment are as follows. We 
use the 200 chunks from each translated corpus, as 
described above. We use as our feature set a list of 
300 function words taken from LIWC (Pennebak-
er, 2001) and represent each chunk as a vector of 
size 300 in which each entry represents the fre-
quency of the corresponding feature in the chunk. 
The restriction to function words is crucial; we 
wish to rely only on stylistic differences rather than 
content differences that might be artifacts of the 
corpus. 
We use Bayesian logistic regression (Madigan, 
2005) as our learning method in order to learn a 
classifier that classifies a given text into one of five 
classes representing the different source languages. 
We use 10-fold cross-validation as our testing me-
thod.  
We find that 92.7% of documents are correctly 
classified.  
In Table 1 we show the confusion matrix for the 
five languages. As can be seen, there are more mis-
takes across the three cognate languages than be-
tween those three languages and German and still 
fewer mistakes involving the more distant Finnish 
language. 
 
 It Fr Es De Fi 
It 169 19 8 4 0 
Fr 18 161 12 8 1 
Es 3 11 172 11 3 
De 4 12 3 178 3 
Fi 0 1 2 5 192 
Table 1: Confusion matrix for 10-fold cross validation 
experiment to determine source language of texts trans-
lated into English 
1319
 
This result strengthens that of van Halteren 
(2008) in a similar experiment. Van Halteren, also 
using Europarl (but with Dutch as the fifth source 
language, rather than Finnish), obtained accuracy 
of 87.2%-96.7% for a two-way decision on source 
language, and 81.5%-87.4% for a six-way decision 
(including the original which has no source lan-
guage). Significantly, though, van Halteren?s fea-
ture set included content words and he notes that 
many of the most salient differences reflected dif-
ferences in thematic emphasis. By restricting our 
feature set to function words, we neutralize such 
effects. 
In Table 2, we show the two words most over-
represented and the two words most under-
represented in translations from each source lan-
guage (ranked according to an unpaired T-test). 
For each of these, the difference between frequen-
cy of use in the indicated language and frequency 
of use in the other languages in aggregate is signif-
icant at p<0.01. 
 
 over-represented under-represented 
Fr of, finally here, also 
It upon, moreover also, here 
Es with, therefore too, then 
De here, then of, moreover 
Fi be, example me, which 
Table 2: Most salient markers of translations from each 
source language. 
 
The two most underrepresented words for 
French and Italian, respectively, are in fact identic-
al. Furthermore, the word too which is underrepre-
sented for Spanish is a near synonym of also which 
appears in both French and Spanish. This suggests 
the possibility that interference effects in cognate 
languages such as French, Italian and Spanish 
might be similar. We will see presently that this is 
in fact the case.  
When a less related language is involved we see 
the opposite picture. For German, both underrepre-
sented items appear as overrepresented in the 
Romance languages, and, conversely, underrepre-
sented items in the Romance languages appear as 
overrepresented items for German. This may cast 
doubt on the idea that all translations share univer-
sal properties and that at best we may claim that 
particular properties are shared by closely related 
languages but not others. In the experiments pre-
sented in the next subsection, we?ll find that trans-
lationese is gradable: closely related languages 
share more features, yet even further removed lan-
guages share enough properties to hold the general 
translationese hypothesis as valid.  
2.3 Identifying translationese per source lan-
guage  
We now wish to measure in a subtler manner the 
extent to which interference affects translation. In 
this experiment, the challenge is to learn a classifi-
er that classifies a text as belonging to one of only 
two classes: original English (O) or translated-into-
English (T). The catch is that all our training texts 
for the class T will be translations from some fixed 
source language, while all our test documents in T 
will be translations from a different source lan-
guage. What accuracy can be achieved in such an 
experiment? The answer to this question will tell 
us a great deal about how much of translationese is 
general and how much of it is language dependent. 
If accuracy is close to 100%, translationese is pure-
ly general (Baker, 1993). (We already know from 
the previous experiment that that's not the case.). If 
accuracy is near 50%, there are no general effects, 
just language-dependent ones. Note that, whereas 
in our first experiment above pair-specific interfe-
rence facilitated good classification, in this expe-
riment pair-specific interference is an impediment 
to good classification. 
The details of the experiment are as follows. We 
create, for example, a ?French? corpus consisting 
of the 200 chunks of text translated from French 
and 200 original English texts. We similarly create 
a corpus for each of the other source languages, 
taking care that each of the 1000 original English 
texts appears in exactly one of the corpora. As 
above, we represent each chunk in terms of fre-
quencies of function words. Now, using Bayesian 
logistic regression, we learn a classifier that distin-
guishes T from O in the French corpus. We then 
apply this learned classifier to the texts in, for ex-
ample, the equivalent ?Italian? corpus to see if we 
can classify them as translated or original. We re-
peat this for each of the 25 ?train_corpus, 
test_corpus? pairs. 
In Table 3, we show the accuracy obtained for 
each such pair. (For the case where the training 
corpus and testing corpus are identical ? the di-
1320
agonal of the matrix ? we show results for ten-fold 
cross-validation.)  
We note several interesting facts. First, results of 
cross-validation within each corpus are very 
strong. For any given source language, it is quite 
easy to distinguish translations from original Eng-
lish. This corroborates results obtained by Baroni 
and Bernardini (2006), Ilisei et al (2010), Kuro-
kawa et al (2009) and van Halteren (2008), which 
we will discuss below.  
We note further, that for the cases where we 
train on one source language and test on another, 
results are far worse. This clearly indicates that 
interference effects from one source language 
might be misleading when used to identify transla-
tions from a different language. Thus, for example, 
in the Finnish corpus, the word me is a strong indi-
cator of original English (constituting 0.0003 of 
tokens in texts translated from Finnish as opposed 
to 0.0015 of tokens in original English texts), but 
in the German corpus, me is an indicator of trans-
lated text (constituting 0.0020 of tokens in text 
translated from German). 
The most interesting result that can be seen in 
this table is that the accuracy obtained when train-
ing using language x and testing using language y 
depends precisely on the degree of similarity be-
tween x and y. Thus, for training and testing within 
the three cognate languages, results are fairly 
strong, ranging between 84.5% and 91.5%. For 
training/testing on German and testing/training on 
one of the other European languages, results are 
worse, ranging from 68.5% to 83.3%. Finally, for 
training/testing on Finnish and testing/training on 
any of the European languages, results are still 
worse, hovering near 60% (with the single unex-
plained outlier for training on German and testing 
on Finnish).  
Finally, we note that even in the case of training 
or testing on Finnish, results are considerably bet-
ter than random, suggesting that despite the con-
founding effects of interference, some general 
properties of translationese are being picked up in 
each case. We explore these in the following sec-
tion. 
 
3 General Properties of Translationese  
Having established that there are source-language-
dependent effects on translations, let?s now con-
sider source-language-independent effects on 
translation. 
3.1 Identifying translationese 
In order to identify general effects on translation, 
we now consider the same two-class classification 
problem as above, distinguishing T from O, except 
that now the translated texts in both our train and 
test data will be drawn from multiple source lan-
guages. If we succeed at this task, it must be be-
cause of features of translationese that cross 
source-languages.  
The details of our experiment are as follows. We 
use as our translated corpus, the 1000 translated 
chunks (200 from each of five source languages) 
and as our original English corpus all 1000 original 
English chunks. As above, we represent each 
chunk in terms of function words frequencies. We 
use Bayesian logistic regression to learn a two-
class classifier and test its accuracy using ten-fold 
cross-validation.  
Remarkably, we obtain accuracy of 96.7%.  
This result extends and strengthens results re-
ported in some earlier studies. Ilisei et al (2010), 
Kurokawa (2009) and van Halteren (2008) each 
obtained above 90% accuracy in distinguishing 
translation from original. However, in each case 
the translations were from a single source lan-
guage. (Van Halteren considered multiple source 
languages, but each learned classifier used only 
one of them.) Thus, those results do not prove that 
translationese has distinctive source-language-
independent features. To our knowledge, the only 
earlier work that used a learned classifier to identi-
fy translations in which both test and train sets in-
volved multiple source languages is Baroni and 
Bernardini (2006), in which the target language 
was Italian and the source languages were known 
to be varied. The actual distribution of source lan-
guages was, however, not known to the research-
ers. They obtained accuracy of 86.7%. Their result 
was obtained using combinations of lexical and 
syntactic features. 
 
   Train     
 It Fr Es De Fi 
It 98.3 91.5 86.5 71.3 61.5 
Fr 91 97 86.5 68.5 60.8 
Es 84.5 88.3 95.8 76.3 59.5 
De 82 83.3 78.5 95 80.8 
Fi 56 60.3 56 62.3 97.3 
Table 3: Results of learning a T vs. O classifier us-
ing one source language and testing it using another 
source language 
1321
3.2 Some distinguishing features 
Let us now consider some of the most salient func-
tion words for which frequency of usage in T dif-
fers significantly from that in O. While there are 
many such features, we focus on two categories of 
words that are most prominent among those with 
the most significant differences.  
   First, we consider animate pronouns. In Table 4, 
we show the frequencies of animate pronouns in O 
and T, respectively (the possessive pronouns, mine, 
yours and hers, not shown, are extremely rare in 
the corpus). As can be seen, all pronouns are un-
der-represented in T; for most (bolded), the differ-
ence is significant at p<0.01.  
By contrast, the word the is significantly overre-
presented in T (15.32% in T vs. 13.73% in O; sig-
nificant at p<0.01).  
 
 
word freq O freq T 
I 2.552% 2.148% 
we 2.713% 2.344% 
you 0.479% 0.470% 
he 0.286% 0.115% 
she 0.081% 0.039% 
me 0.148% 0.141% 
us 0.415% 0.320% 
him 0.066% 0.033% 
her 0.091% 0.056% 
my 0.462% 0.345% 
our 0.696% 0.632% 
your 0.119% 0.109% 
his 0.218% 0.123% 
Table 4: Frequency of pronouns  in O and T in the Eu-
roparl corpus. Bold indicates significance at p<0.01. 
 
In Table 5, we consider cohesive markers, 
tagged as adverbs (Schmid, 2004). (These are ad-
verbs that can appear at the beginning of a sen-
tence followed immediately by a comma.)  
 
word freq O freq T 
therefore 0.153% 0.287% 
thus 0.015% 0.041% 
consequently 0.006% 0.014% 
hence 0.007% 0.013% 
accordingly 0.006% 0.011% 
however 0.216% 0.241% 
nevertheless 0.019% 0.045% 
also 0.460% 0.657% 
furthermore 0.012% 0.048% 
moreover 0.008% 0.036% 
indeed 0.098% 0.053% 
actually 0.065% 0.042% 
Table 5: Frequency of cohesive adverbs  in O and T in 
the Europarl corpus. Bold indicates significance at 
p<0.01. 
 
We note that the preponderance of such cohesive 
markers are significantly more frequent in transla-
tions. In fact, we also find that a variety of phrases 
that serve the same purpose as cohesive adverbs, 
such as in fact and as a result are significantly 
more frequent in translationese. 
The general principle underlying these pheno-
mena is subject to speculation. Previous research-
ers have noted the phenomenon of explicitation, 
according to which translators tend to render im-
plicit utterances in the source text into explicit ut-
terances in the target text (Blum-Kulka, 1986, 
Laviosa-Braithwaite, 1998), for example by filling 
out elliptical expressions or adding connectives to 
increase cohesion of the text (Laviosa-Braithwaite, 
1998). It is plausible that the use of cohesive ad-
verbs is an instantiation of this phenomenon. 
With regard to the under-representation of pro-
nouns and the over-representation of the, there are 
a number of possible interpretations. It may be that 
this too is the result of explicitation, in which ana-
phora is resolved by replacing pronouns with noun 
phrases (e.g., the man instead of he). But it also 
might be that this is an example of simplification 
(Laviosa- Braithwaite 1998, Laviosa 2002), ac-
cording to which the translator simplifies the mes-
sage, the language, or both. Related results 
confirming the simplification hypothesis were 
found by Ilisei et al (2010) on Spanish texts. In 
particular, they found that type-to-token ratio (lexi-
cal variety/richness), mean sentence length and 
proportion of grammatical words (lexical densi-
ty/readability) are all smaller in translated texts.  
We note that Van Halteren (2008) and Kurokawa 
et al (2009), who considered lexical features, 
found cultural differences, like over-representation 
of ladies and gentlemen in translated speeches. 
Such differences, while of general interest, are or-
thogonal to our purposes in this paper.  
1322
3.3 Overriding language-specific effects 
We found in Section 2.3 that when we trained in 
one language and tested in another, classification 
succeeded to the extent that the source languages 
used in training and testing, respectively, are re-
lated to each other. In effect, general differences 
between translationese and original English were 
partially overwhelmed by language-specific differ-
ences that held for the training language but not the 
test language. We thus now revisit that earlier ex-
periment, but restrict ourselves to features that dis-
tinguish translationese from original English 
generally.  
To do this, we use the small development corpus 
described in Section 2.1.  We use Bayesian logistic 
regression to learn a classifier to distinguish be-
tween translationese and original English. We se-
lect the 10 highest-weighted function-word 
markers for T and the 10 highest-weighted func-
tion-word markers for O in the development cor-
pus. We then rerun our train-on-source-language-x, 
test-on-source-language-y experiment using this 
restricted set as our feature set. We now find that 
even in the difficult case where we train on Finnish 
and test on another language (or vice versa), we 
succeed at distinguishing translationese from orig-
inal English with accuracy above 80%. This consi-
derably improves the earlier results shown in Table 
3. Thus, a bit of feature engineering facilitates 
learning a good classifier for T vs. O even across 
source languages. 
4 Other Genres and Language Families  
We have found both general and language-specific 
differences between translationese and original 
English in one large corpus. It might be wondered 
whether the phenomena we have found hold in 
other genres and for a completely different set of 
source languages. To test this, we consider a 
second corpus. 
4.1 The IHT corpus  
Our second corpus includes three translated corpo-
ra, each of which is an on-line local supplement to 
the International Herald Tribune (IHT): Kathime-
rini (translated from Greek), Ha?aretz (translated 
from Hebrew), and the JoongAng Daily (translated 
from Korean). In addition, the corpus includes 
original English articles from the IHT. Each of the 
four components contains four different domains 
balanced roughly equally: news (80,000 words), 
arts and leisure (50,000), business and finance 
(50,000), and opinion (50,000) and each covers the 
period from April-September 2004. Each compo-
nent consists of about 230,000 tokens. (Unlike for 
our Europarl corpus, the amount of English text 
available is not equal to the aggregate of the trans-
lated corpora, but rather equal to each of the indi-
vidual corpora.) 
It should be noted that the IHT corpus belongs 
to the writing modality while the Europarl corpus 
belongs to the speaking modality (although possi-
bly post-edited). Furthermore, the source languag-
es (Hebrew, Greek and Korean) in the IHT corpus 
are more disparate than those in the Europarl cor-
pus.  
Our first objective is to confirm that the results 
we obtained earlier on the Europarl corpus hold for 
the IHT corpus as well.  
Perhaps more interestingly, our second objective 
is to see if the gradability phenomenon observed 
earlier (Table 3) generalizes to families of lan-
guages. Our first hypothesis is that a classifier for 
identifying translationese that is trained on Euro-
parl will succeed only weakly to identify transla-
tionese in IHT. But our second hypothesis is that 
there are sufficient general properties of translatio-
nese that cross language families and genres that a 
learned classifier can accurately identify transla-
tionese even on a test corpus that includes both 
corpora, spanning eight disparate languages across 
two distinct genres. 
4.2 Results on IHT corpus 
Running essentially the same experiments as de-
scribed for the Europarl corpus, we obtain the fol-
lowing results.  
First of all, we can determine source language 
with accuracy of 86.5%. This is a somewhat weak-
er result than the 92.7% result obtained on Euro-
parl, especially considering that there are only 
three classes instead of five. The difference is most 
likely due to the fact that the IHT corpus is about 
half the size of the Europarl corpus. Nevertheless, 
it is clear that source language strongly affects 
translationese in this corpus. 
Second, as can be seen in Table 6, we find that 
the gradability phenomenon occurs in this corpus 
as well. Results are strongest when the train and 
1323
test corpora involve the same source language and 
trials involving Korean, the most distant language, 
are somewhat weaker than those across Greek and 
Hebrew. 
 
                  Train 
 Gr He Ko 
Gr 89.8 73.4 64.8 
He 82.0 86.3 65.5 
Ko 73.0 72.5 85.0 
Table 6: Results of learning a T vs. O classifier using 
one source language and testing it using another source 
language 
 
Third, we find in ten-fold cross-validation expe-
riments that we can distinguish translationese from 
original English in the IHT corpus with accuracy 
of 86.3%. Thus, despite the great distance between 
the three source languages in this corpus, general 
differences between translationese and original 
English are sufficient to facilitate reasonably accu-
rate identification of translationese.  
 
4.3 Combining the corpora 
First, we consider whether a classifier learned on 
the Europarl corpus can be used to identify trans-
lationese in the IHT corpus, and vice versa. It 
would be consistent with our findings in Section 
2.3, that we would achieve better than random 
results but not high accuracy, since there are no 
doubt features common to translations from the 
five European languages of Europarl that are dis-
tinct from those of translations from the very dif-
ferent languages in IHT.  
   In fact, we find that training on Europarl and 
testing on IHT yields accuracy of 64.8%, while 
training on IHT and testing on Europarl yields 
accuracy of 58.8%. The weak results reflect both 
differences between the families of source lan-
guages involved in the respective corpora, as well 
as genre differences. Thus, for example, we find 
that of the pronouns shown in Table 4 above, only 
he and his are significantly under-represented in 
translationese in the IHT corpus. Thus, that effect 
is specific either to the genre of Europarl or to the 
European languages considered there.  
   Now, we combine the two corpora and check if 
we can identify translationese across two genres 
and eight languages.  We run the same experiments 
as described above, using 200 texts from each of 
the eight source languages and 1600 non-translated 
English texts, 1000 from Europarl and 600 from 
IHT.  
   In 10-fold cross-validation, we find that we can 
distinguish translationese from non-translated Eng-
lish with accuracy of 90.5%. 
   This shows that there are features of translatio-
nese that cross genres and widely disparate lan-
guages. Thus, for one prominent example, we find 
that, as in Europarl, the word the is over-
represented in translationese in IHT (15.36% in T 
vs. 13.31% in O; significant at p<0.01). In fact, the 
frequencies across corpora are astonishingly con-
sistent. 
   To further appreciate this point, let?s look at the 
frequencies of cohesive adverbs in the IHT corpus. 
    We find essentially, the same pattern in IHT as 
we did in Europarl. The preponderance of cohesive 
adverbs are over-represented in translationese, 
most of them with differences significant at 
p<0.01. Curiously, the word actually is a counter-
example in both corpora. 
5 Conclusions 
We have found that we can learn classifiers that 
determine source language given a translated text, 
as well as classifiers that distinguish translated text 
from non-translated text in the source language. 
These text categorization experiments suggest that 
both source language and the mere fact of being 
word freq O freq T 
therefore 0.011% 0.031% 
thus 0.011% 0.027% 
consequently 0.000% 0.004% 
hence 0.003% 0.007% 
accordingly 0.003% 0.003% 
however 0.078% 0.129% 
nevertheless 0.008% 0.018% 
also 0.305% 0.453% 
furthermore 0.003% 0.011% 
moreover 0.009% 0.008% 
indeed 0.018% 0.024% 
actually 0.032% 0.018% 
Table 7: Frequency of cohesive adverbs in O and T 
in the IHT corpus. Bold indicates significance at 
p<0.01.  
1324
translated play a crucial role in the makeup of a 
translated text.  
    It is important to note that our learned classifiers 
are based solely on function words, so that, unlike 
earlier studies, the differences we find are unlikely 
to include cultural or thematic differences that 
might be artifacts of corpus construction. 
In addition, we find that the exploitability of dif-
ferences between translated texts and non-
translated texts are related to the difference be-
tween source languages: translations from similar 
source languages are different from non-translated 
texts in similar ways. 
Linguists use a variety of methods to quantify 
the extent of differences and similarities between 
languages. For example, Fusco (1990) studies 
translations between Spanish and Italian and con-
siders the impact of structural differences between 
the two languages on translation quality. Studying 
the differences and distance between languages by 
comparing translations into the same language may 
serve as another way to deepen our typological 
knowledge. As we have seen, training on source 
language x and testing on source language y pro-
vides us with a good estimation of the distance be-
tween languages, in accordance with what we find 
in standard works on typology (cf. Katzner, 2002).   
In addition to its intrinsic interest, the finding 
that the distance between languages is directly cor-
related with our ability to distinguish translations 
from a given source language from non-translated 
text is of great importance for several computa-
tional tasks. First, translations can be studied in 
order to shed new light on the differences between 
languages and can bear on attested techniques for 
using cognates to improve machine translation 
(Kondrak & Sherif, 2006). Additionally, given the 
results of our experiments, it stands to reason that 
using translated texts, especially from related 
source languages, will prove beneficial for con-
structing language models and will outperform 
results obtained from non-translated texts. This, 
too, bears on the quality of machine translation. 
Finally, we find that there are general properties 
of translationese sufficiently strong that we can 
identify translationese even in a combined corpus 
that is comprised of eight very disparate languages 
across two distinct genres, one spoken and the oth-
er written. Prominent among these properties is the 
word the, as well as a number of cohesive adverbs, 
each of which is significantly over-represented in 
translated texts.  
References  
Mona Baker. 1993. Corpus linguistics and translation 
studies: Implications and applications. In Gill Francis 
Mona Baker and Elena Tognini Bonelli, editors, Text 
and technology: in honour of John Sinclair, pages 
233-252. John Benjamins, Amsterdam. 
Marco Baroni and Silvia Bernardini. 2006. A new ap-
proach to the study of Translationese: Machine-
learning the difference between original and trans-
lated text. Literary and Linguistic Computing, 
21(3):259-274. 
Shoshan Blum-Kulka. Shifts of cohesion and coherence 
in translation. 1986. In Juliane House and Shoshana 
Blum-Kulka (Eds), Interlingual and Intercultural 
Communication (17-35). T?bingen: G?nter Narr Ver-
lag.  
William Frawley. 1984. Prolegomenon to a theory of 
translation. In William Frawley (ed), Translation. Li-
terary, Linguistic and Philosophical Perspectives 
(179-175). Newark: University of Delaware Press. 
Maria Antonietta Fusco. 1990. Quality in conference 
interpreting between cognate languages: A prelimi-
nary approach to the Spanish-Italian case. The Inter-
preters? Newsletter, 3, 93-97.  
Martin Gellerstam. 1986. Translationese in Swedish 
novels translated from English, in Lars Wollin & 
Hans Lindquist (eds.), Translation Studies in Scandi-
navia (88-95). Lund: CWK Gleerup. 
Iustina Ilisei, Diana Inkpen, Gloria Corpas Pastor, and 
Ruslan Mitkov. Identification of translationese: A 
machine learning approach. In Alexander F. Gel-
bukh, editor, Proceedings of CICLing-2010: Compu-
tational Linguistics and Intelligent Text Processing, 
11th International, volume 6008 of Lecture Notes in 
Computer Science, pages 503-511. Springer, 2010. 
Kenneth Katzner. 2002. The Languages of the World. 
Routledge. 
Grzegorz Kondrak and Tarek Sherif. 2006. Evaluation 
of several phonetic similarity algorithms on the task 
of cognate identification. In Proceedings of the 
Workshop on Linguistic Distances (LD '06). 43-50.  
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 
2009. Automatic detection of translated text and its 
impact on machine translation. In Proceedings of 
MT-Summit XII.  
Sara Laviosa: 1997. How Comparable can 'Comparable 
Corpora' Be?. Target, 9 (2), pp. 289-319.  
1325
Sara Laviosa-Braithwaite. 1998. In Mona Baker (ed.) 
Routledge Encyclopedia of Translation Studies. Lon-
don/New York: Routledge, pp.288-291. 
Sara Laviosa. 2002. Corpus-based Translation Studies. 
Theory, Findings, Applications. Amsterdam/New 
York: Rodopi. 
David Madigan, Alexander Genkin, David D. Lewis and 
Dmitriy Fradkin 2005. Bayesian Multinomial Logis-
tic Regression for Author Identification, In Maxent 
Conference, 509-516. 
James W. Pennebaker, Martha E. Francis, and Roger J. 
Booth. 2001. Linguistic Inquiry and Word Count 
(LIWC): LIWC2001 Manual. Erlbaum Publishers, 
 Mahwah, NJ, USA. 
Helmut Schmid. Probabilistic Part-of-Speech Tagging 
Using Decision Trees. 2004. In Proceedings of In-
ternational Conference on New Methods in Lan-
guage Processing.  
Larry Selinker.1972. Interlanguage. International Re-
view of Applied Linguistics. 10, 209-241. 
Gideon Toury. 1995. Descriptive Translation Studies 
and beyond. John Benjamins, Amsterdam / Philadel-
phia. 
Hans van Halteren. 2008. Source language markers in 
EUROPARL translations. In COLING '08: Proceed-
ings of the 22nd International Conference on Compu-
tational Linguistics, pages 937-944. 
1326
Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 279?287,
Atlanta, Georgia, June 13 2013. c?2013 Association for Computational Linguistics
Identifying the L1 of non-native writers: the CMU-Haifa system
Yulia Tsvetkov? Naama Twitto? Nathan Schneider? Noam Ordan?
Manaal Faruqui? Victor Chahuneau? Shuly Wintner? Chris Dyer?
?Language Technologies Institute
Carnegie Mellon University
Pittsburgh, PA
cdyer@cs.cmu.edu
?Department of Computer Science
University of Haifa
Haifa, Israel
shuly@cs.haifa.ac.il
Abstract
We show that it is possible to learn to identify, with
high accuracy, the native language of English test
takers from the content of the essays they write.
Our method uses standard text classification tech-
niques based on multiclass logistic regression, com-
bining individually weak indicators to predict the
most probable native language from a set of 11 pos-
sibilities. We describe the various features used for
classification, as well as the settings of the classifier
that yielded the highest accuracy.
1 Introduction
The task we address in this work is identifying the
native language (L1) of non-native English (L2) au-
thors. More specifically, given a dataset of short
English essays (Blanchard et al, 2013), composed
as part of the Test of English as a Foreign Lan-
guage (TOEFL) by authors whose native language is
one out of 11 possible languages?Arabic, Chinese,
French, German, Hindi, Italian, Japanese, Korean,
Spanish, Telugu, or Turkish?our task is to identify
that language.
This task has a clear empirical motivation. Non-
native speakers make different errors when they
write English, depending on their native language
(Lado, 1957; Swan and Smith, 2001); understand-
ing the different types of errors is a prerequisite for
correcting them (Leacock et al, 2010), and systems
such as the one we describe here can shed interest-
ing light on such errors. Tutoring applications can
use our system to identify the native language of
students and offer better-targeted advice. Forensic
linguistic applications are sometimes required to de-
termine the L1 of authors (Estival et al, 2007b; Es-
tival et al, 2007a). Additionally, we believe that the
task is interesting in and of itself, providing a bet-
ter understanding of non-native language. We are
thus equally interested in defining meaningful fea-
tures whose contribution to the task can be linguis-
tically interpreted. Briefly, our features draw heav-
ily on prior work in general text classification and
authorship identification, those used in identifying
so-called translationese (Volansky et al, forthcom-
ing), and a class of features that involves determin-
ing what minimal changes would be necessary to
transform the essays into ?standard? English (as de-
termined by an n-gram language model).
We address the task as a multiway text-
classification task; we describe our data in ?3 and
classification model in ?4. As in other author attri-
bution tasks (Juola, 2006), the choice of features for
the classifier is crucial; we discuss the features we
define in ?5. We report our results in ?6 and con-
clude with suggestions for future research.
2 Related work
The task of L1 identification was introduced by Kop-
pel et al (2005a; 2005b), who work on the Inter-
national Corpus of Learner English (Granger et al,
2009), which includes texts written by students from
5 countries, Russia, the Czech Republic, Bulgaria,
France, and Spain. The texts range from 500 to
850 words in length. Their classification method
is a linear SVM, and features include 400 standard
function words, 200 letter n-grams, 185 error types
and 250 rare part-of-speech (POS) bigrams. Ten-
279
fold cross-validation results on this dataset are 80%
accuracy.
The same experimental setup is assumed by Tsur
and Rappoport (2007), who are mostly interested
in testing the hypothesis that an author?s choice of
words in a second language is influenced by the
phonology of his or her L1. They confirm this hy-
pothesis by carefully analyzing the features used by
Koppel et al, controlling for potential biases.
Wong and Dras (2009; 2011) are also motivated
by a linguistic hypothesis, namely that syntactic er-
rors in a text are influenced by the author?s L1.
Wong and Dras (2009) analyze three error types sta-
tistically, and then add them as features in the same
experimental setup as above (using LIBSVM with a
radial kernel for classification). The error types are
subject-verb disagreement, noun-number disagree-
ment and misuse of determiners. Addition of these
features does not improve on the results of Kop-
pel et al. Wong and Dras (2011) further extend
this work by adding as features horizontal slices of
parse trees, thereby capturing more syntactic struc-
ture. This improves the results significantly, yielding
78% accuracy compared with less than 65% using
only lexical features.
Kochmar (2011) uses a different corpus, the Cam-
bridge Learner Corpus, in which texts are 200-400
word long, and are authored by native speakers of
five Germanic languages (German, Swiss German,
Dutch, Swedish and Danish) and five Romance lan-
guages (French, Italian, Catalan, Spanish and Por-
tuguese). Again, SVMs are used as the classification
device. Features include POS n-grams, character n-
grams, phrase-structure rules (extracted from parse
trees), and two measures of error rate. The classi-
fier is evaluated on its ability to distinguish between
pairs of closely-related L1s, and the results are usu-
ally excellent.
A completely different approach is offered by
Brooke and Hirst (2011). Since training corpora for
this task are rare, they use mainly L1 (blog) cor-
pora. Given English word bigrams ?e1, e2?, they try
to assess, for each L1, how likely it is that an L1 bi-
gram was translated literally by the author, resulting
in ?e1, e2?. Working with four L1s (French, Span-
ish, Chinese, and Japanese), and evaluating on the
International Corpus of Learner English, accuracy is
below 50%.
3 Data
Our dataset in this work consists of TOEFL essays
written by speakers of eleven different L1s (Blan-
chard et al, 2013), distributed as part of the Na-
tive Language Identification Shared Task (Tetreault
et al, 2013). The training data consists of 1000
essays from each native language. The essays are
short, consisting of 10 to 20 sentences each. We
used the provided splits of 900 documents for train-
ing and 100 for development. Each document is an-
notated with the author?s English proficiency level
(low, medium, high) and an identification (1 to 8) of
the essay prompt. All essays are tokenized and split
into sentences. In table 1 we provide some statistics
on the training corpora, listed by the authors? profi-
ciency level. All essays were tagged with the Stan-
ford part-of-speech tagger (Toutanova et al, 2003).
We did not parse the dataset.
Low Medium High
# Documents 1,069 5,366 3,456
# Tokens 245,130 1,819,407 1,388,260
# Types 13,110 37,393 28,329
Table 1: Training set statistics.
4 Model
For our classification model we used the creg re-
gression modeling framework to train a 11-class lo-
gistic regression classifier.1 We parameterize the
classifier as a multiclass logistic regression:
p?(y | x) =
exp
?
j ? jh j(x, y)
Z?(x)
,
where x are documents, h j(?) are real-valued feature
functions of the document being classified, ? j are the
corresponding weights, and y is one of the eleven L1
class labels. To train the parameters of our model,
we minimized the following objective,
L = ?
`2 reg.
?????
j
?2j ?
?
{(xi,yi)}
|D|
i=1
(
log likelihood
?          ??          ?
log p?(yi | xi) +
?Ep?(y? |xi) log p?(y
? | xi)
?                      ??                      ?
?conditional entropy
)
,
1https://github.com/redpony/creg
280
which combines the negative log likelihood of the
training dataset D, an `2 (quadratic) penalty on the
magnitude of ? (weighted by ?), and the negative en-
tropy of the predictive model (weighted by ?). While
an `2 weight penalty is standard in regression prob-
lems like this, we found that the the additional en-
tropy term gave more reliable results. Intuitively,
the entropic regularizer encourages the model to re-
main maximally uncertain about its predictions. In
the metaphor of ?maximum entropy?, the entropic
prior finds a solution that has more entropy than the
?maximum? model that is compatible with the con-
straints.
The objective cannot be minimized in closed
form, but it does have a unique minimum and
is straightforwardly differentiable, so we used L-
BFGS to find the optimal weight settings (Liu et al,
1989).
5 Feature Overview
We define a large arsenal of features, our motivation
being both to improve the accuracy of classification
and to be able to interpret the characteristics of the
language produced by speakers of different L1s.
While some of the features were used in prior
work (?2), we focus on two broad novel categories
of features: those inspired by the features used
to identify translationese by Volansky et al (forth-
coming) and those extracted by automatic statisti-
cal ?correction? of the essays. Refer to figure 1 to
see the set of features and their values that were ex-
tracted from an example sentence.
POS n-grams Part-of-speech n-grams were used in
various text-classification tasks.
Prompt Since the prompt contributes information
on the domain, it is likely that some words (and,
hence, character sequences) will occur more fre-
quently with some prompts than with others. We
therefore use the prompt ID in conjunction with
other features.
Document length The number of tokens in the text
is highly correlated with the author?s level of flu-
ency, which in turn is correlated with the author?s
L1.
Pronouns The use of pronouns varies greatly
among different authors. We use the same list
of 25 English pronouns that Volansky et al (forth-
coming) use for identifying translationese.
Punctuation Similarly, different languages use
punctuation differently, and we expect this to taint
the use of punctuation in non-native texts. Of
course, character n-grams subsume this feature.
Passives English uses passive voice more fre-
quently than other languages. Again, the use of
passives in L2 can be correlated with the author?s
L1.
Positional token frequency The choice of the first
and last few words in a sentence is highly con-
strained, and may be significantly influenced by
the author?s L1.
Cohesive markers These are 40 function words
(and short phrases) that have a strong discourse
function in texts (however, because, in fact,
etc.). Translators tend to spell out implicit utter-
ances and render them explicitly in the target text
(Blum-Kulka, 1986). We use the list of Volansky
et al (forthcoming).
Cohesive verbs This is a list of manually compiled
verbs that are used, like cohesive markers, to spell
out implicit utterances (indicate, imply, contain,
etc.).
Function words Frequent tokens, which are mostly
function words, have been used successfully for
various text classification tasks. Koppel and Or-
dan (2011) define a list of 400 such words, of
which we only use 100 (using the entire list was
not significantly different). Note that pronouns
are included in this list.
Contextual function words To further capitalize
on the ability of function words to discriminate,
we define pairs consisting of a function word from
the list mentioned above, along with the POS tag
of its adjacent word. This feature captures pat-
terns such as verbs and the preposition or particle
immediately to their right, or nouns and the deter-
miner that precedes them. We also define 3-grams
consisting of one or two function words and the
POS tag of the third word in the 3-gram.
Lemmas The content of the text is not considered a
good indication of the author?s L1, but many text
categorization tasks use lemmas (more precisely,
the stems produced by the tagger) as features ap-
proximating the content.
Misspelling features Learning to perceive, pro-
duce, and encode non-native phonemic contrasts
281
Firstly the employers live more savely because they are going to have more money to spend for luxury .
Presence Considered alternatives/edits
Characters
"CHAR_l_y_ ": log 2 + 1
"CharPrompt_P5_g_o_i": log 1 + 1
"MFChar_e_ ": log 1 + 1
"Punc_period": log 1 + 1
"DeleteP_p_.": 1.0
"InsertP_p_,": 1.0
"MID:SUBST:v:f": log 1 + 1
"SUBST:v:f": log 1 + 1
Words
"DocLen_": log 19 + 1
"MeanWordRank": 422.6
"CohMarker_because": log 1 + 1
"MostFreq_have": log 1 + 1
"PosToken_last_luxury": log 1 + 1
"Pronouns_they": log 1 + 1
"MSP:safely": log 1 + 1
"Match_p_to": 0.5
"Delete_p_to": 0.5
"Delete_p_are": 1.0
"Delete_p_because": 1.0
"Delete_p_for": 1.0
POS "POS
_VBP_VBG_TO": log 1 + 1
"POS_p_VBP_VBG_TO": 0.059
Words + POS "VBP
_VBG_to": log 1 + 1
"FW__more RB": log 1 + 1
Figure 1: Some of the features extracted for an L1 German sentence.
is extremely difficult for L2 learners (Hayes-Harb
and Masuda, 2008). Since English?s orthogra-
phy is largely phonemic?even if it is irregular
in many places, we expect leaners whose na-
tive phoneme contrasts are different from those
of English to make characteristic spelling errors.
For example, since Japanese and Korean lack a
phonemic /l/-/r/ contrast, we expect native speak-
ers of those languages to be more likely to make
spelling errors that confuse l and r relative to
native speakers of languages such as Spanish in
which that pair is contrastive. To make this in-
formation available to our model, we use a noisy
channel spelling corrector (Kernighan, 1990) to
identify and correct misspelled words in the train-
ing and test data. From these corrections, we ex-
tract minimal edit features that show what inser-
tions, deletions, substitutions and joinings (where
two separate words are written merged into a sin-
gle orthographic token) were made by the author
of the essay.
Restored tags We focus on three important token
classes defined above: punctuation marks, func-
tion words and cohesive verbs. We first remove
words in these classes from the texts, and then
recover the most likely hidden tokens in a se-
quence of words, according to an n-gram lan-
guage model trained on all essays in the training
corpus corrected with a spell checker and con-
taining both words and hidden tokens. This fea-
ture should capture specific words or punctuation
marks that are consistently omitted (deletions),
or misused (insertions, substitutions). To restore
hidden tokens we use the hidden-ngram util-
ity provided in SRI?s language modeling toolkit
(Stolcke, 2002).
Brown clusters (Brown et al, 1992) describe an al-
gorithm that induces a hierarchical clustering of
a language?s vocabulary based on each vocabu-
lary item?s tendency to appear in similar left and
right contexts in a training corpus. While origi-
nally developed to reduce the number of parame-
ters required in n-gram language models, Brown
clusters have been found to be extremely effective
as lexical representations in a variety of regres-
sion problems that condition on text (Koo et al,
2008; Turian et al, 2010; Owoputi et al, 2013).
Using an open-source implementation of the al-
gorithm,2 we clustered 8 billion words of English
into 600 classes.3 We included log counts of all
4-grams of Brown clusters that occurred at least
100 times in the NLI training data.
5.1 Main Features
We use the following four feature types as the base-
line features in our model. For features that are sen-
sitive to frequency, we use the log of the (frequency-
plus-one) as the feature?s value. Table 2 reports the
accuracy of using each feature type in isolation (with
2https://github.com/percyliang/brown-cluster
3http://www.ark.cs.cmu.edu/cdyer/en-600/
cluster_viewer.html
282
Feature Accuracy (%)
POS 55.18
FreqChar 74.12
CharPrompt 65.09
Brown 72.26
DocLen 11.81
Punct 27.41
Pron 22.81
Position 53.03
PsvRatio 12.26
CxtFxn (bigram) 62.79
CxtFxn (trigram) 62.32
Misspell 37.29
Restore 47.67
CohMark 25.71
CohVerb 22.85
FxnWord 42.47
Table 2: Independent performance of feature types de-
tailed in ?5.1, ?5.2 and ?5.3. Accuracy is averaged over
10 folds of cross-validation on the training set.
10-fold cross-validation on the training set).
POS Part-of-speech n-grams. Features were ex-
tracted to count every POS 1-, 2-, 3- and 4-gram
in each document.
FreqChar Frequent character n-grams. We exper-
imented with character n-grams: To reduce the
number of parameters, we removed features only
those character n-grams that are observed more
than 5 times in the training corpus, and n ranges
from 1 to 4. High-weight features include:
TUR:<Turk>; ITA:<Ital>; JPN:<Japa>.
CharPrompt Conjunction of the character n-gram
features defined above with the prompt ID.
Brown Substitutions, deletions and insertions
counts of Brown cluster unigrams and bigrams in
each document.
The accuracy of the classifier on the development set
using these four feature types is reported in table 3.4
5.2 Additional Features
To the basic set of features we now add more spe-
cific, linguistically-motivated features, each adding
a small number of parameters to the model. As
above, we indicate the accuracy of each feature type
in isolation.
4For experiments in this paper combining multiple types of
features, we used Jonathan Clark?s workflow management tool,
ducttape (https://github.com/jhclark/ducttape).
Feature Group # Params Accuracy (%) `2
POS 540,947 55.18 1.0
+ FreqChar 1,036,871 79.55 1.0
+ CharPrompt 2,111,175 79.82 1.0
+ Brown 5,664,461 81.09 1.0
Table 3: Dev set accuracy with main feature groups,
added cumulatively. The number of parameters is always
a multiple of 11 (the number of classes). Only `2 regular-
ization was used for these experiments; the penalty was
tuned on the dev set as well.
DocLen Document length in tokens.
Punct Counts of each punctuation mark.
Pron Counts of each pronoun.
Position Positional token frequency. We use the
counts for the first two and last three words be-
fore the period in each sentence as features. High-
weight features for the second word include:
ARA:2<,>; CHI:2<is>; HIN:2<can>.
PsvRatio The proportion of passive verbs out of all
verbs.
CxtFxn Contextual function words. High-weight
features include: CHI:<some JJ>;
HIN:<as VBN>.
Misspell Spelling correction edits. Features
included substitutions, deletions, insertions,
doubling of letters and missing doublings of
letters, and splittings (alot?a lot), as well as the
word position where the error occurred.
High-weight features include: ARA:DEL<e>,
ARA:INS<e>, ARA:SUBST<e>/<i>;
GER:SUBST<z>/<y>; JPN:SUBST<l>/<r>,
JPN:SUBST<r>/<l>; SPA:DOUBLE<s>,
SPA:MID_INS<s>, SPA:INS<s>.
Restore Counts of substitutions, deletions and
insertions of predefined tokens that we restored
in the texts. High-weight features include:
CHI:DELWORD<do>; GER:DELWORD<on>;
ITA:DELWORD<be>
Table 4 reports the empirical improvement that each
of these brings independently when added to the
main features (?5.1).
5.3 Discarded Features
We also tried several other feature types that did not
improve the accuracy of the classifier on the devel-
opment set.
CohMark Counts of each cohesive marker.
283
Feature Group # Params Accuracy (%) `2
main + Position 6,153,015 81.00 1.0
main + PsvRatio 5,664,472 81.00 1.0
main 5,664,461 81.09 1.0
main + DocLen 5,664,472 81.09 1.0
main + Pron 5,664,736 81.09 1.0
main + Punct 5,664,604 81.09 1.0
main + Misspell 5,799,860 81.27 5.0
main + Restore 5,682,589 81.36 5.0
main + CxtFxn 7,669,684 81.73 1.0
Table 4: Dev set accuracy with main features plus addi-
tional feature groups, added independently. `2 regulariza-
tion was tuned as in table 3 (two values, 1.0 and 5.0, were
tried for each configuration; more careful tuning might
produce slightly better accuracy). Results are sorted by
accuracy; only three groups exhibited independent im-
provements over the main feature set.
CohVerb Counts of each cohesive verb.
FxnWord Counts of function words. These features
are subsumed by the highly discriminative CxtFxn
features.
6 Results
The full model that we used to classify the test set
combines all features listed in table 4. Using all
these features, the accuracy on the development set
is 84.55%, and on the test set it is 81.5%. The values
for ? and ? were tuned to optimize development set
performance, and found to be ? = 5, ? = 2.
Table 5 lists the confusion matrix on the test set,
as well as precision, recall and F1-score for each L1.
The largest error type involved predicting Telugu
when the true label was Hindi, which happened 18
times. This error is unsurprising since many Hindi
and Telugu speakers are arguably native speakers of
Indian English.
Production of L2 texts, not unlike translating from
L1 to L2, involves a tension between the impos-
ing models of L1 (and the source text), on the one
hand, and a set of cognitive constraints resulting
from the efforts to generate the target text, on the
other. The former is called interference in Trans-
lation Studies (Toury, 1995) and transfer in second
language acquisition (Selinker, 1972). Volansky et
al. (forthcoming) designed 32 classifiers to test the
validity of the forces acting on translated texts, and
found that features sensitive to interference consis-
tently yielded the best performing classifiers. And
indeed, in this work too, we find fingerprints of the
source language are dominant in the makeup of L2
texts. The main difference, however, between texts
translated by professionals and the texts we address
here, is that more often than not professional trans-
lators translate into their mother tongue, whereas L2
writers write out of their mother tongue by defini-
tion. So interference is ever more exaggerated in
this case, for example, also phonologically (Tsur and
Rappoport, 2007).
We explore the effects of interference by analyz-
ing several patterns we observe in the features. Our
classifier finds that the character sequence alot is
overrepresented in Arabic L2 texts. Arabic has no
indefinite article and we speculate that Arabic speak-
ers conceive a lot as a single word; the Arabic equiv-
alent for a lot is used adverbially like an -ly suffix
in English. For the same reason, another promi-
nent feature is a missing definite article before nouns
and adjectives. Additionally, Arabic, being an Ab-
jad language, rarely indicates vowels, and indeed we
find many missing e?s and i?s in the texts of Arabic
speakers. Phonologically, because Arabic conflates
/I/ and /@/ into /i/ (at least in Modern Standard Ara-
bic), we see that many e?s are indeed substituted for
i?s in these texts.
We find that essays that contain hyphens are more
likely to be from German authors. We again find
evidence of interference from the native language
here. First, relative clauses are widely used in Ger-
man, and we see this pattern in L2 English of L1
German speakers. For example, any given rational
being ? let us say Immanual Kant ? we find that.
Another source of extra hyphens stems from com-
pounding convention. So, for example, we find well-
known, community-help, spare-time, football-club,
etc. Many of these reflect an effort to both connect
and separate connected forms in the original (e.g.,
Fussballklub, which in English would be more natu-
rally rendered as football club). Another unexpected
feature of essays by native Germans is a frequent
substitution of the letter y for z and vice versa. We
suspect this owes to their switched positions on Ger-
man keyboards.
Lexical item frequency also provides clues to the
L1 of the essay writers. The word that occurs more
frequently in the texts of German L1 speakers. We
284
true? ARA CHI FRE GER HIN ITA JPN KOR SPA TEL TUR Precision (%) Recall (%) F1 (%)
ARA 80 0 2 1 3 4 1 0 4 2 3 80.8 80.0 80.4
CHI 3 80 0 1 1 0 6 7 1 0 1 88.9 80.0 84.2
FRE 2 2 81 5 1 2 1 0 3 0 3 86.2 81.0 83.5
GER 1 1 1 93 0 0 0 1 1 0 2 87.7 93.0 90.3
HIN 2 0 0 1 77 1 0 1 5 9 4 74.8 77.0 75.9
ITA 2 0 3 1 1 87 1 0 3 0 2 82.1 87.0 84.5
JPN 2 1 1 2 0 1 87 5 0 0 1 78.4 87.0 82.5
KOR 1 5 2 0 1 0 9 81 1 0 0 80.2 81.0 80.6
SPA 2 0 2 0 1 8 2 1 78 1 5 77.2 78.0 77.6
TEL 0 1 0 0 18 1 2 1 1 73 3 85.9 73.0 78.9
TUR 4 0 2 2 0 2 2 4 4 0 80 76.9 80.0 78.4
Table 5: Official test set confusion matrix with the full model. Accuracy is 81.5%.
hypothesize that in English it is optional in rela-
tive clauses whereas in German it is not, so Ger-
man speakers are less comfortable using the non-
obligatory form. Also, often is over represented. We
hypothesize that since it is cognate of German oft, it
is not cognitively expensive to retrieve it. We find
many times?a literal translation of muchas veces?
in Spanish essays.
Other informative features that reflect L1 features
include frequent misspellings involving confusions
of l and r in Japanese essays. More mysteriously,
the characters r and s are misused in Chinese and
Spanish, respectively. The word then is dominant
in the texts of Hindi speakers. Finally, it is clear
that authors refer to their native cultures (and, conse-
quently, native languages and countries); the strings
Turkish, Korea, and Ita were dominant in the texts of
Turkish, Korean and Italian native speakers, respec-
tively.
7 Discussion
We experimented with different classifiers and a
large set of features to solve an 11-way classifica-
tion problem. We hope that studying this problem
will improve to facilitate human assessment, grad-
ing, and teaching of English as a second language.
While the core features used are sparse and sensitive
to lexical and even orthographic features of the writ-
ing, many of them are linguistically informed and
provide insight into how L1 and L2 interact.
Our point of departure was the analogy between
translated texts as a genre in its own and L2 writ-
ers as pseudo translators, relying heavily on their
mother tongue and transferring their native models
to a second language. In formulating our features,
we assumed that like translators, L2 writers will
write in a simplified manner and overuse explicit
markers. Although this should be studied vis-?-vis
comparable outputs of mother tongue writers in En-
glish, we observe that the best features of our clas-
sifiers are of the ?interference? type, i.e. phonolog-
ical, morphological and syntactic in nature, mostly
in the form of misspelling features, restoration tags,
punctuation and lexical and syntactic modeling.
We would like to stress that certain features indi-
cating a particular L1 have no bearing on the quality
of the English produced. This has been discussed
extensively in Translation Studies (Toury, 1995),
where interference is observed by the overuse or un-
deruse of certain features reflecting the typological
differences between a specific pair of languages, but
which is still within grammatical limits. For exam-
ple, the fact that Italian native speakers favor the
syntactic sequence of determiner + adjective + noun
(e.g., a big risk or this new business) has little pre-
scriptive value for teachers.
A further example of how L2 quality and the
ability to predict L1 are uncorrelated, we noted
that certain L2 writers often repeat words appear-
ing in their essay prompts, and including informa-
tion about whether the writer was reusing prompt
words improved classification accuracy. We suggest
this reflects different educational backgrounds. This
feature says nothing about the quality of the text, just
as the tendency of Korean and Italian writers to men-
tion their home country more often does not.
285
Acknowledgments
This research was supported by a grant from the Is-
raeli Ministry of Science and Technology.
References
Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife
Cahill, and Martin Chodorow. 2013. TOEFL11: A
corpus of non-native English. Technical report, Edu-
cational Testing Service.
Shoshana Blum-Kulka. 1986. Shifts of cohesion and co-
herence in translation. In Juliane House and Shoshana
Blum-Kulka, editors, Interlingual and intercultural
communication Discourse and cognition in translation
and second language acquisition studies, volume 35,
pages 17?35. Gunter Narr Verlag.
Julian Brooke and Graeme Hirst. 2011. Native language
detection with ?cheap? learner corpora. In Conference
of Learner Corpus Research (LCR2011), Louvain-la-
Neuve, Belgium. Presses universitaires de Louvain.
Peter F. Brown, Peter V. de Souza, Robert L. Mercer, Vin-
cent J. Della Pietra, and Jenifer C. Lai. 1992. Class-
based n-gram models of natural language. Computa-
tional Linguistics, 18(4).
Dominique Estival, Tanja Gaustad, Son Bao Pham, Will
Radford, and Ben Hutchinson. 2007a. Author profil-
ing for English emails. In Proc. of PACLING, pages
263?272, Melbourne, Australia.
Dominique Estival, Tanja Gaustad, Son Bao Pham, Will
Radford, and Ben Hutchinson. 2007b. TAT: An author
profiling tool with application to Arabic emails. In
Proc. of the Australasian Language Technology Work-
shop, pages 21?30, Melbourne, Australia, December.
Sylviane Granger, Estelle Dagneaux, Fanny Meunier,
and Magali Paquot. 2009. International Corpus of
Learner English. Presses universitaires de Louvain,
Louvain-la-Neuve.
Rachel Hayes-Harb and Kyoko Masuda. 2008. Devel-
opment of the ability to lexically encode novel second
language phonemic contrasts. Second Language Re-
search, 24(1):5?33.
Patrick Juola. 2006. Authorship attribution. Founda-
tions and Trends in Information Retrieval, 1(3):233?
334.
Mark D. Kernighan. 1990. A spelling correction pro-
gram based on a noisy channel model. In Proc. of
COLING.
Ekaterina Kochmar. 2011. Identification of a writer?s na-
tive language by error analysis. Master?s thesis, Uni-
versity of Cambridge.
Terry Koo, Xavier Carreras, and Michael Collins. 2008.
Simple semi-supervised dependency parsing. In Proc.
of ACL.
Moshe Koppel and Noam Ordan. 2011. Translationese
and its dialects. In Proc. of ACL-HLT, pages 1318?
1326, Portland, Oregon, USA, June. Association for
Computational Linguistics.
Moshe Koppel, Jonathan Schler, and Kfir Zigdon. 2005a.
Automatically determining an anonymous author?s na-
tive language. Intelligence and Security Informatics,
pages 41?76.
Moshe Koppel, Jonathan Schler, and Kfir Zigdon. 2005b.
Determining an author?s native language by mining
a text for errors. In Proc. of KDD, pages 624?628,
Chicago, IL. ACM.
Robert Lado. 1957. Linguistics across cultures: applied
linguistics for language teachers. University of Michi-
gan Press, Ann Arbor, Michigan, June.
Claudia Leacock, Martin Chodorow, Michael Gamon,
and Joel Tetreault. 2010. Automated Grammatical
Error Detection for Language Learners. Morgan and
Claypool.
Dong C. Liu, Jorge Nocedal, Dong C. Liu, and Jorge No-
cedal. 1989. On the limited memory BFGS method
for large scale optimization. Mathematical Program-
ming B, 45(3):503?528.
Olutobi Owoputi, Brendan O?Connor, Chris Dyer, Kevin
Gimpel, Nathan Schneider, and Noah A. Smith. 2013.
Improved part-of-speech tagging for online conversa-
tional text with word clusters. In Proc. of NAACL.
Larry Selinker. 1972. Interlanguage. International
Review of Applied Linguistics in Language Teaching,
10(1?4):209?232.
Andreas Stolcke. 2002. SRILM?an extensible lan-
guage modeling toolkit. In Procedings of Interna-
tional Conference on Spoken Language Processing,
pages 901?904.
Michael Swan and Bernard Smith. 2001. Learner En-
glish: A Teacher?s Guide to Interference and Other
Problems. Cambridge Handbooks for Language
Teachers. Cambridge University Press.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013.
A report on the first native language identification
shared task. In Proc. of the Eighth Workshop on Inno-
vative Use of NLP for Building Educational Applica-
tions, Atlanta, GA, USA, June. Association for Com-
putational Linguistics.
Gideon Toury. 1995. Descriptive Translation Studies
and beyond. John Benjamins, Amsterdam / Philadel-
phia.
Kristina Toutanova, Dan Klein, Christopher D. Manning,
and Yoram Singer. 2003. Feature-rich part-of-speech
tagging with a cyclic dependency network. In Proc.
of HLT-NAACL, pages 173?180, Edmonton, Canada,
June. Association for Computational Linguistics.
286
Oren Tsur and Ari Rappoport. 2007. Using classifier fea-
tures for studying the effect of native language on the
choice of written second language words. In Proc. of
the Workshop on Cognitive Aspects of Computational
Language Acquisition, pages 9?16, Prague, Czech Re-
public, June. Association for Computational Linguis-
tics.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010.
Word representations: A simple and general method
for semi-supervised learning. In Proc. of ACL.
Vered Volansky, Noam Ordan, and Shuly Wintner. forth-
coming. On the features of translationese. Literary
and Linguistic Computing.
Sze-Meng Jojo Wong and Mark Dras. 2009. Contrastive
analysis and native language identification. In Proc.
of the Australasian Language Technology Association
Workshop, pages 53?61, Sydney, Australia, December.
Sze-Meng Jojo Wong and Mark Dras. 2011. Exploiting
parse structures for native language identification. In
Proc. of EMNLP, pages 1600?1610, Edinburgh, Scot-
land, UK., July. Association for Computational Lin-
guistics.
287