Statistical Machine Translation using Moses

-Nirav Shah and Sumit Goswami

Machine translation is a growing research area due to its application in

providing fast and meaningful translation of text and speech from one language

to another language. It can be done based either on rules where rules are

applied to convert one language into another language or through statistical

machine translation (SMT). Statistical machine translation uses statistical

methods to translate with the help of parallel corpus. It uses word based

translation method or phrase based translation. The success of machine

translation system depends on how well one language's words are aligned with

another language's words. Statistical machine translation system allows training

of translation model for any language. The requirement for doing statistical

machine translation is a bi-lingual parallel corpus. There are various types of

translation methods which are used like factored, beam-search and phrase based.

Various machine translation tools are available like Apertium (GNU license),

OpenLogos which is the open source version of Logos Machine Translation System,

SYSTRAN which is one of the oldest Machine Translation company and Moses (GNU

General Public License).

Moses is a phrase based machine translation tool for converting one language

to another language. Technical details regarding this are available on Moses

website, www.statmt.org/moses. In this article, we will give a short

step-by-step process for converting text from one language to another, for

example Hindi to English using Moses machine translation tool.

Parallel Corpus

Prepare Parallel Corpus for source (English) and target (Hindi) language

which will be used for training the language model. This corpus can be prepared

from your existing translated data or can be obtained from Internet free of cost

or with a price for e.g. EMILLE corpus, a free version of which is available for

research purposes. Similarly, a parallel corpus with a smaller size is required

for tuning as well as for testing the model.

Data preparation

The parallel corpus is converted to a format that is suitable for Giza++.

Giza is an open source tool based on IBM model and is used for word alignment.

Before training Moses the following software should be downloaded:

• SRILM < http:// tinyurl. com/ dx8m5m > - This is the tool developed by
  
  Stanford research institute for building statistical language model.
• GIZA++ < http:// tinyurl. com/ cdem45 > or < http:// giza-pp. googlecode.
  
  com/ > — This tool is developed by Franz Josef Och. This tool implements
  
  different models like HMM and also performs word alignment.
• MKCLS <http://tinyurl.com/ c83mpx > or < http://www.fjoch. com/mkcls. html
  
  > - This tool is also developed by Franz Josef Och and used for training word
  
  classes which is used in SMT model. For MKCLS and GIZA++ latest GNU compiler
  
  is required.
• Moses < http://sourceforge.net/ projects/mosesdecoder >
• Additional scripts < http://tinyurl. com/ cp8xz7 > - These are the
  
  additional scripts for Moses training and tuning.

For this article we are keeping /usr/home/PCQ/demo as the root directory for

installation. The steps given below are relative to this root directory.

Getting started

Create a directory 'srilm' in root directory, move downloaded srilm tar file to

this directory, extract and run the 'make file'.

Then move GIZA++ tar file to the root directory and extract. This process will

create a directory GIZA++-v2. Run the make file inside GIZA++-v2 and thereafter

again run make with argument snt2cooc.out from the same directory. This will

produce GIZA++ and snt2cooc.out. Create a directory 'bin' in root directory and

copy these two files to 'bin'.

Now move the mkcls tar file to root directory and extract. This will make

mkcls-v2 directory in which you should run the make file. This will produce

mkcls file which is copied to 'bin'.

Create a directory named Moses under root and copy Moses tar file to this

directory and extract it. Now change to Moses directory and execute regenerate—makefiles.sh.

and thereafter execute configure script with option ./configure --with-srilm=/usr/home/PCQ/demo/srilm

and then run 'make — J 4'.

Now move to 'bin' directory under root and create 'moses-scripts' directory

in it. Thereafter move to moses/scripts directory under root and run make

release. Move scripts tar file to root directory and extract it. This process

will complete the set up of Moses.

Training the Translator

Now we can process our parallel corpus. Create directories

working-dir/corpus in root directory and copy the English and Hindi corpus in

it. Filter out long sentences and lowercase the data. We do not require to

lowercase Hindi data because it is in wx format. Now we create a directory 'lm'

inside 'working-dir' directory to build the language model. Again we need to

lowercase the English language data. These two steps creates English language

model data. Now we can build the model using SRILM. After this process, our

language model is ready and now we can train our model. After training, for

better performance we can tune it also. However, tuning is not mandatory.

For translating, English language sentence can be given as input echo 'This

is a small House.' | /usr/home/PCQ/ demo/moses/moses-cmd/src/moses -f moses. ini

> out.txt.

We can find its translated Hindi language sentence by 'cat out.txt' which

will contain this: yaha eka CotA AvAsagqha Hai.

Conclusion

If the corpus is large in size then it requires huge memory, at least 2 GB

for building the translation model. Besides, a few steps in the process can take

few minutes to few hours depending on the processing power, memory and size of

training corpus. The model given here is base line model and research is going

on to improve the results of translation. If the corpus is large enough, then

the trained model will have a high translation accuracy.