Language Identifier

A basic web app to Identify the language using Character trigrams and neural network (98.6% test accuracy)

How to Use

• Clone this repo
• Run server.py from your terminal
• Copy the url in the output in your browser to run the web app locally
• Give a language string you need to identify in the input field
• Check the obtained Result !

How this works ??

• The string given to the input form is passed on to a pretrained Neural network model (model_1.h5) after cleaning.
• The model outputs a set of probabilities from which the most probable language is returned to the web application as the prediction
• Checkout my 5 min discription of the project on youtube here... !

N-gram based Language Detection Model

• An N-gram is an N-character slice of a longer string. Although in the literature the term can include the notion of any co-occurring set of characters in a string (e.g., an N-gram made up of the first and third character of a word), in this project we use the term for contiguous slices only. Typically, we will be using

  character trigrams

• eg; the word TEXT will become _TE, TEX, EXT, XT_, T_ _ (begining and end of the word is padded with white space)

• The idea is to identify Character Trigrams unique to each language and use them to classify the languages

• The model was trained on a subset of Tatoeba dataset which contains over 300 languages.

• Currently the model is able to identify dutch, finnish, polish, lituanian, czech, swedish, arabic, macedonian, danish, and serbian (10 languages. But will soon be updated to identify more languages)

• After cleaning the dataset, The most common Language Trigrams were extracted from each of the languages, and a vocabulary for each language was created from the training set based on which the whole training, validation and test set were transformed using sklearn.CountVectorizer

• These extracted features were fed to a simple feedforward Neural network with 3 hidden layers to classify the languages.

• Early stopping and model checkpoint callbacks to prevent overfitting of the model

References

• William B. Cavnar et. al. , N-Gram-Based Text Categorization
• A. Simões, J.J. Almeida, and S.D. Byers, Language identification: a neural network approach. (2014)

I would very much like to extend my gratitude to user @BobVonBob in discord (code bullet and co server, Machine Learning Channel) for suggesting the use of bigram frequencies to identify language. Which later led me to read The paper on N-gram based Text recognition (William B. Cavnar et.al) and other useful literatures