Hindi Corpus Used for building unigram and bigram models: "corpus.zip"
Given Validation File : hi-wordseg.txt
My Codes : Codes
Unigrams and Bigrams: unigrams_bigrams.zip
The precision and recall:
PARAMETER1: Length of the largest word (Most optimal value is 23) - Starting with a value of 20 I experimented with the value 23,25,27,33,35. Increasing length improve recall for the words which belong to the training corpus as the probability increases whereas for words not present in corpus recall decreases as the negatively contribute to the corpus.
I removed all the punctuations from the validation set - three were found !,?,. I also included <s> and </s> to mark the begining and end of the sentence.
Validation File with spaces removed: hi-wseg-gt.txt
Results
Precision
Recall
f-score
Unigrams-With segmentation Maximum Word Length=20
88.7973640857
95.9074733096
92.2155688623
Unigrams+Bigrams+Segmentation Maximum Word Length=20
88.4897959184
96.4412811388
92.294593444
Unigrams-Maximum Word Length=23
93.339100346
95.9964412811
94.649122807
Unigrams+Bigrams+Maximum Word Length=23
92.8999144568
96.6192170819
94.7230702137
Unigrams-Maximum Word Length=35
97.9742173112
94.6619217082
96.2895927602
Unigrams+Bigrams+Maximum Word Length=35
97.4729241877
96.0854092527
96.7741935484
Discussion of Results and parameters
PARAMETER2: Probability of a word which is not present in the corpus. The probability is inversely proportional to the length of the word with value around 1/(10^6).
PARAMETER3: Another major parameter that affects the accuracy is the choice of the probability model used. Using unigrams+bigrams instead of just unigrams increases probability alot.
Other Optimizations