While segmenting the word major factors is the language model one is using like the unigram model or the bigram model. Other factors are:

• Language Model ie unigram , bigram
• Length of largest word (L)
• Frequency of non-occuring

• Unigram
• Unigram frequency file -> unigram_corpus
• The best result are obtaines with L=30 and probability for non-occuring words are set inversely proportional to their length
• Table below shows the precision,recall and fscore values obtained for various settings:
  
  



Value of L	Precision	Recall	F Score
20	88.7335526316	95.9964412811	92.2222222222
25	96.0538116592	95.2846975089	95.6677087986
30	97.6168652612	94.7508896797	96.1625282167

• Bigram
• Bigram frequency file -> bigram_corpus
• The best result are obtaines with L=30 and probability for non-occuring words are set inversely proportional to their length
• Table below shows the precision,recall and fscore values obtained for various settings:
  
  
  

  Value of L	Precision	Recall	F Score
  20(without end tag)	88.2736156352	96.4412811388	92.1768707483
  25(without end tag)	95.4946996466	96.1743772242	95.8333333333
  40(without end tag)	97.7375565611	96.0854092527	96.9044414536
  30(without end tag)	97.2122302158	96.1743772242	96.6905187835
  30(with end tag)	97.4752028855	96.4317573595	96.9506726457

• The above figures were obtained from the input files given by tutor.


• Results for some random files as obtained by the best parameters obtained above are as:-



Random File	Precision	Recall	F Score
random1(1044 words)	99.4	98.6111111111	99.0039840637
random2(811)	96.2593516209	98.0940279543	97.1680302077

The python files used are code
The result files can be seen at: data files