This folder contains the code for pruning the original dataset. This includes removing all kinds of special characters, numeric characters, as well as English alphabets present in the original corpus.
The files included in this are :-

--- replace.py ---
This file takes the corpus (Hi_Newspapers.txt) as input, and removes all the unnecessary characters using regular expressions. To run this script use the command.
 
  $python replace.py > new1.txt

--- trim.py ---
This file takes the output of replace.py as input (that is, new1.txt), and contains code for trimming all the unnecesary spaces and new lines. To execute this script, use.

  $python trim.py > new2.txt
 
--- wordcount.py ---
 This file takes the output of trim.py as input, which now has all the words, and counts the occurence of each word, and stores the word and its frequency in a dictionary. It then prints them in the decreasing order of frequency. Tu run this script, use.
 
  $python wordcount.py > new3.txt
  
--- trim2.py ---
 This file takes the output of wordcount.py and trims all the extra spaces and new lines. To run,
 
   $python trim2.py > hindi.csv

--- Makefile ---
This contains all the commands to run ll the scripts together. Tu run,
  
  $make

The file "hindi.csv" created at the end of this, is the actual file which contains the frequency and the word together, and thus serves as the input to "UnsupervisedWordSegmentation.cpp". The output of UnsupervisedWordSegmentation.cpp is stored in 
  Output/finalSegmentation.txt
  
--- output.txt ---
This file contains the output of Undivide++ for the TEST data. (extracted from the segmentation of the complete corpus). The results have been verified manually, between this and the hand-annotated test data.

--- finalSegmentation.txt ---
This file contains the segmentation of the COMPLETE corpus, including the learning phase. 
(The last 300 lines contain segmentation of the test data, which has been extracted separately in output.txt)
   
--- compare.py ---
This file is the script which is used to compare between the segmented data and the hand-annotated data. The hand-annotated data is in "hindi2.txt" and the test data is in "output.txt". It returns the Precision, Recall, and 
F-score.
