CS671 - Natural Language Processing

Assignment 1 - Morphological Structure Discovery

Corpus Pruning:
1. Go to directory corpusPrune/.
2. Put Hi_Blogs.txt and Hi_Newspapers.txt in it.
3. Execute prune_corpus.bash (present in that directory)
4. Corpus will be pruned.
(Steps executed by prune_corpus.bash are mentioned on the assignment page:
 "http://home.iitk.ac.in/~sakaar/cs671/hw1/")
5. File vocab.txt is produced after running the above script. This
   file is fed into Undivide++

(It takes a lot of time to run this script so the output file is already placed in the folder)

Running Undivide++:
1. Go to Directory FinalDistribution.
2. Run:
   g++ UnsuperwisedWordSegmentation.cpp
   ./a.out ../corpusPrune/vocab.txt 1 1 1 1 0
3. Output files will be generated.

Getting Precision/Recall/F-Scores:
1. Go to Directory finalComputations
2. Run:
   ./run.bash
   This generates final_output.txt, and scores.txt
3. final_output.txt:
   This file contains comparisons made by code between actual segmentation
   and segmentation made by Undivide++.
   For each word it gives:
   a. Correct Segmentation
   b. Undivide++ Segmentation
   c. insertions, deletions and hits for that word
   d. Cumulative insertions, deletions and hits till that word.
   At the end of the file are the precision recall and fscore.
4. scores.txt:
   This file contains the computed f-score, precision and recall.
5. calculator.c: C code to calculate precision, recall and f-score
6. hand_annotate.txt: self-segmented list38.txt