Stanford Word Segmenter 1.6.4

Chinese is standardly written without spaces between words (as are some other languages). This software will split Chinese text into a sequence of words, defined according to some word segmentation standard. It is a Java implementation of the CRF-based Chinese Word Segmenter described in:<br /><br /> Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky and Christopher Manning. "A Conditional Random Field Word Segmenter." In Fourth SIGHAN Workshop on Chinese Language Processing. 2005. <br /><br />As of March 9, 2012, the segmenter package also includes an Arabic model, produced by Spence Green.<br /><br />The system requires Java 1.6+ to be installed. For the current setting in the scripts files, 1G of memory is required. But with smaller files, you can change the option java -mx1g to smaller numbers. Two models with two different segmentation standards are included: Chinese Penn Treebank standard and Peking University standard.<br /><br />On May 21, 2008, we released a version that makes use of lexicon features. With external lexicon features, the segmenter segments more consistently and also achieves higher F measure when we train and test on the bakeoff data. This version is close to the CRF-Lex segmenter described in:<br /><br /> Pi-Chuan Chang, Michel Galley and Chris Manning. "Optimizing Chinese Word Segmentation for Machine Translation Performance." in ACL Third Workshop on Statistical Machine Translation, 2008.

Requirements:	No special requirements
Platforms:	Mac, Mac OSX
Keyword:	Included,  Makes,  Models,  Peking,  Released,  Standards,  Treebank,  University,  Version
Users rating:	0/10