8.7.2.2. sklearn.feature_extraction.text.WordNGramAnalyzer¶
- class sklearn.feature_extraction.text.WordNGramAnalyzer(charset='utf-8', min_n=1, max_n=1, preprocessor=RomanPreprocessor(), stop_words='english', token_pattern=u'\b\w\w+\b')¶
Simple analyzer: transform text document into a sequence of word tokens
- This simple implementation does:
- lower case conversion
- unicode accents removal
- token extraction using unicode regexp word bounderies for token of minimum size of 2 symbols (by default)
- output token n-grams (unigram only by default)
The stop words argument may be “english” for a built-in list of English stop words or a collection of strings. Note that stop word filtering is performed after preprocessing, which may include accent stripping.
Methods
analyze(text_document) From documents to token set_params(**params) Set the parameters of the estimator. - __init__(charset='utf-8', min_n=1, max_n=1, preprocessor=RomanPreprocessor(), stop_words='english', token_pattern=u'\b\w\w+\b')¶
- analyze(text_document)¶
From documents to token
- set_params(**params)¶
Set the parameters of the estimator.
The method works on simple estimators as well as on nested objects (such as pipelines). The former have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.
Returns : self :