Contents

6.15.5. scikits.learn.feature_extraction.text.CountVectorizer

CountVectorizer(analyzer=WordNGramAnalyzer(stop_words=set(['all', 'six', 'less', 'being', 'indeed', 'over', 'move', 'anyway', 'four', 'not', 'own', 'through', 'yourselves', 'fify', 'where', 'mill', 'only', 'find', 'before', 'one', 'whose', 'system', 'how', 'somewhere', 'with', 'thick', 'show', 'had', 'enough', 'should', 'to', 'must', 'whom',...'amoungst', 'yours', 'their', 'rather', 'without', 'so', 'five', 'the', 'first', 'whereas', 'once']),
max_n=1, token_pattern='\b\w\w+\b', charset='utf-8', min_n=1,
preprocessor=RomanPreprocessor()), vocabulary={}, max_df=1.0, max_features=None, dtype=<type 'long'>)

Convert a collection of raw documents to a matrix of token counts

This implementation produces a dense representation of the counts using a numpy array.

If you do not provide an a-priori dictionary and you do not use an analyzer that does some kind of feature selection then the number of features (the vocabulary size found by analysing the data) might be very large and the count vectors might not fit in memory.

For this case it is either recommended to use the sparse.CountVectorizer variant of this class or a HashingVectorizer that will reduce the dimensionality to an arbitrary number by using random projection.

Parameters :

analyzer: WordNGramAnalyzer or CharNGramAnalyzer, optional :

vocabulary: dict, optional :

A dictionary where keys are tokens and values are indices in the matrix. This is useful in order to fix the vocabulary in advance.

dtype: type, optional :

Type of the matrix returned by fit_transform() or transform().

Methods

fit
fit_transform
transform
CountVectorizer.fit(raw_documents, y=None)

Learn a vocabulary dictionary of all tokens in the raw documents

Parameters :

raw_documents: iterable :

an iterable which yields either str, unicode or file objects

Returns :

self :

CountVectorizer.fit_transform(raw_documents, y=None)

Learn the vocabulary dictionary and return the count vectors

This is more efficient than calling fit followed by transform.

Parameters :

raw_documents: iterable :

an iterable which yields either str, unicode or file objects

Returns :

vectors: array, [n_samples, n_features] :

CountVectorizer.transform(raw_documents)

Extract token counts out of raw text documents

Parameters :

raw_documents: iterable :

an iterable which yields either str, unicode or file objects

Returns :

vectors: array, [n_samples, n_features] :