This documentation is for scikit-learn version 0.11-gitOther versions


If you use the software, please consider citing scikit-learn.

This page

8.3.11. sklearn.cross_validation.permutation_test_score

sklearn.cross_validation.permutation_test_score(estimator, X, y, score_func, cv=None, n_permutations=100, n_jobs=1, labels=None, random_state=0, verbose=0)

Evaluate the significance of a cross-validated score with permutations

Parameters :

estimator: estimator object implementing ‘fit’ :

The object to use to fit the data

X: array-like of shape at least 2D :

The data to fit.

y: array-like :

The target variable to try to predict in the case of supervised learning.

score_func: callable :

Callable taking as arguments the test targets (y_test) and the predicted targets (y_pred) and returns a float. The score functions are expected to return a bigger value for a better result otherwise the returned value does not correspond to a p-value (see Returns below for further details).

cv : integer or crossvalidation generator, optional

If an integer is passed, it is the number of fold (default 3). Specific crossvalidation objects can be passed, see sklearn.cross_validation module for the list of possible objects

n_jobs: integer, optional :

The number of CPUs to use to do the computation. -1 means ‘all CPUs’.

labels: array-like of shape [n_samples] (optional) :

Labels constrain the permutation among groups of samples with a same label.

random_state: RandomState or an int seed (0 by default) :

A random number generator instance to define the state of the random permutations generator.

verbose: integer, optional :

The verbosity level

Returns :

score: float :

The true score without permuting targets.

permutation_scores : array, shape = [n_permutations]

The scores obtained for each permutations.

pvalue: float :

The returned value equals p-value if score_func returns bigger numbers for better scores (e.g., zero_one). If score_func is rather a loss function (i.e. when lower is better such as with mean_squared_error) then this is actually the complement of the p-value: 1 - p-value.


This function implements Test 1 in:

Ojala and Garriga. Permutation Tests for Studying Classifier Performance. The Journal of Machine Learning Research (2010) vol. 11