3. Supervised learning

• 3.1. Generalized Linear Models
  • 3.1.1. Ordinary Least Squares
    • 3.1.1.1. Ordinary Least Squares Complexity
  • 3.1.2. Ridge Regression
    • 3.1.2.1. Ridge Complexity
    • 3.1.2.2. Setting the regularization parameter: generalized Cross-Validation
  • 3.1.3. Lasso
    • 3.1.3.1. Setting regularization parameter
      • 3.1.3.1.1. Using cross-validation
      • 3.1.3.1.2. Information-criteria based model selection
  • 3.1.4. Elastic Net
  • 3.1.5. Least Angle Regression
  • 3.1.6. LARS Lasso
    • 3.1.6.1. Mathematical formulation
  • 3.1.7. Orthogonal Matching Pursuit (OMP)
  • 3.1.8. Bayesian Regression
    • 3.1.8.1. Bayesian Ridge Regression
    • 3.1.8.2. Automatic Relevance Determination - ARD
  • 3.1.9. Logisitic regression
  • 3.1.10. Stochastic Gradient Descent - SGD
• 3.2. Support Vector Machines
  • 3.2.1. Classification
    • 3.2.1.1. Multi-class classification
    • 3.2.1.2. Unbalanced problems
  • 3.2.2. Regression
  • 3.2.3. Density estimation, novelty detection
  • 3.2.4. Support Vector machines for sparse data
  • 3.2.5. Complexity
  • 3.2.6. Tips on Practical Use
  • 3.2.7. Kernel functions
    • 3.2.7.1. Custom Kernels
      • 3.2.7.1.1. Using python functions as kernels
      • 3.2.7.1.2. Using the Gram matrix
  • 3.2.8. Mathematical formulation
    • 3.2.8.1. SVC
    • 3.2.8.2. NuSVC
  • 3.2.9. Implementation details
• 3.3. Stochastic Gradient Descent
  • 3.3.1. Classification
  • 3.3.2. Regression
  • 3.3.3. Stochastic Gradient Descent for sparse data
  • 3.3.4. Complexity
  • 3.3.5. Tips on Practical Use
  • 3.3.6. Mathematical formulation
    • 3.3.6.1. SGD
  • 3.3.7. Implementation details
• 3.4. Nearest Neighbors
  • 3.4.1. Unsupervised Nearest Neighbors
  • 3.4.2. Nearest Neighbors Classification
  • 3.4.3. Nearest Neighbors Regression
  • 3.4.4. Nearest Neighbor Algorithms
    • 3.4.4.1. Brute Force
    • 3.4.4.2. K-D Tree
    • 3.4.4.3. Ball Tree
    • 3.4.4.4. Choice of Nearest Neighbors Algorithm
    • 3.4.4.5. Effect of leaf_size
• 3.5. Gaussian Processes
  • 3.5.1. An introductory regression example
  • 3.5.2. Mathematical formulation
    • 3.5.2.1. The initial assumption
    • 3.5.2.2. The best linear unbiased prediction (BLUP)
    • 3.5.2.3. The empirical best linear unbiased predictor (EBLUP)
  • 3.5.3. Correlation Models
  • 3.5.4. Regression Models
  • 3.5.5. Implementation details
• 3.6. Partial Least Squares
• 3.7. Naive Bayes
  • 3.7.1. Gaussian Naive Bayes
  • 3.7.2. Multinomial Naive Bayes
  • 3.7.3. Bernoulli Naive Bayes
• 3.8. Decision Trees
  • 3.8.1. Classification
  • 3.8.2. Regression
  • 3.8.3. Complexity
  • 3.8.4. Tips on practical use
  • 3.8.5. Tree algorithms: ID3, C4.5, C5.0 and CART
  • 3.8.6. Mathematical formulation
    • 3.8.6.1. Classification criteria
    • 3.8.6.2. Regression criteria
• 3.9. Ensemble methods
  • 3.9.1. Forests of randomized trees
    • 3.9.1.1. Random Forests
    • 3.9.1.2. Extremely Randomized Trees
    • 3.9.1.3. Parameters
    • 3.9.1.4. Parallelization
• 3.10. Multiclass and multilabel algorithms
  • 3.10.1. One-Vs-The-Rest
    • 3.10.1.1. Multilabel learning with OvR
  • 3.10.2. One-Vs-One
  • 3.10.3. Error-Correcting Output-Codes
• 3.11. Feature selection
  • 3.11.1. Univariate feature selection
  • 3.11.2. Recursive feature elimination
  • 3.11.3. L1-based feature selection
  • 3.11.4. Tree-based feature selection