Time to learn about best practices of evaluating and hyperparameter tuning for ML models

• Streamlining workflows with pipelines:
  • When we applied different preprocessing techniques like PCA, we had to reuse the parameters that were obtaining during the fitting of training data to scale and compress new data. With the Pipeline class in Scikit-learn, we can fit a model including an arbitrary nubmer of transformations and apply it to make predictions about new data
  • Will work with the Breast Cancer Wisconsin dataset for these examples, and pipeline examplewill be in the code
• Using k-fold cross-validation to assess model performance
  • Will learn about common cross-validation techniques: holdout and k-fold cross-validation
  • Holdout method:
    • Split initial dataset into separate training and test datasets. In typical ML applications we are also interested in tuing and comparing different parameter settings, aka model selection. However, if we reuse the same test dataset over and over during training, it will become part of training data and increase chance of overfit
    • Better way of using holdout method is to separate data into training validation and test datasets. The performance of validation dataset is used for model selection.
      • Validation is used to repeatedly evaluate the performance of the model after training using different hyperparameter values
      • Once satisfied, we can then use test dataset to estimate model’s generalized performance
    • Disadvantage of the holdout method is that performance estimate might be very sensitive to how we partition the dataset, and so we can look at a more robust technique for performance estimation
  • K-fold cross-validation:
    • Randomly split the training dataset into k-folds without replacement. (Training and testing dataset already split, so this is only training dataset)
      • k-1 folds are the training folds and are used for model training
      • one fold called the test fold is used for performance evaluation
      • This procedure is done k times so that we obtain k models and performance estimates
      • We then calculate average performance to get a performance estimate that is less sensitive to the sub-partitioning of the training data compare to holdout method
      • K-fold cross-validation is typically used for model tuning, which is finding the optimal hyperparameter values that yield a satisfying generalization performance
      • Once we have found satisfactory hyperparameter values, can retrain model on the complete training set and obtain a final performance estimate using the independent test data
        • We retrain so that we can have a complete singular model and the model gets more training data so it is generally more accurate
    • Since it is a resampling technique without replacement, the advantage is that in each iteration each example is used exactly once, and all test folds are distjoint, meaning there is no overlap between the test folds
      • Thus, k-fold cross-validation makes better use of the dataset than holdout method with a validation set, as all data points are being used for evaluation
    • Good standard for k is 10, as empirical evidence shows that it often offers the best tradeoff between bias and variance
      • However, smaller training sets might want more number of folds, as then more training data will be used in each iteration, which results in a lower pessimistic bias
        • However, large values of k will also increase runtime and yield higher variance
      • Larger datasets can use smaller values of k like k=5, to help reduce overfitting from increase variance
    • Slight improvement over standard k-fold is stratified k-fold which can yield better bias and variance estimates especially with cases of unequal class proportions
      • The class label proportions are preserved in each fold to ensure each fold is representative of the class proprotions in the training dataset
      • Illustrated by StratifiedKFold iterator in scikit-learn in code
• Debugging algorithms with learning and validation curves:
  • Will look at learning curves and validation curves to help improve performance of learning algo
    • Main difference between learning curves and validation curves is learning curves vary the amount of training data while validation curves vary the models hyperparameters
  • Diagnosing bias and variance problems with learning curves
    • One way to reduce overfitting is collecting more data, but this is not always practically feasible
    • Code will show how to use learning curve function in scikit learn to evaluate model
  • Addressing over and underfitting with validation curves
    • They are related to learning curves but instead of plotting training and test accuracies as functions of sample size, we instead vary the model parameters
    • In the example in code we vary the inverse regularization parameter C in logistic regression
• Fine-tuning Machine Learning models via grid search
  • ML has two types of parameters:
    • ones that are learned from training data, ex: weights in logistic regression
    • Parameters of a learning algo that are optimized separately
      • These are the tuning parameters aka hyperparameters of the model
      • Ex: regularization parameter in logistic regression or max depth of decision tree
  • Will take a look at popular hyperparameter optimization technique called grid search, which tries to find the optimal combination of hyperparameter values
  • Tuning hyperparameters via grid search
    • Approach is simple: it is a brute-force exhaustive search paradigm where a list of values for different hyperparams are specified, and the computer goes through each to find the optimal combo from the set
    • example in code
  • Exploring hyperparameter configurations more widely with randomized search
    • Grid search is exhaustive so it is guaranteed to find optimal hyperparams, but it is very expensive in practice on a large grid
    • Can instead use randomized search which doesn’t do an exhaustive search but allows us to find wider ranger of hyperparameter values in a more time and cost effective manner
    • Implementation in code
  • More resource-efficient hyperparameter search with successive halving
    • Taking randomized search a step further, we can do successive halving to making finding suitable hyperparameter configs more suitable
    • Steps:
      1. Draw a large set of candidate configs via random sampling
      2. Train the model with limited resources like a small subset of training data
      3. Discard the bottom 50 percent based on predictive performance
      4. Back to step 2 with increased amount of resources
    • Repeat these steps until only one hyperparameter config still remains
    • Also has a grid search variant where all specified configs are tested instead of random samples
    • Implementation in code
  • Algo Selection with nested cross-validation
    • Using k-fold cross-validation in combo with grid or randomized search can help fine-tune the performance of ML model by varying hyperparameter values
    • Another recommended approach is nested cross-validation, which in a study showed that true error of the estimate is almost unbiased rlative to test dataset when nested cross-validation is used

      • We have an outer k-fold cross-validation loop to split the data into training and testing folds, and inner loop is used to select model using k-fold cross-validation on training fold

      • After model selection, test fold is used to evaluate model performance

      • Example of 5x2 cross validation, where 5 outer and 2 inner folds are used:

        

      • Code implemenation in code

• Looking at different performance evaluation metrics:
  • Some metrics to evaluate performance other than accuracy: recall, precision, F1 score, and Matthews correlation coefficient (MCC)
  • Reading a confusion matrix:
    • Reports counts of TP TN FP and FN
    • Can use Scikit-learn’s confusion_matrix function to display it easily
  • Optimizing the precision and recall of classification model
    • Error is the number of wrong predictions so FP and FN over the # of predictions
    • ACC = 1 – ERR
    • True positive rate (TPR) is true positives over all positives (FN + TP) while False positive rate (FPR) is number of false positives over all negatives (FP+TN)
    • Recall is synonymous with TPR
    • Precision is how many records predicted as relevant are actually relevant
    • For example in tumor diagnosis, we care more about the detection of malignant tumors while it is also important to decrease the amount of benign tumors incorrectly classified.
      • Optimizing for recall helps with minimizing the chance of not detecting a malignant tumor
        • However can come with cost of predicting FP
      • Optimizing for precision will emphasize correctness when predicting, but might miss malignant tumors more frequently ( high FPs)
      • To balance optimizing PRE and REC, harmonic mean is used, aka F1:
      • F1 = 2(PRE X REC) / (PRE + REC)
    • Another metric is the MCC, which is especially popular in biological research contexts:
    • Scoring metrics demo in code
  • Plotting receiver operating characteristic
  • Scoring metrics for multiclass classification
  • Dealing with class imbalance