Uploaded February 2020 | Updated September 2026, 3 days ago
slides: mlvu.github.io/lectures/21.Methodology1.annotated.pdf
course materials: mlvu.github.io
In this lecture, we discuss the practicalities to performing machine learning experiments. How to set up an experiment, what to report and how to report it. We dive into basic metrics to report for binary classification starting with the the confusion matrix and derived metrix like the accuracy, error, precision, recall and true positive rate, false positive rate. We see how to balance different metrics, using the ROC space, precision-recall curves and the area under the curve metric.
For regression, we look at the bias-variance decomposition; a very useful principle in analyzing our model's performance. We also look at how to perform statistical analysis on our results, whether we should do it.
Finally, we have a quick look at the no free lunch theorem, and whether it has any relevance to practical every-day machine learning.
slides: mlvu.github.io/lectures/21.Methodology1.annotated.pdf
course materials: mlvu.github.io
In this lecture, we discuss the practicalities to performing machine learning experiments. How to set up an experiment, what to report and how to report it. We dive into basic metrics to report for binary classification starting with the the confusion matrix and derived metrix like the accuracy, error, precision, recall and true positive rate, false positive rate. We see how to balance different metrics, using the ROC space, precision-recall curves and the area under the curve metric.
For regression, we look at the bias-variance decomposition; a very useful principle in analyzing our model's performance. We also look at how to perform statistical analysis on our results, whether we should do it.
Finally, we have a quick look at the no free lunch theorem, and whether it has any relevance to practical every-day machine learning.
