Tutorial: Prof. Ulf Johansson

How good is your prediction? In risk-sensitive applications, it is crucial to be able to assess the quality of a prediction, but traditional classification and regression models don't provide their users with any information regarding prediction trustworthiness. 
Conformal predictors, on the other hand, are predictive models that associate each of their predictions with a precise measure of confidence. Given a user-defined significance level E, a conformal predictor outputs, for each test pattern, a multivalued prediction region (class label set or real-valued interval) that, under relatively weak assumptions, contains the test pattern’s true output value with probability 1-E. In other words, given a significance level E, a conformal predictor makes an erroneous prediction with probability E. The conformal prediction framework allows any traditional classification or regression model to be transformed into a confidence predictor with little extra work, both in terms of implementation and computational complexity. 
Some key properties of conformal prediction are:

• We obtain probabilities/error bounds per instance
• Probabilities are well-calibrated: 95% means 95%
• We don't need to know the priors 
• We make a single assumption - that the data is exchangeable ~ i.i.d.
• We can apply it to any machine learning algorithm
• It is rigorously proven and straightforward to implement
• There is no magic involved – only mathematics and algorithms

Hence, confidence predictors is an important tool that every data scientist should carry in their toolboxes, and conformal prediction represents a straight-forward way of associating the predictions of any predictive machine learning algorithm with confidence measures.
This tutorial aims to provide an introduction and an example-oriented exposition of the conformal prediction framework, directed at machine learning researchers and professionals. A publicly available Python library, developed by one of the authors of the tutorial, will be used for the running examples. The goal of the tutorial is to provide attendees with the knowledge necessary for implementing functional conformal predictors, and to highlight current research on the subject.