Coverage vs Acceptance-Error Curves for Conformal Classification Models

Evgueni Smirnov

Coverage vs Acceptance-Error Curves for Conformal Classification Models

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

In this paper, we introduce coverage vs acceptance-error graphs as a visualization tool for comparing the performance of conformal predictors at a given significance level ? for any k-class classification task with k = 2. We show that by plotting the performance of each predictor for different significance levels in ? ? [0, 1], we receive a coverage vs acceptance-error curve for that predictor. The area under this curve represents the probability that the p-value of randomly chosen true class-label of any test instance is greater than the p-value of any other false class-label for the same or any other test instance. This area can be used as a metric for predictive efficiency of a conformal predictor, when the validity has been established. The new metric is unique in that it is related to the empirical coverage rate, and extensive experiments confirmed its utility and difference from existing predictive efficiency criteria.

Original language	English
Title of host publication	Proceedings of Machine Learning Research
Editors	Neil Lawrence
Pages	534-545
Number of pages	12
Volume	204
Publication status	Published - 1 Jan 2023
Event	12th Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2023 - Limassol, Cyprus Duration: 13 Sept 2023 → 15 Sept 2023 https://cml.rhul.ac.uk/copa2023/ https://copa-conference.com/

Publication series

Series	Proceedings of Machine Learning Research
ISSN	2640-3498

Conference

Conference	12th Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2023
Abbreviated title	COPA 2023
Country/Territory	Cyprus
City	Limassol
Period	13/09/23 → 15/09/23
Internet address	https://cml.rhul.ac.uk/copa2023/ https://copa-conference.com/

Keywords

Conformal Prediction
Metrics
Performance Curves
Predictive Efficiency

Access to Document

https://proceedings.mlr.press/v204/

Cite this

@inproceedings{6ce8ba8bf61049519a2ea2e6c2b187f7,

title = "Coverage vs Acceptance-Error Curves for Conformal Classification Models",

abstract = "In this paper, we introduce coverage vs acceptance-error graphs as a visualization tool for comparing the performance of conformal predictors at a given significance level ? for any k-class classification task with k = 2. We show that by plotting the performance of each predictor for different significance levels in ? ? [0, 1], we receive a coverage vs acceptance-error curve for that predictor. The area under this curve represents the probability that the p-value of randomly chosen true class-label of any test instance is greater than the p-value of any other false class-label for the same or any other test instance. This area can be used as a metric for predictive efficiency of a conformal predictor, when the validity has been established. The new metric is unique in that it is related to the empirical coverage rate, and extensive experiments confirmed its utility and difference from existing predictive efficiency criteria.",

keywords = "Conformal Prediction, Metrics, Performance Curves, Predictive Efficiency",

author = "Evgueni Smirnov",

note = "Funding Information: I would like to express my gratitude to the anonymous reviewers for their valuable feedback and insightful comments on the paper. Publisher Copyright: {\textcopyright} 2023 E. Smirnov.; 12th Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2023, COPA 2023 ; Conference date: 13-09-2023 Through 15-09-2023",

year = "2023",

month = jan,

day = "1",

language = "English",

volume = "204",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research",

pages = "534--545",

editor = "Neil Lawrence",

booktitle = "Proceedings of Machine Learning Research",

url = "https://cml.rhul.ac.uk/copa2023/, https://copa-conference.com/",

}

Smirnov, E 2023, Coverage vs Acceptance-Error Curves for Conformal Classification Models. in N Lawrence (ed.), Proceedings of Machine Learning Research. vol. 204, Proceedings of Machine Learning Research, pp. 534-545, 12th Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2023, Limassol, Cyprus, 13/09/23. <https://proceedings.mlr.press/v204/>

TY - GEN

T1 - Coverage vs Acceptance-Error Curves for Conformal Classification Models

AU - Smirnov, Evgueni

PY - 2023/1/1

Y1 - 2023/1/1

N2 - In this paper, we introduce coverage vs acceptance-error graphs as a visualization tool for comparing the performance of conformal predictors at a given significance level ? for any k-class classification task with k = 2. We show that by plotting the performance of each predictor for different significance levels in ? ? [0, 1], we receive a coverage vs acceptance-error curve for that predictor. The area under this curve represents the probability that the p-value of randomly chosen true class-label of any test instance is greater than the p-value of any other false class-label for the same or any other test instance. This area can be used as a metric for predictive efficiency of a conformal predictor, when the validity has been established. The new metric is unique in that it is related to the empirical coverage rate, and extensive experiments confirmed its utility and difference from existing predictive efficiency criteria.

AB - In this paper, we introduce coverage vs acceptance-error graphs as a visualization tool for comparing the performance of conformal predictors at a given significance level ? for any k-class classification task with k = 2. We show that by plotting the performance of each predictor for different significance levels in ? ? [0, 1], we receive a coverage vs acceptance-error curve for that predictor. The area under this curve represents the probability that the p-value of randomly chosen true class-label of any test instance is greater than the p-value of any other false class-label for the same or any other test instance. This area can be used as a metric for predictive efficiency of a conformal predictor, when the validity has been established. The new metric is unique in that it is related to the empirical coverage rate, and extensive experiments confirmed its utility and difference from existing predictive efficiency criteria.

KW - Conformal Prediction

KW - Metrics

KW - Performance Curves

KW - Predictive Efficiency

M3 - Conference article in proceeding

VL - 204

T3 - Proceedings of Machine Learning Research

SP - 534

EP - 545

BT - Proceedings of Machine Learning Research

A2 - Lawrence, Neil

T2 - 12th Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2023

Y2 - 13 September 2023 through 15 September 2023

ER -