Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [uncertainty]

20 questions
5
votes
2 answers

How is uncertainty evaluated for results obtained via machine learning techniques?

As machine learning (in its various forms) grows ever more ubiquitous in the sciences, it becomes important to establish logical and systematic ways to interpret machine learning results. While modern ML techniques have shown themselves to be…
PerplexedDimension
  • 153
  • 3
3
votes
2 answers

A deployed model has epistemic or aleatoric uncertainty?

Aleatoric uncertainty refers to the notion of randomness that there is in the outcome of an experiment that is due to inherently random effects. Epistemic uncertainty refers to the ignorance of the decision-maker, due to for example lack of…
Carlos Mougan
  • 6,430
  • 2
  • 20
  • 51
3
votes
2 answers

Using conformal predictors to estimate uncertainty?

I read this interesting book on conformal predictors: https://arxiv.org/abs/2107.07511. Conformal predictors are a way to choose a set that's guaranteed to include the true labels with some pre-chosen certainty. I was wondering if there's a way to…
user151434
  • 31
  • 1
1
vote
0 answers

Research Topics in Conformal Prediction

My background is in econometrics and soon I'll start to work in my master's thesis. One thing that recently got my attention were uncertainty quantification methods, specifically Conformal Prediction. One thing that seems particularly cool is that…
Frederico Portela
  • 31
  • 2
1
vote
0 answers

Model uncertainty quantification

I'm reading a paper about model uncertainty quantification. Specifically, it says epistemic uncertainty is a kind of uncertainty due to lack of knowledge about a particular region in the input space. Also, it makes a mathematical characterization of…
piero
  • 61
  • 5
1
vote
0 answers

What is the best way to combine cross-validation and bootstrapping for one application?

We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates. Currently, we use cross-validation to model the optimal smoothness of our spline models…
Stan Tendijck
  • 111
  • 1
1
vote
2 answers

What is the difference between conformal prediction and uncertainty estimation

Recently I am seeing the topic of Conformal Prediction to be very trendy on social media and research. Awesome Conformal Prediction But what is the main difference between conformal prediction and uncertainty estimation?
Carlos Mougan
  • 6,430
  • 2
  • 20
  • 51
1
vote
0 answers

Is there a way to quantify uncertainty in classification?

I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that outputs a confidence score about how "sure you are…
Tom
  • 85
  • 8
1
vote
1 answer

Can a simple distance to a few nearest data points be used a measure of the uncertainty of a prediction?

One of the 'selling points' of the Gaussian process regression is that it provides not only the model but also the uncertainty estimate of a prediction. Then usually a picture is shown with a curve fitted to the data and a shaded area around it…
Vladislav Gladkikh
  • 1,206
  • 11
  • 21
1
vote
0 answers

How to forecast time series with negative trend in test set and big uncertainty? (uncertainty due to Covid and Ukraine crisis)

Recently I started to create a machine learning model for a European customer for around 800 product time series. The goal is to produce a monthly forecast for the 6 months ahead. Since this customer is a grocery wholesaler, a lot of the products…
LGe
  • 145
  • 6
1
vote
1 answer

Confidence intervals for evaluation on test set

I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set. As far as I can see, there are two different ways of evaluating the accuracy of a metric like, say,…
Tom
  • 43
  • 3
0
votes
0 answers

State-of-the-Art Methods for Uncertainty Quantification

I am looking for an overview of the current state-of-the-art methods in uncertainty quantification (UQ), particularly for high-dimensional systems. I would appreciate any references to review papers, books, or recent research articles that provide a…
TDE
  • 1
0
votes
0 answers

What's a suitable error metric for quantifying uncertainty in future projections using a model ensemble?

Rather than just providing a mean projection, I'm looking to provide a likely range of projections using output from 9 models. Each dataset consists of spatial maximum probability values [0,1] for a specific region, each comprised of about 1000…
liveFreeOrπHard
  • 101
  • 2
0
votes
0 answers

Adressing uncertainty of a spatio-temporal multivariate timeseries with random temporal gaps

Imagine there are multiple locations of interest from where water samples are gathered manually. Each sample is immediately analyzed, converted to a numerical value (a real number) and fed into a database. These values correlate s.t. some…
Laurimann
  • 21
  • 3
0
votes
1 answer

Is it ok to use MC-dropout technique to estimate uncertainty without putting dropout after every weight layer?

In the paper by Kendall and Gal (What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?), dropout is being set after every convolutional layer. However, is it still legit to estimate epistemic and aleatoric uncertainty on the…
user128232
1
2