Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [adaboost]

26 questions
13
votes
1 answer

AdaBoost implementation and tuning for high dimensional feature space in R

I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. There exists a variety of different packages for this purpose; AdaBag, Ada and gbm.…
AfBM
  • 131
  • 3
5
votes
1 answer

Is the way to combine weak learners in AdaBoost for regression arbitrary?

I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression. In paper Improving Regressors using Boosting Techniques, the final prediction is the weighted median. For a particular…
Akira
  • 175
  • 4
4
votes
2 answers

How to use a set of pre-defined classifiers in Adaboost?

Suppose there are some classifiers as follows: dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0) rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0) xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0) knn =…
Aaron
  • 231
  • 1
  • 3
  • 9
2
votes
1 answer

Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after using PCA in terms of accuracy and runtime. For some…
appeldaniel
  • 41
  • 3
2
votes
0 answers

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach breaks down when we have finite data and some way to…
figs_and_nuts
  • 903
  • 1
  • 5
  • 17
2
votes
1 answer

Decreasing n_estimators is increasing accuracy in AdaBoost?

I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset. (X,Y are the predictor columns and the color is the label) As you can see there are exactly 16 points in either side that can be easily miss-classified. TO check…
samarendra chandan bindu Dash
  • 197
  • 11
2
votes
0 answers

AdaBoost vs Gradient Boost

What is the difference? Under which criteria should each type of boost be used? What is the theory behind each of these methods?
Nikita Rogers
  • 41
  • 1
  • 3
1
vote
1 answer

Evaluating optimal values for depth of tree

I'm studying the performance of an AdaBoost model and I wonder how it performs in regard to the depth of the trees. Here's the accuracy for the model with a depth of 1 and here with a depth of 3 From my point of view, I would say the lower one…
Ben
  • 570
  • 5
  • 16
1
vote
1 answer

Forecasting: Multiple Linear Regression (OLS) outperforms Random Forests / Gradient Boosting / AdaBoost

I'm using different forecasting methods on a dataset to try and compare the accuracy of these methods. For some reason, multiple linear regression (OLS) is outperforming RF, GB and AdaBoost when comparing MAE, RMSE R^2 and MAPE. This is very…
0009
  • 31
  • 3
1
vote
1 answer

How Adaboost calculates error for each weak learner in training?

I am studying the Adaboost classification algorithm because i would like to implement it from scratch. I understand how it works, but i am not able to understand where some steps are placed. I will describe the Adaboost training steps in my…
heresthebuzz
  • 395
  • 3
  • 11
1
vote
1 answer

Adaboost with other classifier fitting

There is the opportunity to fit decision trees with other decision trees. For example: adaclassification= AdaBoostClassifier(RandomForestClassifier(n_jobs=-1)) adaclassification.fit(X_train,y_train) I got better results with random forest, so…
martin
  • 329
  • 3
  • 14
1
vote
2 answers

Does gradient boosting algorithm error always decrease faster and lower on training data?

I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which results in the following loss plot: I'd say this is…
Xaume
  • 212
  • 3
  • 14
1
vote
0 answers

AdaBoost.R2 learning rate from scikit learn

AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar. The implementation of regression for AdaBoost in scikit learn uses this algorithm (paper is cited in…
Lucien Ledune
  • 23
  • 2
1
vote
1 answer

Why Adaboost SAMME needs f to be estimable?

I am trying to understand the mathematics behind SAMME AdaBoost: At some stage, the paper adds a constraint for f to be estimable: I do not understand why this is required. Can someone explain a bit better why this restriction is needed? As well,…
Aggamarcel
  • 45
  • 1
  • 5
1
vote
1 answer

Explanation on some steps of AdaBoost.R2

I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really get. The paper is available here, and Adaboost.R2…
Lucien Ledune
  • 23
  • 2
1
2