There are several tutorials and courses explaining how to monitor a model, discussing data drift and the need to retrain our predictive models occasionally. However, when I think about it, I encounter a problem that I don't know how to deal with, and there isn't any clear tutorial showing how to handle it.
For instance, suppose I have a churn model trained with data from January to June. Then I use this model to give discounts to the customers most likely to churn. In October, for example, I find that we have some data drift, some change in customer behavior, and our model starts to make more incorrect predictions than before. I need to retrain it; however, isn't that data "contaminated" by the actions I've taken? I mean, some customers' behaviors were changed by the discounts, weren't they?
I thought about not using these customers, but I don't think that would be fair because it would force a different balance on the target.
Any ideas on this problem?
