I'm studying a paper about Named Entity Recognition. The following is a part of the abstract:
To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity.
I don't know what "label shift" definition is. The paper doesn't explain it and I can not find anything that I could understand by Googling it.
Label shift is the opposite of a covariate shift.
In this case, the assumption is that even though the feature distribution remains the same, the Label distribution might changes.
e.g. Symptoms --> Diseases
It can be different for different country (based on medical education of the Country/Doctor)
It can change with time also based on advancement in Medical knowledge
Similar logic can be built for "Words --> Slang". It can change with time due to the acceptance of these words.
Read the references for a formal explanation.
References-
Dive into Deep Learning
Detecting and Correcting for Label Shift with Black Box Predictors
