Time series forecasting with RNN(stateful LSTM) produces constant values

Question

I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values.

I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

Is this the correct way to forecast time series with LSTMs?

How can this behaviour be explained?

Will this be the case even for a time series with great seasonality?

Is this behaviour expected even for a very large time series data?

Should I rather try to train the network with Y matrix having 31st day to 60th day values for the same X?

My train data looks something like this:

array([[-0.35811423, -0.22393472, -0.39437897, ..., -0.36718042, -0.37080689, -0.35267452], [-0.22393472, -0.39437897, -0.13327289, ..., -0.37080689, -0.35267452, -0.2030825 ], [-0.39437897, -0.13327289, -0.1532185 , ..., -0.35267452, -0.2030825 , -0.25294651]

Architecture: Input LSTM layer (20 neurons) 1 hidden lstm layer 20 neurons 1 output dense layer, batch size as 1. Stateful lstm, I reset model states after each epoch.

Answer 1

Is this the correct way to forecast time series with LSTMs?

Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values

You method could work in theory. But typically, for vanilla LSTM, the setup looks like $x = [x_0, x_1, ..., x_{n-1}]$ and $y = [x_1, x_2 ..., x_n]$. I suspect you will get much better result if you do it this way.

Highly recommend Colah's excellent explaination if you haven't read it already.

I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

This is not recommended because any error in prediction is compounded over time. please find my answer in the following post for a detailed explanation Is time series multi-step ahead forecasting a sequence to sequence problem?

How can this behaviour be explained?

If you are getting constant prediction value, the chance is you are having so-called "gradient explosion problem", try to clip the gradient before backprop might be the solution.

---

Correction. Had a bit better thinking about it. it is not a gradient explosion problem here. More likely it is due to saturated neuron. Because LSTM uses sigmoid function, if the input is not properly scaled or weight is initialised inproperly, it could cause neuron saturation very quickly.

Answer 2

I am working on data for 3 years(daily data) and can see the values changing for all the days. I am trying to predict for next 30 days using Dr Jason Brownlee's tutorial.

The above tutorial helped me predict for next 30 days, I am still trying to improve on the predictions .

EDIT :While trying to improve my results I was experimenting and came across similar situation as yours. Recommendation --->Remove any activation functions from your network and it may help.