SARIMAX

• The (extended) ARIMA family of methods is too big to be properly explained in this course.

  • At NMBU, the course DAT320 goes deeper and explains the data generating processes.

  • An online book with videos: Forecasting: Principles and Practice.

• We therefore skip (almost) directly to the regression models (inspired by phosgene89’s GitHub page, except for their errors) and their usage.

• But first we introduce a dataset and the concepts of stationarity and autocorrelation.

Stationarity:

• The distribution of the time series is independent of which part of the time series you look at.

  • Trends, seasonality (cycles of fixed width) and changes in variance lead to non-stationarity.

  • Differencing (first or second order discrete derivatives) can help.

    • Seasonal differencing means the difference is not between neighbours but higher lags.

• If the data is not stationary, pre-processing or modelling of the specific deviations from stationarity is needed.

Wholesale price index (WPI) data¶

• We will illustrate some concepts and models using the WPI data.

Autocorrelation¶

• Autocorrelation is the correlation between a stationary timeseries and a lagged version of itself.

  • This is a measure of the time dependence in the series, i.e., lack of independence.

  • Can be used to indicate the appropriate lag in moving average (MA) models.

• Partial autocorrelation is the autocorrelation when controlling for (regressed on) all intermediate time lags.

  • Can be used to indicate the appropriate lag in autogregressive (AR) models.

Autoregressive models in Python¶

• Using the statsmodels package, the most complex model is the starting point.

• Setting the various parameters of $SARIMAX(p, d, q)(P, D, Q, s)$, we can obtain any of the below mentioned models.

  • SARIMAX(endog, exog=None, order=(1, 0, 0), seasonal_order=(0, 0, 0, 0), trend=None, ...)

• For integer values of $p/d/q/P/D/Q$, all lags up to the integer are included. For more fine-grained control, lists can be applied, e.g., [1,0,1] includes lags 1 and 3, but not 2.

• There are also various other parameters, e.g., a trend (none, constant, linear, quadratic, polynomial).

Lag operator¶

• Back-shift $n$ time points, i.e., extracts the measurement $n$ time points before $t$.

  $$L^{n} y_{t} = y_{t-n}$$

  (1)

• Also noted as $B^{n}$ in some litterature.

• Combined with a vector of parameters $\Psi$, we define a polynomial function

  $$\Psi(L)^n y_t = y_t + \psi_1 y_{t-1} + \psi_2 y_{t-2} + ... + \psi_n y_{t-n}$$

  (2)

• This will enable compact notation of the SARIMAX models, exchanging sums with the polynomial function.

AR - autoregressive models¶

• For a (single variable) timeseries given by $\{ y_{t} \}$, we can specify the $AR(p)$ model as:

• i.e., the current time is a function of $p$ previous time points and a constant.

• Here, $\theta_0$ is a constant, $\theta_{i}$ is the coefficient for the $p$-th time lag and $\epsilon_{t}$ is the error.

• This can be thought of as stacking subsets of a time series using a moving window and performing ordinary least squares on the resulting matrix/dataframe.

  • In practice, the fitting is performed using maximum likelihood, so performing ordinary regression will not give exactly the same results.

• Using the lag operator, $L^{n} y_{t} = y_{t-n}$, we can redefine the above equation in the form of a polynomial function, $\Theta(L)^{p}$ (signs of coefficients will change) as:

MA - moving average models¶

• MA models are functions of previous errors, rather than previous measurements.

• We can define an $MA(q)$ model as:

• Here, $q$ is the number of time lags and $\Phi(L)^{q}$ is defined as $\Theta$ above, but using the error terms and $\epsilon_{t}$ is with respect to the current model.

ARMA - autoregressive moving average models¶

• When we take the sum of $AR(p)$ and $MA(q)$ models of the same time series, we get $ARMA(p,q)$ models:

• which can be reformulated to:

• Again, $\epsilon_{t}$ is with respect to the current model, but shares name with the previous models.

• This model is learning both from seeing previous samples and from how well these were predicted at previous time steps, thus it can tackle changes in the average.

ARIMA - autoregressive integrated moving average models¶

• To help compensate for lack of stationarity, we add an integration operator, $\Delta^{d}$, defined as:

• Here, $y_{t}^{[0]} = y_{t}$ and the degree of differensing is $d$.

• For instance with $d=2$:

• An $ARMA(p, q)$ model where $y_{t}$ is exchanged with $y_{t}^{[d]}$ would look like this:

• The constant is often omitted and assumed absorbed by the integration. Only the time series is integrated, not the errors.

• Reformulating this using the integration operator, we get an $ARIMA(p,d,q)$ model:

• This model has the properties of the ARMA model, but in addition does the differensing for us for stationarity.

• The terms can be reorganised to get predictions on the original scale instead of predicting difference values.

SARIMA - seasonal autoregressive integrated moving average models¶

• SARIMA shares some resemblance with the STL decomposition introduced previously.

• For seasons of length $s$, the seasonal part is obtained by applying an ARIMA model with lags, $P$ and $Q$, and integration time, $D$, that are multiples of $s$, i.e., if $P=2$, the included time points would be $t-1s$ and $t-2s$.

• After the seasonal part has been removed, another $ARIMA(p, d, q)$ is applied to $\Delta_{s}^{D} y_{t}$ which is equivalent to multiplying the two models together.

• The $SARIMA(p, d, q)(P, D, Q, s)$ model then becomes:

• This model has the ability to combine experience from previous timepoints with seasonal trends.

  • Or if one sets the ARIMA parameters $p=0$, $d=0$, $q=0$, one can have a pure seasonal model.

Exogenous variables¶

• As we showed in the previous chapter, it is possible to make a model purely on other variables measured at the same time, $t$.

• Including these variables into ARIMA and SARIMA, we get the ARIMAX and SARIMAX models.

• In practice, we paste on an extra coefficient vector, $\beta$, and variables, $X_{t}$, to the models, here noted as sums of products $\beta_{i} x^{i}_{t}$:

• $ARIMAX(p, d, q)$:

• $SARIMAX(p, d, q)(P, D, Q, s)$:

• The final model, thus includes autogression and moving averages, can perform differences, deals with seasonality and can leverage external variables from the same timepoint as the predictions.

Exercise¶

• Play with the OilExchange data.

• See if you can improve the fit of the model by adjusting lags and removing the least significant terms in the model.

Interpretation of Model Diagnostic Tests¶

Ljung-Box (L1) (Q)¶

• Purpose: Tests for autocorrelation in the residuals.

• Null hypothesis (H₀): The residuals are independently distributed (no autocorrelation up to lag L1).

• Result:

  • Q: test statistic

  • Prob(Q): >0.05 is good

Jarque-Bera (JB)¶

• Purpose: Tests whether the residuals are normally distributed (based on skewness and kurtosis).

• Null hypothesis (H₀): The residuals are normally distributed.

• Result:

  • JB: test statistic

  • Prob(JB): >0.05 is good

Heteroskedasticity (H)¶

• Purpose: Tests whether the residuals have constant variance (homoskedasticity).

• Null hypothesis (H₀): The residuals have constant variance.

• Result:

  • H: test statistic

  • Prob(H): >0.05 is good

Skew and Kurtosis¶

• Skew: Measures asymmetry of the residuals. 0 means symmetric; positive means right-skewed.

• Kurtosis: Measures “tailedness” of the residuals. 3 is normal; higher means heavier tails.

In practice:

• Independence and constant variance are more important for forecasting than perfect normality.

• Non-normality is common in real-world time series models.

SARIMAX prediction¶

• One-step-ahead prediction uses the estimated parameters and samples within the lags of the model to predict the next time point.

  • This means that the predictions will stay relatively close to the true values, never predicting more than one step away from the truth.

• Dynamic prediction predicts one step as above, then uses predicted values as input instead of true values.

  • This means the predictions can deviate from the truth over time.

Questercise¶

• Does the model made in the previous exercise improve the predictions as visualized above?