Fix/inference dof serialization - #81
Open
ArdiaD wants to merge 7 commits into
Open
Conversation
f_InferenceFun built the delta-method sandwich as t(J) V J. numDeriv::jacobian returns d f_i / d x_j, so the variance of the natural parameters is J V t(J). The two agree only when J is symmetric, which it is not: the working-to-natural map is triangular inside each regime (the sGARCH bound on beta is 0.9999 - alpha1; the gjrGARCH and tGARCH bounds on beta also involve alpha2 and the shape/skew parameters) and the transition-probability block is anti-diagonal. On the MS(2)-GARCH(1,1)-Normal fit to SMI, six of the eight reported standard errors were wrong, by factors from 0.06 to 16, and the two transition probabilities had their standard errors exchanged. The corrected values agree with the observed information computed directly in the natural parameterisation to within 2%, where the old ones were off by up to 85%. The Pr(>|t|) column reported 1 - pnorm(abs(t)), i.e. half a two-sided p-value under a two-sided label.
A MSGARCH_SPEC holds Rcpp module objects. R serializes external pointers as NULL, so a spec or fit written with saveRDS comes back with dead pointers and every method on it fails with "NULL value passed as symbol address". That is what f_check_spec is for, but its recovery branch called spec$rcpp.func$get_mean() and get_sd() -- the very pointer whose failure had just triggered the branch -- so the rebuild could never run. The two values it read there are already kept on the R side, in spec$prior.mean and spec$prior.sd, which the next two lines were using anyway; the C++ round trip was dead code. Dropping it makes the rebuild work: a reloaded spec, ML fit or MCMC fit now returns exactly the same volatilities, state probabilities, forecasts and information criteria as before it was saved, with user-supplied priors preserved. This affects the ordinary workflow of fitting a model, saving it, and analysing it in a later session, and of sending a spec to a parallel worker.
Setting constraint.spec$regime.const ties a parameter across all K regimes, so it leaves one free value where there were K and removes K - 1 degrees of freedom (see f_rename_par, which strips name_2 ... name_K). dofMSGARCH subtracted one per constrained name, which is right only at K = 2. With a regime-constant shape parameter the reported df was 17 instead of 16 at K = 3 and 27 instead of 25 at K = 4, so stats::AIC and stats::BIC over- penalised the constrained model in every K >= 3 model-selection table.
In f_CondVol, vol is a (T + 1) x ndraw matrix and the one-step-ahead value was taken as vol[dim(PredProb)[1]]. A single index into a matrix is linear indexing, so that is the last row of the first column: predict() on a MSGARCH_MCMC_FIT reported the forecast of MCMC draw #1 rather than the posterior mean, and mean() of the resulting scalar was a no-op. On a 100-draw chain fitted to SMI the reported value was 1.019753, the first draw, against a posterior mean of 1.039776 and a range across draws of 0.999837 to 1.097337. Volatility() already averaged across draws correctly, so the two methods disagreed on the same fit. Adding the missing comma leaves the single-parameter (ML) path untouched.
test_Inference.R (new) pins the delta method against an independently computed observed information. It is anchored on a single-regime GARCH(1,1)-Normal because every parameter there is interior, so a central-difference Hessian of the natural-scale negative log-likelihood is well conditioned: it agrees with J V t(J) to 1e-4 in relative terms while the transposed sandwich is off by 44% and 85%. It also checks the exact sandwich on a two-regime fit, where the transition-probability block is the part that gets exchanged, that Pr(>|t|) is two-sided, and that dofMSGARCH and the AIC/BIC arithmetic drop K - 1 values per regime-constant parameter for K = 2, 3 and 4. test_Serialization.R (new) round-trips a spec, an ML fit and an MCMC fit through saveRDS/readRDS. R restores external pointers as NULL, so this reproduces the cross-session failure inside a single session, and the tests require Volatility, State, predict, AIC, summary, DIC and the log-kernel to return values identical to the originals, with user priors preserved. test_Volatility.R gains one block that recomputes the one-step-ahead volatility draw by draw through the public interface -- each such call carries a single parameter vector, so it cannot depend on how the draws are pooled -- and requires predict() on the MCMC fit to equal their mean. Every block opens with a guard asserting its own precondition (the two sandwich orientations really differ; the round trip really did invalidate the pointers; the posterior mean really differs from the first draw) so that none of them can pass vacuously if the surrounding code changes. All eleven blocks fail on the unpatched package.
Also refreshes the Date field, which R CMD check --as-cran flags as stale together with the unchanged version number. Drop this commit if the release number should be decided separately; nothing else in the branch depends on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five bugs in the reporting and persistence layer around the likelihood, plus regression
tests for each. The likelihood itself is not touched: no estimate, log-likelihood,
conditional variance or state probability changes anywhere in this branch. What changes is
what
summary()prints, whatAIC/BICcount, whatpredict()returns for a Bayesianfit, and whether a saved fit can be reloaded at all.
Everything below is reproduced on
data("SMI")with the shipped code.R CMD check --as-cranis unchanged by the branch (2 WARNINGs, 1 NOTE,testthatOK — and the versionbump in the last commit clears one of the two WARNINGs).
1. Standard errors used the delta method transposed —
R/Inference.R:38numDeriv::jacobianreturns∂f_i/∂x_j, soVar(g(θ̂)) = J V J'. The two orientationsagree only if
Jis symmetric, and it is not: the working→natural map is triangular insideeach regime (the sGARCH bound on
betais0.9999 − alpha1; the gjrGARCH and tGARCH boundson
betaalso involvealpha2and the shape/skew parameters), and the transition-probabilityblock is anti-diagonal.
On the default MS(2)-GARCH(1,1)-Normal fit to
SMI, checked against the observed informationcomputed directly in the natural parameterisation:
HJ V J')J' V J)alpha1_1beta_1alpha1_2beta_2P_1_1P_2_1Six of eight were wrong, by factors from 0.06× to 16×. Because the P block is anti-diagonal,
transposing exchanged the two transition probabilities' standard errors. Only
alpha0_k,whose map is a plain
exp, was unaffected.2.
Pr(>|t|)was one-sided —R/Inference.R:431 - pnorm(abs(t))under a two-sided label; now2 * (1 - pnorm(abs(t))).3.
AIC/BICmis-countedregime.const.pars—R/Utils.R:420A regime-constant parameter leaves one free value where there were
K, so it removesK − 1degrees of freedom (cf.
f_rename_par, which stripsname_2 … name_K).dofMSGARCHsubtracted one, which is right only at
K = 2: with a regime-constant shape parameter the dfwas 17 instead of 16 at
K = 3and 27 instead of 25 atK = 4, always over-penalising theconstrained model.
4. Saved specs and fits could not be reloaded —
R/Utils.R:349-353R serializes external pointers as
NULL, so asaveRDS-ed spec or fit comes back with deadpointers.
f_check_specexists to rebuild them, but the rebuild branch calledspec$rcpp.func$get_mean()/get_sd()— the very pointer whose failure had just triggeredthe branch:
The two values read there are already held on the R side in
spec$prior.mean/spec$prior.sd, which the next two lines were using anyway, so the C++ round trip was deadcode. Removing it makes the rebuild work; a reloaded spec, ML fit or MCMC fit now returns
values identical to before it was saved, with user priors preserved.
This is the ordinary workflow of fitting a model, saving it, and analysing it in a later
session — or shipping a spec to a
parLapplyworker.5.
predict()on an MCMC fit returned draw #1 —R/CondVol.R:32volis(T+1) × ndrawand the one-step-ahead value wasvol[dim(PredProb)[1]]— a singleindex into a matrix is linear indexing, i.e. the last row of the first column. On a
100-draw chain fitted to
SMIthe reported value was 1.019753 (the first draw) against aposterior mean of 1.039776, with a spread of 0.999837–1.097337 across draws.
Volatility()already averaged correctly, so the two methods disagreed on the same fit. The
single-parameter (ML) path is unchanged.
Tests
test_Inference.Randtest_Serialization.Rare new;test_Volatility.Rgains one block.11 blocks, 34 assertions, +13s of check time. All 11 fail on the current code.
Two things worth pointing out, since they are what makes the tests worth having:
MS(2) default. Every parameter there is interior, so a central-difference Hessian of the
natural-scale negative log-likelihood is well conditioned: it agrees with
J V J'to 1e-4in relative terms while the transposed sandwich is off by 44% and 85%. That is an
independent check on the value, not a restatement of the formula.
orientations really differ for this model; the round trip really did invalidate the
pointers; the posterior mean really differs from the first draw — so none of them can pass
vacuously if the surrounding code changes.
The
K = 3andK = 4legs of the degrees-of-freedom test run against a fit-shaped listrather than a real fit, because a constrained
K ≥ 3model cannot currently be fitted atall (see below).
Not in this branch
The last commit (version bump +
NEWS) is separable — drop it if the release number shouldbe decided elsewhere; nothing depends on it.
A review of the package turned up nine further issues that are not addressed here,
several of them more serious than some of the above. The two worth flagging now:
src/MSgarch.h:283computes the ergodic distribution with a raw Armadillo.i()onI − P + U, on every likelihood evaluation. Whendo.plm = TRUE— forced byfixed.parsand
regime.const.pars, and hard-coded inFitMCMC— the free transition entries aremapped into
(0,1)independently, so forK ≥ 3a row can leave the simplex, the matrixcan be exactly singular, and the uncaught exception aborts the entire run. Reproducible:
FitMLon aK = 3spec withregime.const = "nu"dies atP = [[1, 0.999973, −0.999973], [0,1,0], [0,0,1]]. This is why the tests above cannot fita constrained
K ≥ 3model.R/FitML.R:133guards optimisation failure withif (llk == 1e+10), butf_nllreturns
+1e10sollkis−1e10; andf_OptimFUNDefaultwrapsoptimintry(), sooptimizer$valueerrors first. Every failure mode — including a singleNAin the data,which
f_check_ylets through — therefore surfaces asError in optimizer$value : $ operator is invalid for atomic vectors.Happy to open these as separate issues or as a follow-up PR, whichever you prefer.