Maximum likelihood: from first principles to logistic regression

Probability and likelihood are not the same function

Suppose 20 patients undergo flexible bronchoscopy and 7 of them desaturate. The binomial density

\[P(Y = y \mid p) = \binom{n}{y} p^{y} (1-p)^{n-y}\]

can be read in two directions, and the difference is the whole idea.

Read as a probability, \(p\) is fixed and \(y\) varies: it tells you how often each possible count occurs. Read as a likelihood, \(y\) is fixed at what you actually observed and \(p\) varies: it tells you how well each candidate value of \(p\) accounts for the data you have. Same algebra, different argument.

The second reading is not a probability distribution. It does not integrate to one over \(p\), and it is not \(P(p \mid y)\) – that would need a prior. It is simply a measure of relative support, and Fisher’s proposal was to take its maximiser as the estimate.

y <- 7; n <- 20

lik <- tibble(p = seq(0, 1, length.out = 801)) |>
  mutate(L = dbinom(y, n, p))

p_hat <- y / n

ggplot(lik, aes(p, L)) +
  geom_area(fill = pal[["blue"]], alpha = 0.18) +
  geom_line(colour = pal[["blue"]], linewidth = 0.9) +
  geom_vline(xintercept = p_hat, linetype = "dashed",
             colour = pal[["orange"]], linewidth = 0.7) +
  annotate("point", x = p_hat, y = dbinom(y, n, p_hat),
           colour = pal[["orange"]], size = 3) +
  annotate("text", x = p_hat + 0.03, y = dbinom(y, n, p_hat),
           label = expression(hat(p) == 0.35), hjust = 0, size = 4,
           colour = pal[["orange"]]) +
  labs(title = "Likelihood of p, having observed 7 desaturations in 20",
       subtitle = "the data are fixed; the parameter varies",
       x = "p (probability of desaturation)",
       y = "L(p) = P(Y = 7 | p)")

The curve peaks at 0.35, which is just \(y/n\). That is reassuring but not very informative – the interesting part is the shape.

Note also the integral:

integrate(function(p) dbinom(y, n, p), 0, 1)$value
## [1] 0.04761905

It is \(1/(n+1)\), not 1. The likelihood is a function of the parameter, not a density over it.

Why the logarithm

Nobody maximises the likelihood itself. Three reasons, in increasing order of importance.

First, arithmetic. With independent observations the likelihood is a product, and a product of 600 numbers below 1 underflows to zero in double precision. The log turns it into a sum.

Second, the maximiser is unchanged – \(\log\) is strictly increasing, so \(\arg\max L = \arg\max \log L\).

Third, and the real reason: the log-likelihood is the function that is approximately quadratic. All the standard machinery – standard errors, Wald tests, likelihood ratio tests, the whole asymptotic apparatus – is built on that quadratic shape. On the raw likelihood scale it is not there.

ll <- tibble(p = seq(0.02, 0.98, length.out = 801)) |>
  mutate(logL = dbinom(y, n, p, log = TRUE))

ggplot(ll, aes(p, logL)) +
  geom_line(colour = pal[["blue"]], linewidth = 0.9) +
  geom_vline(xintercept = p_hat, linetype = "dashed",
             colour = pal[["orange"]], linewidth = 0.7) +
  annotate("point", x = p_hat, y = dbinom(y, n, p_hat, log = TRUE),
           colour = pal[["orange"]], size = 3) +
  coord_cartesian(ylim = c(-15, 0)) +
  labs(title = "The same information on the log scale",
       subtitle = "a single smooth maximum, and near it, a parabola",
       x = "p", y = expression(log~L(p)))

Score and information: the curve’s slope and curvature

Two derivatives carry everything.

The score is the first derivative of the log-likelihood:

\[U(p) = \frac{\partial \log L}{\partial p} = \frac{y}{p} - \frac{n-y}{1-p}\]

Setting it to zero gives the MLE. Here that is immediate: \(y(1-p) = (n-y)p\), so \(\hat p = y/n\).

The observed information is minus the second derivative:

\[I(p) = -\frac{\partial^2 \log L}{\partial p^2} = \frac{y}{p^2} + \frac{n-y}{(1-p)^2}\]

which at \(\hat p\) simplifies to \(n / (\hat p (1 - \hat p))\). Its reciprocal is the variance of the estimate, so

\[\mathrm{SE}(\hat p) = \sqrt{\frac{\hat p (1-\hat p)}{n}}\]

the familiar formula, derived rather than remembered. The interpretation is geometric: a sharply curved log-likelihood means the data strongly discriminate between nearby parameter values, which is exactly what a small standard error means.

score <- function(p) y/p - (n - y)/(1 - p)
info  <- function(p) y/p^2 + (n - y)/(1 - p)^2

c(score_at_mle = score(p_hat),
  info_at_mle  = info(p_hat),
  se_from_info = sqrt(1/info(p_hat)),
  se_textbook  = sqrt(p_hat * (1 - p_hat) / n))
## score_at_mle  info_at_mle se_from_info  se_textbook 
##    0.0000000   87.9120879    0.1066536    0.1066536

The quadratic approximation

Near the maximum, a second-order Taylor expansion gives

\[\log L(p) \approx \log L(\hat p) - \tfrac{1}{2} I(\hat p) (p - \hat p)^2\]

How good is that? It depends entirely on which scale you work on. Below, the same model is expanded on the probability scale and on the log-odds scale \(\beta = \log\{p/(1-p)\}\), where \(\log L(\beta) = y\beta - n\log(1 + e^{\beta})\) and \(I(\hat\beta) = n \hat p (1 - \hat p)\).

b_hat <- qlogis(p_hat)
ll_max <- dbinom(y, n, p_hat, log = TRUE)

# The two scales must carry the SAME additive constant or the panels will not
# be comparable: dbinom(log = TRUE) includes lchoose(n, y), the logit-scale
# kernel y*b - n*log(1 + e^b) does not. Omitting it shifts the log-odds curve
# down by lchoose(20, 7) = 11.26 and pushes it clean off the panel.
ll_b <- function(b) lchoose(n, y) + y * b - n * log1p(exp(b))

lv <- c("probability scale  (p)", "log-odds scale  (logit p)")

grid_p <- tibble(x = seq(0.10, 0.70, length.out = 400)) |>
  mutate(exact  = dbinom(y, n, x, log = TRUE),
         approx = ll_max - 0.5 * info(p_hat) * (x - p_hat)^2,
         scale  = factor(lv[1], levels = lv))

grid_b <- tibble(x = seq(qlogis(0.10), qlogis(0.70), length.out = 400)) |>
  mutate(exact  = ll_b(x),
         approx = ll_max - 0.5 * n * p_hat * (1 - p_hat) * (x - b_hat)^2,
         scale  = factor(lv[2], levels = lv))

both <- bind_rows(grid_p, grid_b)

long <- bind_rows(
  transmute(both, scale, x, value = exact,  curve = "exact log-likelihood"),
  transmute(both, scale, x, value = approx, curve = "quadratic approximation")
)

ggplot(long, aes(x, value, colour = curve, linetype = curve)) +
  geom_line(linewidth = 0.9) +
  facet_wrap(~ scale, scales = "free_x") +
  scale_colour_manual(values = unname(pal[c("blue", "magenta")])) +
  scale_linetype_manual(values = c("solid", "longdash")) +
  coord_cartesian(ylim = c(ll_max - 8, ll_max + 0.4)) +
  labs(title = "The quadratic approximation is a property of the scale, not the model",
       subtitle = "the approximation errs on both scales, but far less in the log-odds",
       x = NULL, y = "log-likelihood", colour = NULL, linetype = NULL) +
  theme(legend.position = "top")

Both panels show some error, so it is worth measuring rather than eyeballing. A parabola is symmetric about its apex; the log-likelihood need not be. Compare how much height is lost two standard errors below the maximum with how much is lost two standard errors above:

se_p <- sqrt(p_hat * (1 - p_hat) / n)
se_b <- sqrt(1 / (n * p_hat * (1 - p_hat)))

drop_p <- dbinom(y, n, p_hat + c(-2, 2) * se_p, log = TRUE) - ll_max
drop_b <- ll_b(b_hat + c(-2, 2) * se_b) - ll_max

c(p_scale_gap = abs(diff(drop_p)), logodds_gap = abs(diff(drop_b))) |> round(3)
## p_scale_gap logodds_gap 
##       1.053       0.348

A perfectly quadratic log-likelihood would give a gap of zero. The log-odds scale is about three times closer to that ideal, and the advantage grows as \(\hat p\) moves away from 0.5.

The probability scale also has a harder failure mode. With 2 events in 20, \(\hat p \pm 2\,\mathrm{SE}\) reaches below zero – the Wald interval leaves the parameter space entirely:

ph2 <- 2 / 20
ph2 + c(-1, 1) * qnorm(0.975) * sqrt(ph2 * (1 - ph2) / 20)
## [1] -0.03147838  0.23147838

The log-odds scale cannot do that: \(\mathrm{logit}^{-1}\) maps the whole real line into \((0, 1)\), so every interval comes back as a valid probability.

That is the real argument for parameterising logistic regression in log-odds. It is not that log-odds are intuitive – they are not. It is that the likelihood is close to Gaussian there and the parameter is unbounded, so symmetric Wald intervals, the normal approximation and the delta method all behave. Transform back to the probability scale after doing the inference, never before.

Finding the maximum: Newton–Raphson

For the binomial the MLE is closed-form. For almost everything else it is not, and we climb to it. Newton–Raphson uses the local quadratic: at the current guess, fit a parabola using the score and the information, jump to that parabola’s apex, repeat.

\[\beta_{k+1} = \beta_k + I(\beta_k)^{-1} U(\beta_k)\]

score_b <- function(b) y - n * plogis(b)
info_b  <- function(b) n * plogis(b) * (1 - plogis(b))

b <- 1.0                 # a deliberately poor start: p = 0.73
path <- tibble(iter = 0, beta = b, logL = ll_b(b),
               score = score_b(b), step = NA_real_)

for (k in 1:5) {
  step <- score_b(b) / info_b(b)
  b    <- b + step
  path <- bind_rows(path,
    tibble(iter = k, beta = b, logL = ll_b(b),
           score = score_b(b), step = step))
}

path |> mutate(across(where(is.numeric), \(x) round(x, 6)))
## # A tibble: 6 × 5
##    iter   beta  logL    score      step
##   <dbl>  <dbl> <dbl>    <dbl>     <dbl>
## 1     0  1     -8.01 -7.62    NA       
## 2     1 -0.938 -1.91  1.37    -1.94    
## 3     2 -0.598 -1.69 -0.0951   0.340   
## 4     3 -0.619 -1.69 -0.00029 -0.0208  
## 5     4 -0.619 -1.69  0       -0.000064
## 6     5 -0.619 -1.69  0        0

Four steps. The score collapses from -7.62 to effectively zero, and beta lands on \(\log(7/13) =\) -0.619039 to six decimals.

curve_b <- tibble(beta = seq(-2.2, 1.4, length.out = 600)) |>
  mutate(logL = ll_b(beta))

ggplot(curve_b, aes(beta, logL)) +
  geom_line(colour = pal[["grey"]], linewidth = 0.8) +
  geom_hline(yintercept = ll_max, linetype = "dotted",
             colour = pal[["blue"]]) +
  geom_path(data = path, colour = pal[["orange"]],
            linewidth = 0.6, linetype = "dashed",
            arrow = arrow(length = unit(0.14, "cm"), type = "closed")) +
  geom_point(data = path, aes(fill = factor(iter)),
             shape = 21, colour = pal[["orange"]], size = 3.6, stroke = 0.7) +
  # iterations 2-5 land on top of one another, so the labels must be repelled
  ggrepel::geom_text_repel(
    data = path, aes(label = iter), colour = pal[["orange"]], size = 3.4,
    nudge_y = -1.1, direction = "both", seed = 1,
    segment.size = 0.25, segment.colour = "grey70",
    min.segment.length = 0.25, box.padding = 0.3, show.legend = FALSE) +
  scale_fill_grey(start = 1, end = 0.35, guide = "none") +
  labs(title = "Newton--Raphson climbing the log-likelihood",
       subtitle = "numbered by iteration; the first step overshoots, then it converges quadratically",
       x = expression(beta == logit(p)), y = "log-likelihood")

The first jump overshoots the maximum – the parabola fitted at \(p = 0.73\) is too narrow – but from there convergence is quadratic, roughly doubling the number of correct digits per step. glm() runs exactly this algorithm, dressed up as iteratively reweighted least squares: each Newton step for a generalised linear model can be written as a weighted least-squares fit, which is why the output reports a working weight matrix rather than a Hessian.

From one parameter to a regression

Nothing conceptual changes when \(p\) becomes a function of covariates. We model

\[\log \frac{p_i}{1 - p_i} = x_i^{\top}\beta, \qquad p_i = \mathrm{logit}^{-1}(x_i^{\top}\beta)\]

and the Bernoulli log-likelihood becomes

\[\log L(\beta) = \sum_i \left\{ y_i x_i^{\top}\beta - \log\left(1 + e^{x_i^{\top}\beta}\right) \right\}\]

The score and information generalise to a vector and a matrix:

\[U(\beta) = X^{\top}(y - p), \qquad I(\beta) = X^{\top} W X, \quad W = \mathrm{diag}\{p_i(1-p_i)\}\]

That is the whole of logistic regression. The score equations \(X^{\top}(y-p) = 0\) say the fitted probabilities must reproduce the observed totals in every direction spanned by the covariates, and \((X^{\top}WX)^{-1}\) is the covariance matrix of \(\hat\beta\).

The likelihood surface

With two parameters the log-likelihood is a surface, and the contours make the correlation between estimates visible.

set.seed(20261118)
nn   <- 600
agec <- round(rnorm(nn, 64, 11)) - 64
yy   <- rbinom(nn, 1, plogis(-1.1 + 0.035 * agec))

loglik2 <- function(b0, b1) {
  eta <- b0 + b1 * agec
  sum(yy * eta - log1p(exp(eta)))
}

fit2  <- glm(yy ~ agec, family = binomial)
mle2  <- coef(fit2)
llmax <- as.numeric(logLik(fit2))

surf <- expand.grid(b0 = seq(mle2[1] - 0.7, mle2[1] + 0.7, length.out = 140),
                    b1 = seq(mle2[2] - 0.045, mle2[2] + 0.045, length.out = 140)) |>
  as_tibble() |>
  rowwise() |>
  mutate(dev = 2 * (llmax - loglik2(b0, b1))) |>
  ungroup()

ggplot(surf, aes(b0, b1)) +
  geom_contour_filled(aes(z = dev),
                      breaks = c(0, 1, 2.3, 4, 6, 9.21, 14, 20, 30)) +
  geom_contour(aes(z = dev), breaks = qchisq(0.95, 2),
               colour = pal[["orange"]], linewidth = 0.9) +
  annotate("point", x = mle2[1], y = mle2[2], colour = "white",
           size = 3.2, shape = 4, stroke = 1.4) +
  scale_fill_ordinal(name = "deviance from\nthe maximum") +
  labs(title = "Log-likelihood surface for a two-parameter logistic model",
       subtitle = "heavy line = joint 95% region; x marks the MLE",
       x = expression(beta[0]~"(intercept)"),
       y = expression(beta[1]~"(age, centred)"))

The contours are only slightly tilted, and that is because age was centred at 64 before fitting. The tilt is the correlation between the two estimates:

fit_raw <- glm(yy ~ I(agec + 64), family = binomial)
c(centred = cov2cor(vcov(fit2))[1, 2],
  raw_age = cov2cor(vcov(fit_raw))[1, 2]) |> round(3)
## centred raw_age 
##  -0.222  -0.988

Centred, the estimates are nearly independent. On the raw age scale the correlation is almost \(-1\), and the same 95% region becomes a thin diagonal sliver: raising the slope and lowering the intercept to compensate leaves the fit essentially unchanged, so the data cannot separate the two. The likelihood there is a long flat ridge rather than a well-defined peak.

This is worth internalising, because it is the same geometry that produces alarming standard errors from uncentred polynomial and interaction terms. The fit is identical – predictions, deviance and \(C\) do not move – but the parameterisation is badly conditioned, and the information matrix is close to singular.

A clinical toy dataset

Something concrete to fit: 600 simulated bronchoscopies with desaturation as the outcome. The data-generating truth gives FEV1 a deliberately curved effect – risk is high and flat in severe obstruction, falls steeply through the middle of the range, then flattens again at normal lung function – plus a linear age effect, a sex effect, and BMI as a genuine null variable.

set.seed(20261118)
n_pt <- 600

age  <- round(rnorm(n_pt, 64, 11))
sex  <- factor(sample(c("female", "male"), n_pt, TRUE, prob = c(0.45, 0.55)))
fev1 <- round(pmin(125, pmax(25, rnorm(n_pt, 78, 18))))
bmi  <- round(rnorm(n_pt, 27.5, 4.5), 1)

# A sigmoid in fev1, not a quadratic. A quadratic whose vertex falls inside the
# observed range would make risk *fall* again at very low FEV1 - clinically
# backwards, however convenient the algebra.
eta <- -1.9 + 0.032 * (age - 64) + 0.70 * (sex == "male") +
        2.2 * plogis(-(fev1 - 65) / 9) + 0.030 * (bmi - 27.5)

d <- tibble(hypox = rbinom(n_pt, 1, plogis(eta)),
            age, sex, fev1, bmi)

d
## # A tibble: 600 × 5
##    hypox   age sex     fev1   bmi
##    <int> <dbl> <fct>  <dbl> <dbl>
##  1     0    63 male      70  20.6
##  2     1    76 female    76  28.2
##  3     1    59 female    51  28.4
##  4     0    65 male     123  20.6
##  5     1    75 male      51  24.1
##  6     1    77 male      84  32.9
##  7     0    68 female    85  24.2
##  8     0    72 female    97  26.1
##  9     1    67 female    59  19.7
## 10     0    82 male      98  30.1
## # ℹ 590 more rows
c(events = sum(d$hypox), n = nrow(d), prevalence = mean(d$hypox))
##      events           n  prevalence 
## 187.0000000 600.0000000   0.3116667

187 events. Harrell’s rule of thumb is roughly 15 events per candidate parameter before you can trust a model without penalisation; with 187 events we can afford the six parameters below, but not many more, and certainly not a round of stepwise selection on top.

ggplot(d, aes(fev1, hypox)) +
  geom_point(aes(colour = sex), alpha = 0.35,
             position = position_jitter(height = 0.035, width = 0.4)) +
  geom_smooth(method = "loess", span = 0.9, se = TRUE,
              colour = pal[["magenta"]], fill = pal[["magenta"]],
              alpha = 0.15, linewidth = 0.9) +
  scale_colour_manual(values = unname(pal[c("blue", "green")])) +
  labs(title = "Desaturation against FEV1 (% predicted)",
       subtitle = "a nonparametric smoother, before any model is imposed",
       x = "FEV1 (% predicted)", y = "desaturation (jittered)",
       colour = NULL)

The smoother bends. A model with fev1 entered linearly would be fitting a straight line through that.

Fitting it Harrell’s way with rms

Three things distinguish the rms idiom from a bare glm() call.

Flexibility by default. A continuous predictor goes in as a restricted cubic spline unless there is a reason to force linearity. Four knots costs three degrees of freedom and buys the ability to detect curvature instead of assuming it away. The restriction – linear beyond the outer knots – keeps the tails from flying off.

datadist. rms stores the distribution of every predictor up front, so that later calls know what “holding the other variables fixed” should mean (medians for continuous, modes for categorical) and what range to plot over.

Everything downstream for free. One fit object supports anova, summary, Predict, validate, calibrate and nomogram without repeating yourself.

# rms stores only the NAME of the datadist object and looks it up in the global
# environment. knitr evaluates each chunk in its own environment, so inside an
# .Rmd this assignment is required - without it lrm() fails with
# "dataset dd not found for options(datadist=)".
dd <- datadist(d)
assign("dd", dd, envir = globalenv())
options(datadist = "dd")

f <- lrm(hypox ~ rcs(fev1, 4) + age + sex + bmi,
         data = d, x = TRUE, y = TRUE)
f
## Logistic Regression Model
## 
## lrm(formula = hypox ~ rcs(fev1, 4) + age + sex + bmi, data = d, 
##     x = TRUE, y = TRUE)
## 
##                        Model Likelihood      Discrimination    Rank Discrim.    
##                              Ratio Test             Indexes          Indexes    
## Obs           600    LR chi2     103.63      R2       0.223    C       0.751    
##  0            413    d.f.             6      R2(6,600)0.150    Dxy     0.501    
##  1            187    Pr(> chi2) <0.0001    R2(6,386.2)0.223    gamma   0.501    
## max |deriv| 4e-05                            Brier    0.180    tau-a   0.215    
## 
##           Coef    S.E.   Wald Z Pr(>|Z|)
## Intercept -0.3565 1.4155 -0.25  0.8011  
## fev1      -0.0469 0.0191 -2.45  0.0142  
## fev1'     -0.0577 0.0565 -1.02  0.3069  
## fev1''     0.4486 0.2538  1.77  0.0772  
## age        0.0377 0.0091  4.13  <0.0001 
## sex=male   0.9473 0.2023  4.68  <0.0001 
## bmi        0.0124 0.0210  0.59  0.5541

max |deriv| in that header is the largest absolute element of the score vector at the reported solution. It should be a very small number: it is rms telling you the optimiser genuinely reached a stationary point rather than stopping somewhere near one. Section 3 is what that line is reporting on.

The Wald table from anova.rms is organised by predictor, not by coefficient, which is what you want once a variable occupies several columns of the design matrix. The Nonlinear line is the joint test of the two spline terms beyond the linear one.

ad <- as.data.frame(unclass(anova(f)))
names(ad) <- c("chisq", "df", "p")

ad_plot <- ad |>
  tibble::rownames_to_column("term") |>
  filter(term != "TOTAL") |>
  mutate(nonlinear = grepl("Nonlinear", term),
         term  = trimws(term),
         label = ifelse(nonlinear, "fev1: nonlinear component", term),
         stat  = chisq - df)

ggplot(ad_plot, aes(stat, reorder(label, stat))) +
  geom_segment(aes(xend = 0, yend = reorder(label, stat)),
               colour = pal[["grey"]], linewidth = 0.5) +
  geom_point(aes(colour = nonlinear), size = 4) +
  scale_colour_manual(values = unname(pal[c("blue", "magenta")]),
                      guide = "none") +
  labs(title = "Relative predictive strength",
       subtitle = expression(Wald~chi^2~minus~its~degrees~of~freedom),
       x = NULL, y = NULL)

Subtracting the degrees of freedom is Harrell’s device for making a 3-df spline comparable with a 1-df binary term: it is the chi-square penalised by what it cost to get. FEV1 dominates by a wide margin, age and sex are real but modest, and BMI sits at the bottom – correctly, since we generated it as noise.

Partial effects

Predict() varies one predictor across its observed range while holding the others at their datadist reference values. It returns a plain data frame, so the plotting is ordinary ggplot work.

pr <- as.data.frame(Predict(f, fev1, fun = plogis))

ggplot(pr, aes(fev1, yhat)) +
  geom_ribbon(aes(ymin = lower, ymax = upper),
              fill = pal[["blue"]], alpha = 0.20) +
  geom_line(colour = pal[["blue"]], linewidth = 1) +
  geom_rug(data = d, aes(x = fev1, y = NULL), sides = "b",
           alpha = 0.25, colour = pal[["grey"]]) +
  scale_y_continuous(labels = scales::percent) +
  labs(title = "Fitted risk of desaturation across FEV1",
       subtitle = "restricted cubic spline, 4 knots; other predictors at their reference values",
       x = "FEV1 (% predicted)", y = "probability of desaturation")

The spline recovers the shape we built in: a high plateau below roughly 55% predicted, a steep fall through the middle of the range, and a low plateau above about 90%. A linear term would have averaged those three regimes into a single slope that is wrong in all of them.

It also does something we did not build in. Above about 100% predicted the fitted curve turns upward again. There is no such effect in the data-generating model – it is the spline chasing noise where there is very little data, and the honest signal that this is happening is the ribbon, which balloons over exactly that stretch. Read the band, not the line.

Two things widen it in the tails. The information matrix \(X^{\top}WX\) is thin on observations there; and \(W = p(1-p)\) shrinks as fitted probabilities approach 0 or 1, so each extreme patient contributes less information than their rarity alone would suggest. This is the practical argument for not reading spline fits beyond the inner knots, and for preferring fewer knots when events are scarce.

one_var <- function(var) {
  as.data.frame(Predict(f, name = var, fun = plogis)) |>
    transmute(x = .data[[var]], yhat, lower, upper, var = var)
}

grid_all <- bind_rows(lapply(c("fev1", "age", "bmi"), one_var))

ggplot(grid_all, aes(x, yhat)) +
  geom_ribbon(aes(ymin = lower, ymax = upper),
              fill = pal[["blue"]], alpha = 0.18) +
  geom_line(colour = pal[["blue"]], linewidth = 0.9) +
  facet_wrap(~ var, scales = "free_x") +
  scale_y_continuous(labels = scales::percent) +
  labs(title = "All continuous partial effects on one page",
       subtitle = "age is linear by specification; bmi is flat because it is noise",
       x = NULL, y = "probability of desaturation")

Effect sizes as odds ratios

summary.rms reports an inter-quartile-range effect by default rather than a per-unit one. For a spline term a per-unit odds ratio would be meaningless – the slope is different everywhere – so the question becomes “how much does risk change between a typical low and a typical high value”, which is what the IQR contrast answers.

sm   <- unclass(summary(f))
keep <- which(sm[, "Type"] == 2)          # the odds-ratio rows

or <- tibble(term = trimws(rownames(sm)[keep - 1]),
             low  = sm[keep, "Low"],
             high = sm[keep, "High"],
             or   = sm[keep, "Effect"],
             lo   = sm[keep, "Lower 0.95"],
             hi   = sm[keep, "Upper 0.95"]) |>
  mutate(label = sprintf("%s  (%g to %g)", term, low, high))

ggplot(or, aes(or, reorder(label, or))) +
  geom_vline(xintercept = 1, linetype = "dashed", colour = pal[["grey"]]) +
  geom_pointrange(aes(xmin = lo, xmax = hi), colour = pal[["blue"]],
                  linewidth = 0.7, size = 0.6) +
  scale_x_log10() +
  labs(title = "Inter-quartile-range odds ratios",
       subtitle = "log scale, so that an odds ratio and its reciprocal are equidistant from 1",
       x = "odds ratio", y = NULL)

The log scale is not decoration. Odds ratios are multiplicative, their sampling distribution is symmetric on the log scale, and only there does an OR of 2 look like the same amount of evidence as an OR of 0.5.

Three ways to test the same thing

Having a likelihood gives three asymptotically equivalent tests, which differ in what they require and in how they behave at finite sample size.

Test What it uses Needs
Wald curvature at \(\hat\beta\) one fit
Likelihood ratio height difference between maxima two fits
Score slope at the null one fit, under \(H_0\)

The Wald test is cheapest and the least trustworthy: it reads the curvature only at the maximum, so a skewed log-likelihood misleads it. The likelihood ratio test compares the actual heights and is the one to prefer. Harrell’s lrtest() does it directly.

f_lin <- lrm(hypox ~ fev1 + age + sex + bmi, data = d)   # force linearity
lrtest(f_lin, f)
## 
## Model 1: hypox ~ fev1 + age + sex + bmi
## Model 2: hypox ~ rcs(fev1, 4) + age + sex + bmi
## 
##   L.R. Chisq         d.f.            P 
## 10.693405157  2.000000000  0.004763833

Compare that with the Wald Nonlinear row from anova(f):

nl <- ad[grepl("Nonlinear", rownames(ad)), ]
c(wald_chisq = nl$chisq, wald_p = nl$p)
##   wald_chisq       wald_p 
## 11.749642609  0.002809296

Same conclusion here, different numbers. When they disagree materially, believe the likelihood ratio.

Profile likelihood

The honest confidence interval for a single coefficient comes from the likelihood itself: fix that coefficient at a candidate value, re-maximise over all the others, and record how much log-likelihood you had to give up. Keep the values that cost less than \(\chi^2_{1,0.95}/2 = 1.92\).

The re-maximisation is done by pushing the fixed term into the offset.

X  <- model.matrix(~ rcs(fev1, 4) + age + sex + bmi, data = d)
j  <- which(colnames(X) == "age")
b_age  <- coef(f)[["age"]]
se_age <- sqrt(diag(vcov(f)))[["age"]]

prof_ll <- function(b) {
  m <- glm.fit(X[, -j, drop = FALSE], d$hypox,
               offset = b * X[, j], family = binomial())
  -m$deviance / 2            # for 0/1 data the saturated log-likelihood is 0
}

ll_full <- -f$deviance[2] / 2

prof <- tibble(b = seq(b_age - 4 * se_age, b_age + 4 * se_age,
                       length.out = 241)) |>
  rowwise() |>
  mutate(profile = 2 * (ll_full - prof_ll(b))) |>
  ungroup() |>
  mutate(wald = ((b - b_age) / se_age)^2)

prof_long <- bind_rows(
  transmute(prof, b, value = profile, curve = "profile deviance"),
  transmute(prof, b, value = wald,    curve = "Wald (quadratic)")
)

cut <- qchisq(0.95, 1)

ggplot(prof_long, aes(b, value, colour = curve, linetype = curve)) +
  geom_line(linewidth = 0.9) +
  geom_hline(yintercept = cut, colour = pal[["orange"]],
             linetype = "dotted", linewidth = 0.7) +
  annotate("text", x = b_age - 4 * se_age, y = cut + 0.55, hjust = 0,
           label = "chi-squared(1) 95% cutoff = 3.84", size = 3.4,
           colour = pal[["orange"]]) +
  scale_colour_manual(values = unname(pal[c("blue", "magenta")])) +
  scale_linetype_manual(values = c("solid", "longdash")) +
  coord_cartesian(ylim = c(0, 16)) +
  labs(title = "Profile deviance for the age coefficient",
       subtitle = "where the curves separate, the Wald interval is the wrong one",
       x = expression(beta[age]), y = "deviance from the maximum",
       colour = NULL, linetype = NULL) +
  theme(legend.position = "top")

root <- function(lo, hi) {
  uniroot(function(b) 2 * (ll_full - prof_ll(b)) - cut,
          lower = lo, upper = hi, tol = 1e-8)$root
}

rbind(profile = c(root(b_age - 6*se_age, b_age), root(b_age, b_age + 6*se_age)),
      wald    = b_age + c(-1, 1) * qnorm(0.975) * se_age) |>
  round(4)
##           [,1]   [,2]
## profile 0.0200 0.0559
## wald    0.0198 0.0556

With 187 events and a well-behaved predictor the two intervals nearly coincide – the quadratic approximation is doing its job. The divergence grows with small samples, sparse cells and coefficients near the boundary of separation, and the profile version is the one that stays correct.

What the likelihood does not tell you

Maximum likelihood answers one question: which parameter values best account for this dataset. It says nothing about how the model will do on the next one. The apparent \(C\) of 0.751 is an in-sample number, and it is optimistic by construction.

validate() quantifies that optimism by bootstrap: refit on each resample, evaluate on the original data, and average the gap.

set.seed(314)
v <- validate(f, B = 300)
round(v[c("Dxy", "R2", "Slope", "Emax"), ], 4)
##       index.orig training   test optimism index.corrected   Lower  Upper   n
## Dxy       0.5011   0.5102 0.4881   0.0221          0.4790  0.3951 0.5581 300
## R2        0.2231   0.2342 0.2135   0.0207          0.2024  0.1277 0.2697 300
## Slope     1.0000   1.0000 0.9470   0.0530          0.9470  0.7102 1.2051 300
## Emax      0.0000   0.0000 0.0407  -0.0407          0.0407 -0.0082 0.1115 300
dxy_corr <- v["Dxy", "index.corrected"]
c(C_apparent = f$stats[["C"]], C_corrected = dxy_corr / 2 + 0.5)
##  C_apparent C_corrected 
##   0.7505406   0.7394880

The corrected Slope below 1 is the calibration-slope shrinkage: the model’s predictions are slightly too extreme, exactly the overfitting you would expect from estimating six parameters from 187 events.

set.seed(315)
cal <- calibrate(f, B = 300)

cal_df <- as.data.frame(unclass(cal)[, c("predy", "calibrated.orig",
                                         "calibrated.corrected")])

cal_long <- bind_rows(
  transmute(cal_df, predy, actual = calibrated.orig,
            curve = "apparent"),
  transmute(cal_df, predy, actual = calibrated.corrected,
            curve = "bias-corrected (bootstrap)")
)

ggplot(cal_long, aes(predy, actual, colour = curve, linetype = curve)) +
  geom_abline(slope = 1, intercept = 0, colour = pal[["grey"]],
              linetype = "dashed") +
  geom_line(linewidth = 0.9) +
  geom_rug(data = d, inherit.aes = FALSE,
           aes(x = predict(f, type = "fitted")), sides = "b",
           alpha = 0.2, colour = pal[["grey"]]) +
  scale_colour_manual(values = unname(pal[c("blue", "magenta")])) +
  scale_linetype_manual(values = c("solid", "longdash")) +
  scale_x_continuous(labels = scales::percent) +
  scale_y_continuous(labels = scales::percent) +
  coord_equal() +
  labs(title = "Calibration: predicted versus observed risk",
       subtitle = "dashed = perfect; the gap between the two curves is the optimism",
       x = "predicted probability", y = "observed proportion",
       colour = NULL, linetype = NULL) +
  theme(legend.position = "top")

The vertical distance between the two curves is the overfitting, made visible. Nothing in the likelihood could have shown you that, because the likelihood was maximised on these same 600 patients. You need resampling, and rms makes it a one-liner.

A calibration curve is also the right place to look before anyone starts talking about thresholds. A model that discriminates well but calibrates badly will produce decision rules that are wrong in a way that no choice of cutoff repairs – which is the subject of the companion post on MLE and the optimal classification threshold.

Closing

The thread running through all of this: the log-likelihood is a surface, its peak is the estimate, its curvature is the precision, and its height is the currency of model comparison. Everything else – score equations, standard errors, Wald and likelihood ratio tests, profile intervals, IRLS – is a statement about the geometry of that surface.

What that leaves out is equally worth stating. The likelihood is maximised on the data in hand, so it cannot speak to out-of-sample performance; it assumes the model class contains something reasonable; and it is silent on the decision problem. rms is built around those gaps: splines so the model class is wide enough, validate and calibrate so the optimism is measured rather than hoped away, and effects reported as interpretable contrasts rather than raw coefficients.

References

  • Harrell FE. Regression Modeling Strategies, 2nd ed. Springer, 2015. Chapters 9–10.
  • Harrell FE. Biostatistics for Biomedical Research, hbiostat.org.
  • Fisher RA. On the mathematical foundations of theoretical statistics. Phil Trans R Soc A 1922;222:309–368.
  • Cox DR, Hinkley DV. Theoretical Statistics. Chapman & Hall, 1974. Chapter 9.
  • Venzon DJ, Moolgavkar SH. A method for computing profile-likelihood-based confidence intervals. Appl Stat 1988;37:87–94.