fr_male <- fr_mortality |> filter(Sex == "Male") |> select(-Sex)13 Functional data
Functional data occur when the natural observational representation is a function. The most commonly occurring examples are where observations are curves (functions over one dimension) or surfaces (functions over two dimensions). For example, mortality rates are naturally represented as 1-dimensional functions of age, and surface temperatures are naturally represented as functions over the 2-dimensional plane of latitude and longitude. For an introduction to functional data analysis, with many worked examples, see Ramsay & Silverman (2002).
Figure 13.1 shows French male mortality rates between 1816 and 1999. In Section 12.2, we looked at these as time series, treating each age as a separate series. Here, we will treat the same data as curves, with each year represented as a function of mortality rates over age. Each line in Figure 13.1 denotes the mortality rates for one year as a function of age, with the colours in rainbow order corresponding to the years of observation. We see a large decrease in mortality rates during early childhood years, then an increase during teenage years. After about age 30, the rates increase almost linearly on a log scale. Comparing the curves over time, we see that the rates have steadily fallen for all ages, with more than a 40-fold reduction in mortality rates at around age 10. The noticeable gap between the curves occurred after World War II due to the effective use of antibiotics, and widespread vaccinations for diphtheria, tetanus, whooping cough and tuberculosis, followed by polio in the late 1950s.
Code
fr_male |>
ggplot() +
aes(x = Age, y = Mortality, colour = Year, group = Year) +
geom_line() +
scale_colour_gradientn(colours = rainbow(100)[1:80]) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate")
As we are interested in anomalies, we will mostly focus on the unusual curves. There are about 10–15 years when the mortality rate for 20–40 year olds is much higher than in other years. These are the war years. Any anomaly detection method for functional data should identify these years as unusual.
The functional data are denoted by y_i(x). Here, i is a discrete index over all observations, and x is the continuous variable over which each function is defined. For the mortality data, y_{i}(x) represents the mortality rate for age x in year i.
It is possible to have multiple indexes (e.g., mortality rates for males and females, or for different countries, or both), and the function may be defined over more than one variable (e.g., mortality rates over age and BMI). It is also possible to have multivariate observations (e.g., mortality and fertility rates).
The methods in this chapter all assume that the curves are already aligned on a common domain, so that x means the same thing for every observation, as is naturally true here since age is measured on the same scale in every year. When curves instead exhibit phase variation (e.g., the same underlying pattern occurring earlier or later, or faster or slower, for different observations), that misalignment can itself masquerade as a shape anomaly, and the curves should first be “registered” to a common scale before comparison (Ramsay & Silverman, 2005).
13.1 Interpolation and smoothing
The distinguishing feature of functional data is that we think of the data as smooth and continuous functions. In practice, our observations are usually at discrete values, even though the underlying functions are continuous. In Figure 13.1, for example, we observe the mortality rate for discrete ages 0, 1, 2, \dots, while we think of the rate as defined for any age x \in [0,\infty).
It is common to form continuous functions using some kind of interpolation or smoothing of the raw data. Smoothing is used if the underlying curve has been observed with measurement error.
The data shown in Figure 13.1 are observed with error, as the “wiggliness” of the curves shows. The mortality rate (or central death rate) at age x and year i is defined as the number of people who died at age x in year i divided by the number of people who were aged x in the middle of year i. We can smooth the data to obtain the mortality rates at noninteger values of x.
In the following code, we use a smoothing method proposed by Hyndman & Ullah (2007), which takes account of the characteristics of mortality rates.
library(vital)
fr_smooth <- fr_male |>
as_vital(index = Year, key = Age, .age = "Age") |>
smooth_mortality(Mortality, k = 21) |>
as_tibble()Figure 13.2 shows the resulting smooth curve for one year along with the original data, while Figure 13.3 shows the smooth curves for all years.
Code
fr_male |>
filter(Year == 1999) |>
ggplot(aes(x = Age, y = Mortality)) +
geom_point() +
geom_line(
data = fr_smooth |> filter(Year == 1999),
aes(y = .smooth)
) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate")
Code
fr_smooth |>
ggplot() +
aes(x = Age, y = .smooth, colour = Year, group = Year) +
geom_line() +
scale_colour_gradientn(colours = rainbow(100)[1:80]) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate")
Working with the smoothed curves rather than the raw mortality rates has several advantages for anomaly detection. Smoothing removes measurement noise that would otherwise be mistaken for genuine functional variation, so isolated noisy observations are not flagged as anomalies, and real anomalies are not masked by noise. It also gives more stable and reliable estimated rates than the raw data alone, by borrowing strength across neighbouring ages, using the fact that mortality varies smoothly with age.
13.2 Dimension reduction using basis functions
We can approximate functional data using a linear combination of basis functions: \hat{y}_i(x) = b_0(x) + \sum_{j=1}^J \beta_{ij} b_j(x), where b_0(x) is a fixed function common to every observation (analogous to an intercept), \{b_1(x),\dots,b_J(x)\} are a set of basis functions, and \beta_{ij} are the coefficients. With enough basis functions, this representation can approximate a wide class of functions. For example, the Fourier basis functions are a set of sine and cosine functions that can be used to approximate any smooth periodic function. The B-spline basis functions are a set of piecewise polynomial functions that can be used to approximate any smooth function. The choice of basis functions depends on the characteristics of the data.
The number of basis functions J is usually much smaller than the number of points at which each curve is observed, with larger J corresponding to a more flexible approximation.
Given a set of basis functions, we can estimate the coefficients \beta_{ij} by minimising the sum of squared errors between the observed data y_i(x) and the approximated data \hat{y}_i(x).
Alternatively, we can use functional principal component analysis to find the empirical basis functions that explain the most variance in the data. Then b_0(x) is set to be the mean function, b_0(x) = \frac{1}{n} \sum_{i=1}^n y_i(x), and the first principal component, b_1(x), is the function that minimises \sum_{i=1}^n \int \big[y_i(x) - b_0(x) - \beta_{i1}b_1(x)\big]^2\,dx subject to the constraint that \int b_1^2(x)\,dx = 1. The second principal component, b_2(x), is the function that minimises \sum_{i=1}^n \int \big[y_i(x) - b_0(x) - \beta_{i1}b_1(x) - \beta_{i2}b_2(x)\big]^2\,dx subject to the constraints that \int b_2^2(x)\,dx = 1 and \int b_1(x)b_2(x)\,dx = 0. And so on. The principal components b_1(x), b_2(x), \dots, are also called the loadings, while the coefficients \beta_{i1},\beta_{i2},\dots are called the scores.
If the functions are discretised on a fine grid over x, we can use regular principal component decomposition to get approximately the same results as functional principal components.
In the following example, we find the first few principal components using the French male mortality data. (The smoothed data are already on a fine grid of ages, so we don’t need to discretise them further.)
# Wide version of smoothed log Mortality with ages on columns
frmort_wide <- fr_smooth |>
mutate(logmx = log(.smooth)) |>
select(Year, Age, logmx) |>
tidyr::pivot_wider(
names_from = Age,
values_from = logmx,
names_prefix = "Age"
)
# Matrix containing only the log Mortality values, not the year
frmort_mat <- frmort_wide |>
select(-Year) |>
as.matrix()
# Compute first three principal components
pca <- frmort_mat |>
prcomp(center = TRUE, scale = FALSE, rank = 3)
summary(pca)Importance of first k=3 (out of 86) components:
PC1 PC2 PC3
Standard deviation 6.717 1.5335 0.7378
Proportion of Variance 0.932 0.0486 0.0112
Cumulative Proportion 0.932 0.9801 0.9913
The first three principal components explain 99.1% of the variation in the data. Instead of working in the 86-dimensional space containing all ages, we can work in the 3-dimensional space of the first three principal component scores, which is a much simpler problem, without losing much information.
Code
# Extract PCA basis functions
pca_bases <- broom::tidy(pca, "loadings") |>
mutate(
Age = readr::parse_number(column),
Basis = paste0("PC", as.factor(PC))
)
pca_bases |>
ggplot() +
aes(x = Age, y = value, group = Basis, colour = Basis) +
geom_line() +
labs(y = TeX("$b_j(x)$"), x = TeX("Age ($x$)"))
Figure 13.4 shows the first three empirical basis functions.
- The first basis function is negative at every age, and largest in magnitude in early childhood, so a high score on this component corresponds to lower mortality at all ages, but especially for children.
- The second basis function contrasts men of fighting age (roughly ages 18–40) against children, so it picks out years where mortality among young men was unusually high (or low) relative to mortality in childhood.
- The third basis function contrasts infant mortality and older mortality (above age 40), with mortality of pre-teens and early 20s.
# Extract PCA scores
pca_scores <- broom::augment(pca, frmort_wide[, "Year"]) |>
select(-.rownames) |>
rename_with(~ gsub("\\.fitted", "", .x))Code
pca_scores |>
tidyr::pivot_longer(
-Year,
names_to = "PC",
values_to = "value"
) |>
ggplot(aes(x = Year, y = value)) +
geom_line(aes(colour = PC)) +
labs(y = TeX("$\\beta_{ij}$"), x = "Year")
Figure 13.5 shows how the three component scores evolve over time. Other than sharp dips during the two world wars and a few earlier crisis years (notably the cholera years 1832, 1849 and 1854–1855 (Bourdelais & Raulot, 1987), and 1870–1871), the first component score increases steadily and fairly smoothly across the study period, reflecting the long-run, near-continuous decline in mortality at all ages. The second component score sits close to zero for most years but spikes sharply during 1914–1918 and 1940–1945, capturing the additional mortality among young French men during the two world wars. The third component is more subtle, and reflects slow changes in mortality patterns over time, rather than the sharp spikes seen in the second component.
pca_scores |>
select(-Year) |>
GGally::ggpairs()
Figure 13.6 plots the pairwise scatterplots of the first three component scores against each other. Most years fall along a narrow band in each plot, but there are some clear anomalies, especially in the second component.
In the next section, we treat points that lie far from the bulk of the data in this principal component space as candidate anomalies.
13.3 Anomalies in the principal component space
Once the functional data have been reduced to a handful of principal component scores, each curve (here, each year) is just a point in a low-dimensional numerical space, and we can apply the same methods introduced previously in this book. For example, we can compute a surprisal probability for each year from the first two component scores, using a GPD approximation for the tail (Section 7.3), and flag any year with a probability below 0.05 as anomalous. We restrict attention to the first two components here so the results can be shown more easily in a scatterplot; we return to all three components below using the O3 plot.
# Find outliers in the PCs
outliers <- pca_scores |>
mutate(prob = surprisals_prob(pick(PC1, PC2), approximation = "gpd")) |>
filter(prob < 0.05)
pca_scores |>
ggplot(aes(x = PC1, y = PC2)) +
geom_point(colour = "#0072b2") +
geom_point(data = outliers, colour = "#D55e00") +
ggrepel::geom_label_repel(data = outliers, aes(label = Year))
Figure 13.7 identifies eight anomalous years: 1871, 1916–1919, 1940, 1943 and 1944. These are unsurprising given Figure 13.1, and most coincide with years already flagged in Section 12.2 using the Hampel identifier applied directly to the raw age-specific series, a quite different method. The following historical events are relevant:
- 1871: Repression of the ‘Commune de Paris’
- 1914–1918: World War I
- 1918–1919: Spanish flu pandemic
- 1940–1945: World War II
To take another perspective, we can apply the O3 plot (Section 9.11) to the first three principal component scores.
o3 <- pca_scores |>
select(-Year) |>
OutliersO3::O3prep(
method = c("PCS", "adjOut", "MCD", "BAC", "DDC"),
tolPCS = 0.05,
toladj = 0.15,
tolMCD = 0.001,
tolBAC = 0.001,
tolDDC = 0.025
)
o3_plots <- OutliersO3::O3plotM(
o3,
coltxtsize = 10,
caseNames = paste0("Y", pca_scores$Year)
)o3_plots$gO3
The years flagged at least 7 times across all five methods and variable combinations are given below.
Code
red_case <- o3_plots$outsTable |>
distinct(Case, Combination, Method) |>
count(Case, Combination, name = "n_agree") |>
filter(n_agree == n_distinct(o3_plots$outsTable$Method))
out_table <- o3_plots$outsTable |>
count(Case, name = "n_flagged") |>
filter(n_flagged > 6) |>
mutate(red = Case %in% red_case$Case)
pca_scores |>
slice(out_table$Case) |>
mutate(n_flagged = out_table$n_flagged, red = out_table$red) |>
arrange(Year)# A tibble: 10 × 6
Year PC1 PC2 PC3 n_flagged red
<int> <dbl> <dbl> <dbl> <int> <lgl>
1 1871 -10.1 2.02 -0.0908 12 FALSE
2 1914 -7.70 8.10 0.349 18 TRUE
3 1915 -8.64 8.21 0.466 18 TRUE
4 1916 -7.43 7.26 0.374 16 FALSE
5 1917 -6.30 5.69 0.0731 13 FALSE
6 1918 -9.15 6.26 0.137 15 TRUE
7 1919 -4.05 2.25 -0.148 9 FALSE
8 1940 -3.89 5.49 -0.795 14 TRUE
9 1943 -2.44 3.56 0.424 13 FALSE
10 1944 -5.38 4.85 0.138 13 FALSE
Reassuringly, the O3 plot, using all three principal components and five quite different outlier-detection methods, flags ten years (1871, 1914–1919, 1940, 1943 and 1944) — largely the same years identified by the two-component surprisal approach above. Of these, four — 1914, 1915, 1918 and 1940 — are flagged by all five methods for at least one variable combination (shown in red in Figure 13.8), while the rest are flagged consistently but not unanimously. This agreement across methods, along with the time series analysis of Section 12.2, gives us some confidence that these years are genuine anomalies rather than artefacts of any one method.
13.4 Functional bagplot
A boxplot summarises a univariate sample using a central box and whiskers, with points beyond the whiskers flagged as outliers. Several different ways of extending this idea to functional data are described over the sections that follow. The functional bagplot and the functional HDR boxplot, covered in this section and the next, were proposed by Hyndman and Shang (2008, 2010). They follow a similar approach: reduce each curve to its first two functional principal component scores (Section 13.2), draw the bivariate bagplot or HDR boxplot of those scores, and then map each region back to a functional envelope formed from the curves whose scores fall inside it. So the inner region of the functional bagplot is bounded by the curves whose scores lie in the bivariate bag, and the outer region is bounded by the curves whose scores lie in the bivariate loop. The bagplot’s central point (the depth median of the scores) need not coincide with any observed point, so the “median curve” is instead taken to be the observed curve whose scores are closest (in Euclidean distance) to that central point.
Figure 13.9 shows the ordinary bivariate bagplot (Section 5.6) of the first two principal component scores. Each point corresponds to one year’s mortality curve; the dark region is the bag, the light region is the loop, and any points beyond the loop are flagged as bivariate outliers, coloured by year to match the corresponding curves in Figure 13.10.
bag <- aplpack::compute.bagplot(as.matrix(pca_scores[, c("PC1", "PC2")]))
bagplot_outliers <- as.data.frame(bag$pxy.outlier) |>
left_join(pca_scores, by = c("PC1", "PC2"))Code
pca_scores |>
gg_bagplot(PC1, PC2) +
geom_point(data = bagplot_outliers, aes(colour = factor(Year)), size = 2) +
labs(colour = "Outlier year")
The bag and loop shown here trace out regions of “typical” curves, while the flagged points correspond to years whose curves depart from that typical shape. Taking the pointwise range of the curves whose scores fall inside the bag, and separately of the curves whose scores fall inside the loop, gives an envelope for the “typical” region of the functional bagplot at every age.
The following code identifies the curves contained in the bag and the loop, and those beyond that loop corresponding to the anomalies in Figure 13.9.
# Pointwise envelope of the actual (smoothed) curves for a set of years
curve_envelope <- function(years) {
fr_smooth |>
filter(Year %in% years) |>
summarise(ymin = min(.smooth), ymax = max(.smooth), .by = Age)
}
bag_years <- as.data.frame(bag$pxy.bag) |>
left_join(pca_scores, by = c("PC1", "PC2")) |>
pull(Year)
loop_years <- as.data.frame(bag$pxy.outer) |>
left_join(pca_scores, by = c("PC1", "PC2")) |>
pull(Year)
bag_env <- curve_envelope(bag_years)
loop_env <- curve_envelope(c(bag_years, loop_years))
# Central curve: the observed curve closest to the bagplot's depth median
centre_point <- which.min(
(pca_scores$PC1 - bag$center[1])^2 + (pca_scores$PC2 - bag$center[2])^2
)
centre_curve <- fr_smooth |> filter(Year == pca_scores$Year[centre_point])
outlier_curves <- fr_male |>
filter(Year %in% bagplot_outliers$Year)Code
bag_colors <- c("#00659e", "#5397BD", "#A7CADD")
ggplot() +
geom_ribbon(
data = loop_env,
aes(x = Age, ymin = ymin, ymax = ymax),
fill = bag_colors[3]
) +
geom_ribbon(
data = bag_env,
aes(x = Age, ymin = ymin, ymax = ymax),
fill = bag_colors[2]
) +
geom_line(
data = centre_curve,
aes(x = Age, y = .smooth),
colour = bag_colors[1]
) +
geom_line(
data = outlier_curves,
aes(x = Age, y = Mortality, group = Year, colour = factor(Year))
) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate", colour = "Outlier year")
Figure 13.10 flags twelve years as functional bagplot outliers: 1871, 1914–1919, 1940 and 1942–1945. These are almost exactly the war years already identified in Section 13.3 and Section 12.2, and the outlying curves are visibly bulging above the typical envelope for young men.
Another implementation of the functional bagplot is provided by rainbow::fboxplot().
13.5 Functional HDR boxplot
The functional HDR boxplot follows the same recipe, but uses the highest density regions of a bivariate kernel density estimate of the scores (Section 3.4) in place of the bagplot’s bag and loop.
Figure 13.11 shows the ordinary bivariate HDR boxplot (Section 5.7) of the first two principal component scores. Because of the unusual shape of the principal component scores, the resulting kernel density estimate is particularly sensitive to the bandwidth choice, and the resulting HDRs are broken into several disconnected pieces. After some experimentation, the “scv” (smoothed cross-validation) bandwidth method was chosen. To roughly match the number of years flagged by the bagplot, a 93% coverage probability is used for the larger HDR. Thus, the dark region is the 50% HDR (containing about 50% of the years), the light region is the 93% HDR (containing about 93% of the years), and the points outside the 93% HDR are flagged as outliers, coloured by year to match the corresponding curves in Figure 13.12.
hdr_labels <- pca_scores[, c("PC1", "PC2")] |>
dist_kde(method = "scv") |>
hdr_regions(prob = c(0.5, 0.93))
pca_scores <- pca_scores |>
mutate(
hdr_region = case_when(
!is.na(hdr_labels$hdr_50) ~ "50%",
!is.na(hdr_labels$hdr_93) ~ "93%",
TRUE ~ "outlier"
),
hdr_cluster = hdr_labels$hdr_93
)
hdr_outliers <- pca_scores |> filter(hdr_region == "outlier")Here, we have used hdr_regions() to classify each year’s scores as either inside the 50% HDR, inside the 93% HDR but outside the 50% HDR, or outside the 93% HDR (and thus flagged as an outlier).
Code
pca_scores |>
gg_hdrboxplot(PC1, PC2, prob = c(0.5, 0.93), method = "scv") +
geom_point(
data = hdr_outliers,
aes(x = PC1, y = PC2, colour = factor(Year)),
size = 2
) +
labs(colour = "Outlier year") +
guides(colour = "legend")
All but one of the thirteen years flagged as outliers have previously been identified as anomalous above (in Figure 13.7, Figure 13.8 or Figure 13.9); the exception, 1855, was not picked out by any of the earlier methods, but corresponds to a year during the Crimean War (Glei et al., 2025).
As with the bagplot, the envelope of each functional HDR is formed directly from the curves whose scores fall inside it, using the hdr_region classification already computed above. Because the bivariate scores split into two separated clusters (Figure 13.11), so does each HDR: hdr_regions() labels which disconnected piece of the HDR each year’s scores fall into, via hdr_cluster, and we build a separate envelope for each piece, so the split remains visible wherever the two pieces’ envelopes do not overlap.
Code
hdr_scores <- pca_scores |> filter(hdr_region != "outlier")
cluster_envelope <- function(years) {
fr_smooth |>
filter(Year %in% years) |>
left_join(hdr_scores[, c("Year", "hdr_cluster")], by = "Year") |>
summarise(
ymin = min(.smooth),
ymax = max(.smooth),
.by = c(Age, hdr_cluster)
)
}
hdr50_env <- cluster_envelope(hdr_scores$Year[hdr_scores$hdr_region == "50%"])
hdr93_env <- cluster_envelope(hdr_scores$Year)
outlier_curves <- fr_male |>
filter(Year %in% hdr_outliers$Year)
ggplot() +
geom_ribbon(
data = hdr93_env,
aes(x = Age, ymin = ymin, ymax = ymax, group = hdr_cluster),
fill = bag_colors[3]
) +
geom_ribbon(
data = hdr50_env,
aes(x = Age, ymin = ymin, ymax = ymax, group = hdr_cluster),
fill = bag_colors[2]
) +
geom_line(
data = outlier_curves,
aes(x = Age, y = Mortality, group = Year, colour = factor(Year))
) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate", colour = "Outlier year")
In Figure 13.12, the bulk of the curves fall into two distinct clusters, corresponding to before and after the end of World War II.
Another implementation of the functional HDR boxplot is provided by rainbow::fboxplot().
13.6 Functional depth
Rather than reducing each curve to a handful of principal component scores, the remaining approaches in this chapter work directly with the curves themselves, via functional depth or a related measure. See Hubert et al. (2015) for a more detailed review of these ideas.
The idea of depth generalises the univariate idea of a median and quantiles to more complex objects. We have already seen this with multivariate numeric data (Section 5.5); now we will extend the idea to functional data. Given a sample of curves y_1(x),\dots,y_n(x), a depth measure assigns each curve y_i a value between 0 and 1 describing how central it is within the sample: curves near the middle of the distribution receive high depth, while unusual curves receive low depth. Curves can then be ranked from the deepest (most typical) to the shallowest (most unusual), giving an ordering analogous to ranking univariate observations from the median outward.
One of the simplest depths is the band depth: it counts how often a curve y_i lies completely within the “band” traced out by every possible group of k curves in the sample (usually k = 2, so the bands are formed by pairs of curves), so that curves often enveloped by others are deep, while curves that stick out — even briefly — have low depth. With k = 2, the band depth of y_i is the proportion of pairs of curves whose band contains it: \text{BD}(y_i) = \binom{n}{2}^{-1} \sum_{1 \le u<v \le n} 𝟙\Big(\min\{y_u(x), y_v(x)\} \le y_i(x) \le \max\{y_u(x), y_v(x)\} \text{ for all } x\Big), where 𝟙(\cdot) is the indicator function, taking value 1 when its argument is true and 0 otherwise. Looking at Figure 13.1, we would expect that some of the most recent years, which have low mortality at almost every age, would be shallow; the war years would also be shallow, since they have high mortality at ages 20–40 that is not seen in other years.
However, curves that cross each other are rarely contained entirely within a band, so many curves have a band depth close to zero, with many ties. For the French mortality curves, 17 of the 184 curves lie entirely within none of the bands except those they help to form, and so share the smallest possible band depth. In practice, to address this problem, the band depth is usually replaced by the modified band depth (MBD), which instead measures the proportion of the domain x over which y_i(x) lies within each band, averaged over all bands (López-Pintado & Romo, 2009):
\text{MBD}(y_i)
= \binom{n}{2}^{-1} \sum_{1 \le u<v \le n} \frac{1}{|\mathcal{X}|} \int_{\mathcal{X}} 𝟙\Big(\min\{y_u(x), y_v(x)\} \le y_i(x) \le \max\{y_u(x), y_v(x)\}\Big)\,dx,
where \mathcal{X} is the domain of the curves and |\mathcal{X}| is its length. In practice, the integral is replaced by an average over the grid of ages at which the curves are evaluated, which is what the mbd() function below computes.
mbd <- function(Y) {
n <- NROW(Y)
m <- NCOL(Y)
r <- apply(Y, 2, rank)
# (r-1)(n-r): pairs of curves whose band contains y at this point.
# + (n-1): pairs where y is itself one of the two band-forming curves.
(rowSums((r - 1) * (n - r)) / m + (n - 1)) / choose(n, 2)
}
depth_tbl <- tibble(Year = frmort_wide$Year, MBD = mbd(frmort_mat))
depth_tbl |> arrange(MBD)# A tibble: 184 × 2
Year MBD
<int> <dbl>
1 1999 0.0142
2 1998 0.0209
3 1871 0.0292
4 1997 0.0307
5 1996 0.0591
6 1995 0.0835
7 1870 0.0891
8 1994 0.0972
9 1854 0.107
10 1993 0.110
# ℹ 174 more rows
Ranking curves by MBD alone identifies the years whose entire trajectory lies furthest from the bulk of the sample: the shallowest curves are 1854, 1870, 1871 and 1993–1999. The seven most recent of these have the lowest mortality in the sample at most ages, while the three from the 1800s are shallow for the opposite reason — their mortality is distinctly higher than in surrounding years at most ages. These earlier shallow years correspond to the 1854 cholera outbreak, the 1870 Franco-Prussian War and the 1871 Repression of the ‘Commune de Paris’.
This illustrates an important feature of depth-based ranking: it can fail to distinguish between curves that are unusual due to location and those that are unusual due to shape. Because French mortality has fallen more or less steadily across the study period, the most recent curves sit at one edge of the whole 184-year sample and are therefore the least “enveloped” by the rest, even though their shape is not unusual. On the other hand, the 1854, 1870 and 1871 curves are unusual relative to the years around them, although only 1871 lies at the upper edge of the whole sample at most ages.
Other types of functional depth are computed by rainbow::fdepth(), and the fda.usc, fdaoutlier and mrfDepth packages.
13.7 Sun–Genton functional boxplot
The Sun–Genton functional boxplot (Sun & Genton, 2011) works directly with the (possibly smoothed) curves, using the modified band depth introduced in Section 13.6. The central region is formed from the 50% deepest curves (those with MBD at or above the median), and a curve is flagged as an outlier if, at any age, it leaves a fence formed by inflating this central region by 1.5 times its own (pointwise) range. Figure 13.13 shows the resulting functional boxplot for the French male mortality curves.
Code
central_years <- depth_tbl$Year[depth_tbl$MBD >= median(depth_tbl$MBD)]
centre_year <- depth_tbl$Year[which.max(depth_tbl$MBD)]
# The fence is a linear inflation of the central region's width, so it must be
# computed on the log scale in this example
central_env <- fr_smooth |>
filter(Year %in% central_years) |>
summarise(cmin = min(log(.smooth)), cmax = max(log(.smooth)), .by = Age) |>
mutate(
flower = cmin - 1.5 * (cmax - cmin),
fupper = cmax + 1.5 * (cmax - cmin)
)
outlier_years <- fr_smooth |>
mutate(logm = log(.smooth)) |>
left_join(central_env, by = "Age") |>
filter(logm < flower | logm > fupper) |>
distinct(Year) |>
pull(Year)
fr_smooth |>
summarise(
cmin = min(.smooth[Year %in% central_years]),
cmax = max(.smooth[Year %in% central_years]),
fmin = min(.smooth[!Year %in% outlier_years]),
fmax = max(.smooth[!Year %in% outlier_years]),
.by = Age
) |>
ggplot() +
geom_ribbon(
aes(x = Age, ymin = fmin, ymax = fmax),
fill = bag_colors[3]
) +
geom_ribbon(
aes(x = Age, ymin = cmin, ymax = cmax),
fill = bag_colors[2]
) +
geom_line(
data = fr_smooth |> filter(Year == centre_year),
aes(x = Age, y = .smooth),
colour = bag_colors[1]
) +
geom_line(
data = fr_smooth |> filter(Year %in% outlier_years),
aes(x = Age, y = .smooth, group = Year, colour = factor(Year))
) +
scale_y_log10(
breaks = 10^(-(0:4)),
labels = format(10^(-(0:4)), scientific = FALSE)
) +
labs(y = "Mortality rate", colour = "Outlier year")
Figure 13.13 fails to identify the war years or pandemics as outliers, and its fences are so wide that they include almost all of the curves. In fact, it flags six years, 1994–1999, none of which are war years. These are simply the most recent years in the sample, which have the lowest mortality at most ages.
This may seem inconsistent with 1854, 1870 and 1871 being the shallowest 19th century curves, but a low MBD only means a curve is poorly enveloped by bands of other curves; it does not mean the curve’s values are pointwise extreme, which is what the boxplot fence tests. The mortality in those three years, though above the central region at many ages, stays well inside the upper fence, which is wide because the central region spans more than a century of declining mortality; only the six most recent years cross the (lower) fence.
The functional bagplot and HDR boxplot do not share the problem of trend. The trend is still there in the principal component scores (it is essentially the first component), but in the score space the most recent years lie at one end of a long, continuous trajectory of first component scores, close to the scores of the years just before them, rather than apart from the bulk of the data. The war years, on the other hand, stand out on the second component, which captures the specific age-pattern (young men versus children) that made them unusual.
While the Sun–Genton functional boxplot did not do well in this example, it can still be a useful tool for identifying unusual curves in other contexts, especially where the first two principal component scores do not capture the most interesting variation in the data.
The functional boxplot was coded from scratch here, but versions are available in other packages including fda::fbplot(), GET::fBoxplot(), fdaoutlier::functional_boxplot() and roahd::fbplot().
The next two sections look at ways of separating genuine shape anomalies from the kind of trend-driven magnitude effect that misled the depth boxplot here.
13.8 Outliergram
The modified band depth used above conflates two different reasons a curve can be unusual: it might have a genuinely atypical shape, or it might sit at one extreme of the bulk of the data, and so be poorly enveloped by the rest of the sample. The outliergram of Arribas-Gil & Romo (2014) separates these two effects by comparing MBD against a second measure, the modified epigraph index (MEI) of López-Pintado & Romo (2011).
Whereas MBD measures how often a curve is enveloped by bands formed from pairs of curves, MEI measures a curve’s rank position: it is close to 1 for a curve lying below almost every other curve across almost the whole domain, close to 0 for a curve lying above almost every other curve, and around 0.5 for a curve in the middle of the pack. It is the proportion of the domain on which y_i lies on or below each curve, averaged over all n curves (including y_i itself): \text{MEI}(y_i) = \frac{1}{n} \sum_{u=1}^{n} \frac{1}{|\mathcal{X}|} \int_{\mathcal{X}} 𝟙\big(y_i(x) \le y_u(x)\big)\,dx. So MEI captures a curve’s overall level relative to the rest of the sample, in much the same way that the first principal component score did in Section 13.2, while MBD captures how well-enveloped it is.
For most collections of curves, curves with an extreme MEI (very close to 0 or 1) are, by virtue of sitting near the edge of the sample, poorly enveloped and so also have somewhat low MBD. In fact, there is a well-defined upper bound on MBD as a function of MEI, tracing out a parabola (Arribas-Gil & Romo, 2014): \begin{align*} \text{MBD}(y_i) & \le a_0 + a_1\,\text{MEI}(y_i) + a_2\,\text{MEI}(y_i)^2, \\ \text{where } a_0 & = -\frac{2}{n(n - 1)}, \quad a_1 = \frac{2(n + 1)}{n - 1},\quad a_2 = -\frac{2n}{n - 1}. \end{align*} A curve with an unusual shape stands out from this pattern: its MBD falls noticeably below the parabola for its MEI value, meaning it is even less enveloped than curves with a similar rank would typically be. The outliergram plots MEI against MBD along with this parabola, and flags as shape outliers those curves whose vertical distance below the parabola exceeds a boxplot-style fence (the third quartile of these distances, plus 1.5 times their interquartile range, by default).
mei <- function(Y) {
n <- NROW(Y)
m <- NCOL(Y)
r <- apply(Y, 2, rank, ties.method = "min")
rowSums(n - r + 1) / (n * m)
}
mei_tbl <- depth_tbl |> mutate(MEI = mei(frmort_mat))
# Theoretical upper bound of MBD as a function of MEI (Arribas-Gil & Romo 2014)
n_years <- NROW(frmort_wide)
a0 <- -2 / (n_years * (n_years - 1))
a1 <- 2 * (n_years + 1) / (n_years - 1)
a2 <- n_years^2 * a0
mbd_bound <- function(mei) a0 + a1 * mei + a2 * mei^2
mei_tbl <- mei_tbl |>
mutate(d = mbd_bound(MEI) - MBD)
# Shape outliers: distance below the parabola exceeds a boxplot-style fence
fence <- quantile(mei_tbl$d, 0.75) + 1.5 * IQR(mei_tbl$d)
og_outliers <- mei_tbl |> filter(d >= fence)Code
parabola <- tibble(MEI = seq(0, 1, length.out = 100)) |>
mutate(MBD = mbd_bound(MEI), lower = MBD - fence)
mei_tbl |>
ggplot(aes(x = MEI, y = MBD)) +
geom_ribbon(
data = parabola,
aes(ymin = lower, ymax = MBD),
fill = "#00659e",
alpha = 0.2
) +
geom_line(data = parabola, aes(y = MBD), colour = "#00659e") +
geom_point(colour = "#555555") +
geom_point(data = og_outliers, colour = "#D55E00") +
ggrepel::geom_label_repel(
data = og_outliers,
aes(label = Year),
colour = "#D55E00",
nudge_y = -0.05,
direction = "y"
)
Figure 13.14 flags three years as shape outliers: 1940, 1941 and 1943. These were all during World War II, and in each of them mortality at some ages was much higher than the curve’s overall level would suggest: for young men in 1940 and 1943, and for older adults in 1940 and 1941.
More shape anomalies would be identified if we reduced the fence multiplier from 1.5. Unlike the depth boxplot fence of Figure 13.13, none of the low-mortality recent years are flagged, because their low MBD is entirely explained by their extreme MEI — they sit at one end of the rank ordering, but they do not deviate from the shape that curves at that end of the ranking are expected to have.
13.9 Magnitude-shape plot
A second way to separate magnitude and shape effects is the magnitude-shape plot of Dai & Genton (2018). For each curve, and at each point x on the grid, they compute a directional outlyingness: a signed measure of how far, and in which direction, the curve departs from the centre of the distribution of values at that x. Averaging this quantity over the domain gives a magnitude score (MO) for each curve, taking a value far from zero when a curve sits consistently on one side of the bulk of the data. The variation of the directional outlyingness over the domain gives a separate shape score (VO), which is large when a curve’s departure from the centre changes direction or size as x varies (as happens when a curve crosses others or has an unusual local pattern), even if its overall MO is modest. Plotting VO against MO for every curve gives the magnitude-shape plot; Dai & Genton (2018) then flag outliers using a robust distance in this two-dimensional (MO, VO) space, with a finite-sample correction due to Hardin & Rocke (2005).
These calculations are implemented in the fdaoutlier package, which we use here.
ms <- fdaoutlier::msplot(frmort_mat, seed = 1, plot = FALSE)
ms_tbl <- tibble(
Year = frmort_wide$Year,
MO = ms$mean_outlyingness,
VO = ms$var_outlyingness
)
ms_outliers <- ms_tbl |> slice(ms$outliers)Code
ms_tbl |>
ggplot(aes(x = MO, y = VO)) +
geom_point(colour = "#0072B2") +
geom_point(data = ms_outliers, colour = "#D55E00")
Figure 13.15 flags sixty-two years as outliers, far more than either the outliergram or the depth boxplot. Almost all of these are on the left of the plot and cover every year after World War II. The remainder, 1871, 1914–1918, 1940 and 1944, are war and conflict years already familiar from earlier sections. So the magnitude-shape plot re-discovers the same problem as the Sun–Genton functional boxplot of Section 13.7: because mortality has fallen steadily across the study period, every sufficiently recent curve has a large, consistently signed directional outlyingness relative to the whole sample, driving up its MO score regardless of whether anything unusual happened in that year.
13.10 Summary
The methods in this chapter fall into two groups, and they do not always agree, because they are not measuring the same thing.
The first group reduces each curve to a few principal component scores, and then applies multivariate methods (such as surprisal probabilities, the O3 plot, the bagplot or the HDR boxplot) to the scores. These methods find curves whose scores sit apart from the rest, so they detect curves that are unusual relative to the main modes of variation captured by the components. A gradual trend over time becomes a continuous path through the score space, rather than a set of isolated points, so these methods are not easily misled by curves at the end of a trend.
The second group works directly with the curves, via functional depth (such as the modified band depth) or directional outlyingness, leading to the functional boxplot and the magnitude-shape plot. These measures are dominated by how far a curve lies from the centre of the sample. The functional boxplot cannot distinguish a curve with a genuinely unusual shape from one that lies at one extreme of the sample, such as the last curve of a long, gradual trend; the magnitude-shape plot separates the two effects visually, but its outlier rule uses both scores, so curves at the end of a trend are still flagged. The outliergram addresses this by comparing a curve’s modified band depth with its rank position (via the modified epigraph index), so that curves which are poorly enveloped only because they are extreme are not flagged, while curves with unusual shapes are.
No single method is more “correct” than the others. In practice, applying more than one method, and using the disagreements between them to understand why particular curves are flagged, is more informative than relying on any single method to decide what counts as anomalous.