Skip to main content

14  Image and video data

Many anomaly detection problems involve images or video. Manufacturers photograph products on a production line to find defective items, radiologists look for abnormalities in X-rays and MRI scans, and surveillance cameras are monitored for unusual events. In each case, most images are similar to each other, and we are looking for the few that differ, or for the part of an image that differs from what we would expect.

A greyscale image of p \times q pixels can be stored as a vector of length m = pq, where each element is the intensity of one pixel. A colour image has three intensities per pixel (red, green and blue), so it can be stored as a vector of length 3pq by stacking the three colour channels. Alternatively, we can convert a colour image to greyscale first, by taking a weighted average of the three channels at each pixel; this is done for the video data in Section 14.4. So a collection of n images is an n \times m data matrix, with one row per image, and in principle any of the multivariate methods from earlier chapters could be applied to it. In practice, m is large (even a small 28 \times 28 image has 784 pixels), and neighbouring pixels are strongly correlated, so the data lie close to a much lower-dimensional subspace. That makes images a natural application for the high-dimensional methods of Chapter 9.

There are two broad strategies. The first is to project the images onto a small number of principal components, and look for anomalies among the component scores and in the residuals. The second is to compute a small number of interpretable features from each image, and look for anomalies among the features, much as was done for time series features in Section 12.3. Both strategies are demonstrated on the same collection of images, using many of the multivariate methods from earlier chapters.

Video adds a time dimension: a video is a sequence of images (frames), usually taken by a fixed camera. At the end of the chapter, each frame is treated as an observation to find the frames in which something happens, and each frame is compared with a robust estimate of the static background to find where in the frame it happens.

14.1 Fashion-MNIST data

The Fashion-MNIST data (Xiao et al., 2017) consist of 70,000 greyscale images of clothing items from the online retailer Zalando, in ten classes (T-shirts, trousers, pullovers, dresses, coats, sandals, shirts, sneakers, bags and ankle boots). The original product photos have been converted to 28 \times 28 greyscale images for use in machine learning, and the data set is split into a training set of 60,000 images and a test set of 10,000 images.

The examples use the 1000 sneakers from the test set, together with 10 images from other classes: four ankle boots, three sandals, two bags and one pair of trousers. Some of these planted images are very different from a sneaker (the trousers and bags), while others are quite similar (the ankle boots), so we know at least some of the anomalies, and can see which methods find them. The weird package contains these data as the tibble fashion, with one row per image, the original test set index id, the class label, a logical column planted, and a 1010 \times 784 matrix column pixels of pixel intensities between 0 (white) and 255 (black).

fashion
# A tibble: 1,010 × 4
      id label   planted pixels[,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9]
   <int> <fct>   <lgl>        <int> <int> <int> <int> <int> <int> <int> <int> <int>
 1    10 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 2    13 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 3    23 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 4    37 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 5    39 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 6    44 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 7    46 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 8    61 Sneaker FALSE            0     0     0     0     0     0     0     0     0
 9    62 Sneaker FALSE            0     0     0     0     0     0     0     0     0
10    71 Sneaker FALSE            0     0     0     0     0     0     0     0     0
# ℹ 1,000 more rows
# ℹ 1 more variable: pixels[10:784] <int>

Figure 14.1 shows 20 of the sneakers along with the planted images.

Code
# Plot images given as rows of a matrix with 784 columns
plot_images <- function(pixels, titles = NULL, ncol = 10) {
  if (is.null(titles)) {
    titles <- seq_len(NROW(pixels))
  }
  titles <- factor(titles, levels = unique(titles))
  tibble(
    title = rep(titles, each = 784),
    value = c(t(pixels)),
    col = rep(rep(1:28, 28), NROW(pixels)),
    row = rep(rep(1:28, each = 28), NROW(pixels))
  ) |>
    ggplot(aes(x = col, y = row, fill = value)) +
    geom_raster() +
    scale_y_reverse() +
    scale_fill_gradient(low = "white", high = "black", guide = "none") +
    facet_wrap(~title, ncol = ncol) +
    coord_equal() +
    theme_void() +
    theme(strip.text = element_text(size = 7), strip.clip = "off")
}
shown <- c(sample(which(!fashion$planted), 20), which(fashion$planted))
plot_images(
  fashion$pixels[shown, ],
  titles = paste0(seq_along(shown), ": ", fashion$label[shown])
)
Figure 14.1: Top two rows: 20 randomly selected sneakers from the Fashion-MNIST test set. Bottom row: the ten planted images from other classes.

14.2 Principal components of images

The first strategy is to project the images onto a few principal components, and then look for anomalies among the component scores and in the parts of the images that the components fail to reconstruct.

First, we rescale the intensities to lie between 0 and 1, and drop the pixels that are white in every image (mostly in the corners), as they carry no information. We do not standardise each pixel, because all pixels are measured on the same scale, and standardising would inflate the noise in the nearly white pixels near the edges of the images.

X <- fashion$pixels / 255
used <- apply(X, 2, sd) > 0
X <- X[, used]
pca_fashion <- PcaClassic(X, k = 50)

This leaves m = 714 pixels per image. The first two principal components explain 38% of the variance, the first ten explain 65%, and 50 components are needed to explain 86%. So the images are not as close to a low-dimensional subspace as the air quality data of Chapter 9, but the first few components still capture much of the structure.

Each principal component direction \bm{a}_j is itself a vector of 714 pixel weights, and so it can be displayed as an image. Figure 14.2 shows the first ten components, often called eigen-images.

Code
loadings <- matrix(0, nrow = 10, ncol = 784)
loadings[, used] <- t(pca_fashion$loadings[, 1:10])
tibble(
  pc = factor(rep(paste0("PC", 1:10), each = 784), levels = paste0("PC", 1:10)),
  value = c(t(loadings)),
  col = rep(rep(1:28, 28), 10),
  row = rep(rep(1:28, each = 28), 10)
) |>
  ggplot(aes(x = col, y = row, fill = value)) +
  geom_raster() +
  scale_y_reverse() +
  scale_fill_distiller(
    palette = "RdBu",
    limits = c(-1, 1) * max(abs(loadings)),
    guide = "none"
  ) +
  facet_wrap(~pc, ncol = 10) +
  coord_equal() +
  theme_void()
Figure 14.2: The first ten principal component directions of the sneaker images, shown as images. Red pixels have positive weights, blue pixels have negative weights, and pale pixels have weights close to zero.

Figure 14.3 shows the first two principal component scores, with the planted images highlighted.

Code
fashion_scores <- fashion |>
  select(id, label, planted) |>
  mutate(
    PC1 = pca_fashion$scores[, 1],
    PC2 = pca_fashion$scores[, 2]
  )
fashion_scores |>
  ggplot(aes(x = PC1, y = PC2)) +
  geom_point(data = filter(fashion_scores, !planted), col = "grey70") +
  geom_point(
    data = filter(fashion_scores, planted),
    aes(col = label),
    size = 2
  ) +
  labs(col = "Planted")
Figure 14.3: The first two principal component scores of the sneaker images. The planted images from other classes are shown in colour.

The two bags and the trousers lie towards the edges of the point cloud, but none of the planted images is far from the sneakers in these two dimensions. Most of the weights in the first eigen-image are positive, so the first component mostly measures how dark the image is overall (its correlation with the mean intensity is 0.83), and the two bags and most of the ankle boots have large values because they are dark. Differences in shape, which would distinguish a sandal or a boot from a sneaker, are spread over many components.

To look for anomalies, we will use k = 10 principal components, explaining 65% of the variance. Each image then has two summaries of how unusual it is, as described in Section 9.4: its score distance, which measures how far its 10-dimensional scores are from the centre of the score distribution, and its orthogonal distance, which measures how badly the image is reconstructed by the first ten eigen-images.

pca10 <- PcaClassic(X, k = 10)
set.seed(1967)
robpca10 <- PcaHubert(X, k = 10)

Figure 14.4 shows the outlier maps from classical PCA and from ROBPCA (Section 9.5).

Code
# Label the planted images with their numbers in fig-fashion
label_planted <- function(p) {
  ggrepel::geom_label_repel(
    data = tibble(
      sd = p@sd[fashion$planted],
      od = p@od[fashion$planted],
      image = match(which(fashion$planted), shown)
    ),
    aes(x = sd, y = od, label = image),
    inherit.aes = FALSE,
    size = 3,
    label.padding = 0.1,
    box.padding = 0.5,
    min.segment.length = 0,
    seed = 1967
  )
}
patchwork::wrap_plots(
  outlier_map(pca10) + label_planted(pca10) + ggtitle("Classical PCA"),
  outlier_map(robpca10) + label_planted(robpca10) + ggtitle("ROBPCA"),
  guides = "collect"
) &
  theme(legend.position = "bottom")
Figure 14.4: Outlier maps for the sneaker images using ten principal components. Left: classical PCA. Right: robust PCA (ROBPCA). The planted images are labelled with their numbers from Figure 14.1.

The two bags (images 28 and 29) and the trousers (image 30) are bad leverage points in both maps: they have extreme scores on the first ten components, and those components reconstruct them poorly. The high-heeled sandal (image 25 in Figure 14.1) is an orthogonal outlier in both maps: its scores are typical of a sneaker, but it lies far from the subspace spanned by the components. A second sandal (image 26) lies just above the orthogonal distance cutoff. The ankle boots are spread across the classical map: one (image 21) is a clear bad leverage point, two (images 22 and 24) lie close to the corner where the cutoffs meet, and one (image 23) is a regular observation.

Comparing methods

The scores live in a 10-dimensional space, so we can also apply any of the multivariate methods from Chapter 6, Chapter 7 and Chapter 9 to them. The function below computes seven of them for a matrix W, with larger values indicating more anomalous observations in every case. Following Section 9.5, the isolation forest is applied to the scores together with the orthogonal distances. We use 10 nearest neighbours for the kNN, LOF and GLOSH scores, rather than the 20 used in Chapter 6. The smaller value is needed because the same function is used on the image features in Section 14.3, where with 20 neighbours HDBSCAN finds no clusters, and GLOSH then gives every observation a score of 1.

anomaly_scores <- function(W, od = NULL, k = 10) {
  set.seed(1967)
  gmm <- Mclust(W, verbose = FALSE) |> dist_mclust()
  iso_data <- as_tibble(W)
  if (!is.null(od)) {
    iso_data$od <- od
  }
  tibble(
    kNN = dbscan::kNNdist(W, k = k),
    LOF = lof_scores(W, k = k),
    GLOSH = glosh_scores(W, k = k),
    `Isolation forest` = predict(
      isotree::isolation.forest(iso_data, seed = 1967),
      iso_data
    ),
    Stray = stray_scores(W),
    `Stahel-Donoho` = robustbase::adjOutlyingness(W)$adjout,
    `GMM surprisal` = surprisals(W, distribution = gmm)
  )
}
pca_methods <- anomaly_scores(pca10$scores, od = pca10$od) |>
  mutate(
    `Score distance` = pca10$sd,
    `Orthogonal distance` = pca10$od,
    `ROBPCA OD` = robpca10$od,
    .before = 1
  )

To compare the methods, we convert each set of scores to ranks (1 = most anomalous), and count how many of the ten planted images are among the 20 most anomalous images.

method_ranks <- function(scores) {
  scores |>
    mutate(across(everything(), \(x) rank(-x, ties.method = "min"))) |>
    bind_cols(fashion |> select(id, label, planted))
}
planted_found <- function(ranks, top = 20) {
  ranks |>
    filter(planted) |>
    summarise(across(-c(id, label, planted), \(x) sum(x <= top))) |>
    tidyr::pivot_longer(
      everything(),
      names_to = "Method",
      values_to = "Planted"
    )
}
pca_ranks <- method_ranks(pca_methods)
planted_found(pca_ranks) |> arrange(desc(Planted))
# A tibble: 10 × 2
   Method              Planted
   <chr>                 <int>
 1 ROBPCA OD                 7
 2 Isolation forest          7
 3 Score distance            6
 4 Orthogonal distance       5
 5 kNN                       5
 6 GLOSH                     5
 7 Stahel-Donoho             5
 8 LOF                       4
 9 Stray                     4
10 GMM surprisal             4

Every method finds between 4 and 7 of the planted images. Apart from the GMM surprisals, which miss the trousers, all methods put the two bags and the trousers in their top 20, and each method finds between one and three of the four ankle boots. The sandals are harder to find: the high-heeled sandal is an orthogonal outlier, so methods that use only the scores miss it, while the orthogonal distances, and the isolation forest which uses them, find it. The other two sandals are not in any method’s top 20: image 26 is about the same size, shape and darkness as a sneaker, while image 27 has thin straps and little dark area, so most of its pixels match the pale background of a sneaker image. Ankle boot 23, a regular observation in the classical outlier map, is not in the top 100 for any method; it has a low profile and a pale sole, and looks much like a high-top sneaker.

The results from ROBPCA, the isolation forest, and the Stahel–Donoho outlyingness depend on random subsets or random directions. Refitting with other seeds changes the number of planted images in their top 20 by at most one.

Figure 14.5 shows the ten most anomalous images for each method, with the planted images outlined. Most of the unplanted images are unusual sneakers: high-tops, wedge heels, and sneakers with bold contrasting patterns. Score distance, kNN, GLOSH, the isolation forest and Stahel–Donoho outlyingness largely agree: every pair of them shares at least five of their ten most anomalous images, and kNN and GLOSH have the same top ten. LOF and the GMM surprisals behave differently, sharing at most four of their top ten with any other method. Most of the planted images lie in the sparse outer region of the score distribution (Figure 14.3), where the sneakers that surround them are also sparse, so their local density relative to their neighbours (which is what LOF measures) is not unusual; only the trousers and one bag are in the LOF top ten. The Gaussian mixture model chosen by Mclust() has 9 components, one of which (with 79 images) covers the dark region containing the two bags and three of the ankle boots. So these images are not in a region of especially low density under the fitted mixture: the two bags rank below four sneakers, and two of the three ankle boots in that component are outside the top 20.

Code
plot_top <- function(ranks, top = 10) {
  methods <- setdiff(names(ranks), c("id", "label", "planted"))
  best <- ranks |>
    tidyr::pivot_longer(
      all_of(methods),
      names_to = "method",
      values_to = "rank"
    ) |>
    group_by(method) |>
    slice_min(rank, n = top, with_ties = FALSE) |>
    mutate(rank = row_number()) |>
    ungroup() |>
    mutate(method = factor(method, levels = methods))
  tibble(
    method = rep(best$method, each = 784),
    rank = rep(best$rank, each = 784),
    value = c(t(fashion$pixels[match(best$id, fashion$id), ])),
    col = rep(rep(1:28, 28), nrow(best)),
    row = rep(rep(1:28, each = 28), nrow(best))
  ) |>
    ggplot(aes(x = col, y = row, fill = value)) +
    geom_raster() +
    geom_rect(
      data = best |> filter(planted) |> mutate(lo = 0.5, hi = 28.5),
      aes(xmin = lo, xmax = hi, ymin = lo, ymax = hi),
      fill = NA,
      col = "#D55E00",
      linewidth = 0.8,
      inherit.aes = FALSE
    ) +
    scale_y_reverse() +
    scale_fill_gradient(low = "white", high = "black", guide = "none") +
    facet_grid(method ~ rank, switch = "y") +
    coord_equal() +
    theme_void() +
    theme(
      strip.text.y.left = element_text(angle = 0, hjust = 1, size = 9),
      strip.text.x = element_blank()
    )
}
plot_top(pca_ranks)
Figure 14.5: The ten most anomalous images according to each method applied to the first ten principal component scores, ordered from left to right. The planted images are outlined.

14.3 Image features

The second strategy is to describe each image by a few interpretable features, and then look for anomalies among the features. This is the same idea used for time series in Section 12.3. The choice of features determines which anomalies can be found, since an image that is unusual in a way that none of the features measure will look ordinary.

We will compute seven simple features from each image. Pixels with intensity above 0.1 (on the 0–1 scale) are treated as part of the item, and the rest as background.

  • darkness: the mean intensity of the item pixels;
  • contrast: the standard deviation of the intensity of the item pixels;
  • area: the proportion of the image covered by the item;
  • aspect: the height of the item divided by its width;
  • vcentre: the row of the centre of mass of the intensity (larger values are lower in the image);
  • edges: the total absolute difference between horizontally and vertically adjacent pixels, divided by the number of item pixels;
  • entropy: the entropy of the intensities of the item pixels, grouped into nine bins of width 0.1.
image_features <- function(x) {
  img <- matrix(x / 255, 28, 28, byrow = TRUE)
  item <- img > 0.1
  rows <- which(rowSums(item) > 0)
  cols <- which(colSums(item) > 0)
  grad_x <- img[, -1] - img[, -28]
  grad_y <- img[-1, ] - img[-28, ]
  counts <- hist(img[item], breaks = seq(0.1, 1, by = 0.1), plot = FALSE)$counts
  prop <- counts[counts > 0] / sum(counts)
  tibble(
    darkness = mean(img[item]),
    contrast = sd(img[item]),
    area = mean(item),
    aspect = length(rows) / length(cols),
    vcentre = sum(row(img) * img) / sum(img),
    edges = (sum(abs(grad_x)) + sum(abs(grad_y))) / sum(item),
    entropy = -sum(prop * log(prop))
  )
}
features <- lapply(seq_len(nrow(fashion)), \(i) {
  image_features(fashion$pixels[i, ])
}) |>
  bind_rows()
features
# A tibble: 1,010 × 7
   darkness contrast  area aspect vcentre edges entropy
      <dbl>    <dbl> <dbl>  <dbl>   <dbl> <dbl>   <dbl>
 1    0.467    0.245 0.269  0.429    14.9 0.477    2.07
 2    0.543    0.157 0.273  0.393    16.7 0.361    1.85
 3    0.719    0.219 0.305  0.464    16.3 0.397    1.84
 4    0.586    0.322 0.270  0.464    14.9 0.542    1.97
 5    0.672    0.284 0.305  0.5      14.9 0.471    2.01
 6    0.776    0.203 0.329  0.607    15.6 0.384    1.49
 7    0.365    0.180 0.337  0.607    17.4 0.366    1.83
 8    0.559    0.218 0.258  0.444    15.3 0.395    1.96
 9    0.614    0.157 0.321  0.5      16.5 0.366    1.79
10    0.510    0.240 0.265  0.519    15.8 0.509    2.13
# ℹ 1,000 more rows

Nearly 90% of the sneakers span the full width of 28 pixels, which is why we use the aspect ratio rather than the height and width separately: a variable that takes the same value for most images has a robust scale of zero, which breaks the robust scaling. We compute robust multivariate z-scores of the features using mvscale() (Section 3.7), and then apply the same methods as before, along with the robust Mahalanobis distance, which is the length of each vector of robust z-scores.

Z <- mvscale(features)
feature_methods <- anomaly_scores(as.matrix(Z)) |>
  mutate(Mahalanobis = sqrt(rowSums(Z^2)), .before = 1)
feature_ranks <- method_ranks(feature_methods)
planted_found(feature_ranks)
# A tibble: 8 × 2
  Method           Planted
  <chr>              <int>
1 Mahalanobis            5
2 kNN                    5
3 LOF                    5
4 GLOSH                  5
5 Isolation forest       5
6 Stray                  5
7 Stahel-Donoho          5
8 GMM surprisal          5

Every method finds the same five planted images: the trousers (which are the most anomalous image for every method), the high-heeled sandal, the two bags, and one ankle boot. The trousers are much taller than they are wide, and the high-heeled sandal is also taller than it is wide, unlike any sneaker. The two bags and ankle boot 21 cover more of the image than any sneaker. The other three ankle boots are no taller than the tallest high-top sneakers, and none of the seven features separates them clearly from the sneakers.

Figure 14.6 shows that the features also pick out different unplanted images from those found using the principal components. Most of these have unusually uniform intensities, and so low entropy; some are very pale, and others are dark all over. Four of the five unplanted images found by at least five of the eight methods are very pale sneakers. These were not unusual on the principal components: their scores on the first component are not extreme, as many sneakers are paler on that measure, and their orthogonal distances are below the median, with the two palest among the best reconstructed of all the images.

Code
plot_top(feature_ranks)
Figure 14.6: The ten most anomalous images according to each method applied to the seven image features, ordered from left to right. The planted images are outlined.

The principal components are learned from the data, and so can detect any image that is a poor fit to the main patterns of variation, even when we could not say in advance what would make it unusual. But each component is a weighted combination of hundreds of pixels, so it is hard to explain why an image is anomalous. Features are chosen by the analyst, so they only detect the kinds of anomaly the analyst anticipated, but the anomalies they find are easy to explain.

For larger and more complex images, such as photographs, hand-crafted features like these are usually replaced by features learned by a convolutional neural network that has been trained on a large collection of images. The output of one of the final layers of the network provides a few hundred features for each image, and any of the multivariate methods in this book can then be applied to those features.

14.4 Video data

The PETS2006 video, from the CDnet 2014 change detection benchmark (Wang et al., 2014), was taken by a fixed camera overlooking the concourse of a railway station. For most of the video, a man stands near the glass wall at the back of the concourse with a bag, while other people walk past. He then puts the bag down, and eventually walks away, leaving it behind.

The fetch_pets2006() function in the weird package downloads all 1200 frames of the video, converted to greyscale. To keep the data manageable, each frame has been reduced from 720 \times 576 pixels to 120 \times 96 pixels by averaging over blocks of 6 \times 6 pixels. The data are returned as a tibble, with the frame numbers in frame and the pixel intensities (from 0 for black to 255 for white) in the matrix column pixels, with one row per frame. The benchmark uses the first 299 frames to initialise background models, and evaluates methods on frames 300 to 1200, so we use the same frames here.

pets2006 <- fetch_pets2006() |>
  filter(frame >= 300)
pets2006
# A tibble: 901 × 2
   frame pixels[,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9] [,10] [,11] [,12]
   <int>      <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
 1   300         87    52    52    15    48    39    31     8     7     8    28    17
 2   301         87    52    52    15    49    40    30     7     7     8    29    17
 3   302         87    52    52    15    49    40    30     8     6     8    29    17
 4   303         87    52    52    15    48    40    31     8     6     8    29    16
 5   304         87    52    52    15    49    40    30     7     6     8    28    16
 6   305         88    53    52    15    50    40    31     8     6     9    28    16
 7   306         87    53    51    14    47    39    31     7     6     9    29    15
 8   307         87    53    51    15    47    40    30     7     7    10    28    16
 9   308         87    53    52    15    48    40    30     7     7     9    28    15
10   309         87    53    51    14    48    40    30     7     6     8    28    16
# ℹ 891 more rows
# ℹ 1 more variable: pixels[13:11520] <int>

As with the Fashion-MNIST images, we rescale the intensities to lie between 0 and 1. Figure 14.7 shows six of the frames.

V <- pets2006$pixels / 255
Code
# Plot rows of a matrix as video frames
plot_frames <- function(values, titles, height = 96, width = 120, ...) {
  values <- matrix(values, ncol = height * width)
  titles <- factor(rep(titles, each = height * width), levels = unique(titles))
  tibble(
    title = titles,
    value = c(t(values)),
    row = rep(rep(seq_len(height), width), nrow(values)),
    col = rep(rep(seq_len(width), each = height), nrow(values))
  ) |>
    ggplot(aes(x = col, y = row, fill = value)) +
    geom_raster() +
    scale_y_reverse() +
    facet_wrap(~title, ...) +
    coord_equal() +
    theme_void()
}
shown <- match(c(450, 735, 900, 951, 1101, 1197), pets2006$frame)
plot_frames(V[shown, ], paste("Frame", pets2006$frame[shown]), ncol = 3) +
  scale_fill_gradient(low = "black", high = "white", guide = "none")
Figure 14.7: Six frames from the PETS2006 video, after conversion to greyscale and downsampling to 120 by 96 pixels.

Anomalous frames

If each frame is treated as an observation, the video becomes a 901 \times 11520 data matrix, and we can use the same tools as for the Fashion-MNIST images. The background is the same in every frame, so most of the variation between frames is due to the people moving through the scene, and to small changes in lighting. A frame containing an unusual event should then be poorly reconstructed by principal components fitted to the bulk of the frames, giving it a large orthogonal distance. We use ROBPCA, so that the components are not pulled towards the unusual frames, and let PcaHubert() choose the number of components.

set.seed(1967)
pets_robpca <- PcaHubert(V)
# Number of components chosen by PcaHubert()
pets_robpca@k
[1] 10
Code
pets_frames <- tibble(
  frame = pets2006$frame,
  od = pets_robpca@od,
  above_cutoff = od > pets_robpca@cutoff.od
)
frames_above <- pets_frames$frame[pets_frames$above_cutoff]
period1 <- range(frames_above[frames_above < 850])
period2 <- range(frames_above[frames_above > 850])

Figure 14.8 shows the orthogonal distance of each frame over time. The dashed line is the orthogonal distance cutoff used in the outlier map (Section 9.4), so a frame above it would be classified as an orthogonal outlier or a bad leverage point. The cutoff is the 97.5% quantile of a Normal approximation to the distribution of \text{OD}^{2/3}, so we would expect only about 23 of the 901 frames to lie above it by chance, but 150 do. These frames all fall in two periods: between frames 670 and 753, when a person walks across the foreground close to the camera (frame 735 in Figure 14.7), and between frames 904 and 1010, when the man puts down his bag while two other people walk past in the foreground (frame 951; they are just coming into view in frame 900). In the middle of the first period, from frame 682 to frame 722, the orthogonal distance dips below the cutoff while the person crosses the middle of the concourse. People in the foreground cover many more pixels than people at the back of the concourse, so they dominate the orthogonal distances.

Code
pets_frames |>
  ggplot(aes(x = frame, y = od)) +
  geom_line(col = "grey50") +
  geom_point(
    data = filter(pets_frames, above_cutoff),
    col = "#D55E00",
    size = 1
  ) +
  geom_hline(yintercept = pets_robpca@cutoff.od, linetype = "dashed") +
  labs(x = "Frame", y = "Orthogonal distance")
Figure 14.8: Orthogonal distances of the PETS2006 frames from the robust principal components, with the orthogonal distance cutoff from the ROBPCA outlier map shown as a dashed line and frames above it in orange.

The orthogonal distances form a time series, so the methods of Chapter 12 could also be used here, for example to find frames that are unusual relative to the frames just before them.

Anomalous pixels

The orthogonal distance tells us when something happens, but not where. To find the pixels that differ from the background, we can treat each pixel location separately. The median of the 901 intensities at a pixel location gives a robust estimate of the background at that location, provided the location is covered by a moving object in fewer than half of the frames. The difference between a frame and this median background is then large only where something is in front of the background.

background <- apply(V, 2, median)
resid <- sweep(V, 2, background)

Figure 14.9 shows the absolute residuals for the frame with the largest orthogonal distance in each of the two periods found above, and for the last frame of the video, which has a small orthogonal distance.

Code
# Frame with the largest orthogonal distance in each period, and the last frame
top_frame <- function(period) {
  in_period <- pets_frames$frame >= period[1] & pets_frames$frame <= period[2]
  which(in_period)[which.max(pets_frames$od[in_period])]
}
resid_shown <- c(top_frame(period1), top_frame(period2), nrow(V))
titles <- paste("Frame", pets2006$frame[resid_shown])
grey_scale <- scale_fill_gradient(low = "black", high = "white", guide = "none")
patchwork::wrap_plots(
  plot_frames(V[resid_shown, ], titles, ncol = 3) + grey_scale,
  plot_frames(pmin(abs(resid[resid_shown, ]), 0.3), titles, ncol = 3) +
    grey_scale +
    theme(strip.text = element_blank()),
  ncol = 1
)
Figure 14.9: Three PETS2006 frames (top row) and their absolute differences from the median background (bottom row). The first two frames have the largest orthogonal distances in the two periods above the cutoff in Figure 14.8; the third is the final frame of the video. Lighter pixels have larger differences.

In frames 675 and 968, the large residuals are where the people are. By frame 1200, the man has walked away, leaving his bag behind. The bag covers too few pixels to give the frame a large orthogonal distance, but it stands out in the residuals, because it is present in fewer than half of the frames, and so is not part of the median background.

14.5 Summary

An image with m pixels is a point in m-dimensional space, so collections of images can be analysed with the high-dimensional methods of Chapter 9. Because neighbouring pixels are strongly correlated, the images lie close to a lower-dimensional subspace, and principal component analysis can help find it. Each image is then summarised by its scores on the first k components and by its orthogonal distance from the subspace. Any multivariate anomaly detection method can be applied to the scores, while the orthogonal distance identifies images whose shape is not captured by the main patterns of variation, which the scores alone can miss. Robust PCA gives components that describe the bulk of the images rather than being pulled towards the anomalies.

An alternative approach is to compute a small number of interpretable features from each image, scale them robustly, and apply the same multivariate methods to the features. Features can only reveal the kinds of anomaly they were designed to measure, but the anomalies they find are easy to explain. Principal components need no features to be chosen in advance, but the anomalies they find are harder to interpret. The two approaches tend to find different anomalies, so it is worth using both.

A video is a collection of images (frames) with a time order, so the methods for images apply directly to videos, with each frame treated as an observation. For a fixed camera, robust principal components describe the static background and its usual variation, and the orthogonal distance of each frame measures how much is happening in it. The time order and the fixed camera add two things. The orthogonal distances form a time series, which can be monitored with the methods of Chapter 12. And because each pixel location shows the same part of the scene in every frame, the median over frames gives a robust estimate of the background, and the residuals show where in a frame something differs from the background.