Transforming Probability Spaces

All probability theory chapters can be found here.

A repository containing all of the files used to generate this chapter is available on GitHub.

We learned how basic structures, such as metrics and topologies, are modified when we transform the ambient set in In Chapter 2. In this chapter, we’ll learn how measure-theoretic structures are transformed, including probability distributions, expectation values, and probability density functions.

1 Transforming \sigma-Algebras

We’ve already seen how to transform subsets in Chapter 2. Given two spaces X and Y and any function f : X \rightarrow Y, we can always push subsets in X forward to subsets in Y by combining the output of each point in the input subset (Figure 1 (a)), \begin{alignat*}{6} f_{*} :\; & 2^{X} & &\rightarrow& \; & 2^{Y} & \\ & \mathsf{x} & &\mapsto& & f_{*} \mathsf{x} = \{ f(x) \mid x \in \mathsf{x} \} &. \end{alignat*} Similarly, we can pull subsets in Y back to subsets in X by combining the preimages of every output point (Figure 1 (b)), \begin{alignat*}{6} f^{*} :\; & 2^{Y} & &\rightarrow& \; & 2^{X} & \\ & \mathsf{y} & &\mapsto& & \phi^{*} \mathsf{y} = \{ x \in X \mid f(x) \in \mathsf{y} \} &. \end{alignat*}

 

(a)

 

 

(b)

 

Figure 1: Transformations of a set induce transformations of subsets. (a) Subsets of the input space \mathsf{x} \subset X can be pushed forward into subsets of the output space, f_{*}(\mathsf{x}) \subset Y. (b) Similarly, subsets of the output space \mathsf{y} \subset Y can be pulled back into subsets of the input space, f^{*}(\mathsf{y})\subset X.

There is an asymmetry, however, between these two induced transformations when we consider set operations. Both the pushforward and pullback maps are compatible with the union operation, \begin{align*} f_{*}(\cup_{i} \mathsf{x}) &= \cup_{i} f_{*}(\mathsf{x}) \\ f^{*}(\cup_{i} \mathsf{y}) &= \cup_{i} f^{*}(\mathsf{y}), \end{align*} but only the pullback map is always compatible with the intersection operation, f^{*}(\cap_{i} \mathsf{y}) = \cap_{i} f^{*}(\mathsf{y}). In general, the intersection of any collection of input subsets pushes forward to a subset of the intersection of the individual pushforward subsets (Figure 2). f_{*}(\cup_{i} \mathsf{x}) \subseteq \cup_{i} f_{*}(\mathsf{x}).

 

(a)

 

 

(b)

 

 

(c)

 

Figure 2: In general, (a) subset pushforwards are not consistent with all of the binary set operations. (b) The union of the pushforwards of two input subsets is always the same as the pushforward of the union of the input subsets. (c) On the other hand, the intersection of the pushforwards of two input subsets is generally only a subset of the pushforward of the intersection of the two input subsets. In other words, the pushforward operation commutes with the union operator, allowing us to apply them in either order and achieve the same result, but the pushforward operation does not commute with the intersection operator. The pullback operation, however, commutes with both the union and intersection operators.

This has an immediate consequence for \sigma-algebras: the pushforward of the intersection of measurable input subsets isn’t necessarily a measurable output subset. Consequently, a \sigma-algebra of measurable subsets on X doesn’t always push forward into a \sigma-algebra of measurable subsets on Y.

On the other hand, a \sigma-algebra on Y does always pull back to a well-behaved \sigma-algebra on X. If \mathcal{Y} is a \sigma-algebra on Y, then f^{*} \mathcal{Y} = \{ f^{*}(\mathsf{y}) \mid \mathsf{y} \in \mathcal{Y} \} is referred to as the pullback \sigma-algebra along f, or the \sigma-algebra generated by f.

In order for a function f: X \rightarrow Y to preserve the structure of two measurable spaces (X, \mathcal{X}) and (Y, \mathcal{Y}), every measurable subset \mathsf{y} \in \mathcal{Y} needs to pull back to a measurable subset in \mathcal{X}, f^{*}(\mathsf{y}) \in \mathcal{X}, Equivalently, f preserves measurable structure only when f^{*} \mathcal{Y} \subseteq \mathcal{X}. Note that this does not require that every measurable subset \mathsf{x} \in \mathcal{X} pushes forward to a measurable subset in \mathcal{Y}; we can safely ignore some measurable input subsets without compromising the \sigma-algebra on the output space.

Functions that preserve measurable structure are known as (\mathcal{X}, \mathcal{Y})-measurable functions. When the \sigma-algebras on the input and output space are unambiguous, this is often shortened to just measurable functions. I will also use the more compact notation f : (X, \mathcal{X}) \rightarrow (Y, \mathcal{Y}) to denote (\mathcal{X}, \mathcal{Y})-measurable functions.

We’ve already encountered measurable functions in Chapter 5 when I introduced measure-informed integration. A real-valued function f : X \rightarrow \mathbb{R} can be integrated on the measure space (X, \mathcal{X}, \mu) if every half-open interval on the output space pulls back to a measurable subset on the input space, f^{*}( \, (-\infty, x) \, ) \in \mathcal{X}. This condition, however, is equivalent to every subset in the Borel \sigma-algebra of the real line, \mathsf{y} \in \mathcal{B}_{\mathbb{R}}, pulling back to a measurable subset on the input space, f^{*}( \mathsf{y} ) \in \mathcal{X}.

In other words, what we referred to as “\mathcal{X}-measurable real-valued functions” in Chapter 5 are more formally (\mathcal{X}, \mathcal{B}_{\mathbb{R}})-measurable functions. The former notation takes the Borel \sigma-algebra on the real line for granted, while the latter makes it more explicit. This is a common shorthand – references to “measurable functions” without any specification almost always imply Borel \sigma-algebras on the input and output spaces.

Fortunately, this shorthand isn’t too problematic in practice because we will almost always be working with measures defined over Borel \sigma-algebras derived from the topological structure of the relevant spaces. Consequently, a function f : X \rightarrow Y mapping the Borel measurable space (X, \mathcal{B}_{X}) into the Borel measurable space (Y, \mathcal{B}_{Y}) might be described as (\mathcal{B}_{X}, \mathcal{B}_{Y})-measurable, Borel measurable, or even just “measurable”.

Continuous functions that respect the topological structure of the input and outputs spaces are always Borel measurable, but so too are functions that are only piece-wise continuous. Ultimately, Borel measurability is a much weaker condition than topological continuity; we can map open subsets in the output space into not only open subsets in the output space, but also closed subsets in the output space, and even any subset that we can derive from unions and intersections of open and closed subsets in the input space.

Figure 3: Most of the functions that we encounter in practice are measurable with respect to the standard \sigma-algebras. For example, functions f: \mathbb{R} \rightarrow \mathbb{R} that are smooth, non-differentiable but continuous, and even discontinuous at a countable number of points are all (\mathcal{B}_{\mathbb{R}}, \mathcal{B}_{\mathbb{R}})-measurable.

When working with finite-dimensional spaces in practice, it is safe to assume that not only all but the most pathological subsets are measurable, but also that all but the most pathological functions are measurable. Infinite-dimensional spaces are another matter, but those spaces will largely be outside of the scope of this book.

2 Transforming Measures

Conveniently, the pullback of measurable subsets allows us to pushforward measures from the input space to a compatible measure on the output space. Given a (\mathcal{X}, \mathcal{Y})-measurable function f : X \rightarrow Y, any measure \mu : \mathcal{X} \rightarrow [0, \infty] defines a pushforward measure by the allocations \begin{alignat*}{6} f_{*} \mu :\; & \mathcal{Y} & &\rightarrow& \; & \mathbb{R}^{+} & \\ & \mathsf{y} & &\mapsto& & f_{*} \mu (\mathsf{y}) = \mu(f^{*}(\mathsf{y})) &. \end{alignat*} In words, the pushforward measure allocated to any measurable subset on the output space \mathsf{y} \in \mathcal{Y} is computed by pulling the subset back to the input space, f^{*}(\mathsf{y}) \in \mathcal{X}, and then evaluating the initial measure, \mu(f^{*}(\mathsf{y})) (Figure 4).

Figure 4: Because measurable output subsets pull back to measurable input subsets along measurable functions, we can push forward measure allocations. In particular, the pushforward measure allocated to the measurable output subset \mathsf{y} \in \mathcal{Y} is given by pulling it back to the input space, f^{*}(\mathsf{y}) \in \mathcal{X}, and then querying the initial measure, \mu(f^{*}(\mathsf{y})).

The exact interpretation of a pushforward measure will depend on the interpretation of the input measure and the transformation. Consider, for example, a probability distribution \pi defined on the input space that we interpret as quantifying uncertainty. This probability distribution captures our uncertainty about the inputs to a function f, while the pushforward probability distribution f_{*} \pi quantifies the corresponding uncertainty in the output of the function. In other words, the pushforward transformation propagates the initial uncertainty through the deterministic mapping.

At the same time, certain transformations can endow their corresponding pushforward measures with particular interpretations.

2.1 Pushforward Terminology

Pushforward measures are ubiquitous in applied probability theory, although they are often better known by other names.

Consider, for example, a finite input space X, X = \{ \blacksquare, \clubsuit, \bigcirc, \diamondsuit, \triangle, \bowtie \}, a finite output space Y, Y = \{ \heartsuit, \spadesuit, \bigstar \}, and a function f : X \rightarrow Y defined by the relations (Figure 5 (a)) \begin{align*} f(\blacksquare) &= \spadesuit \\ f(\clubsuit) &= \heartsuit \\ f(\bigcirc) &= \bigstar \\ f(\diamondsuit) &= \spadesuit \\ f(\triangle) &= \heartsuit \\ f(\bowtie) &= \spadesuit \end{align*}

These relationships between input and output points become particularly well-organized when we arrange the input elements into a table, with each row collecting all of the input elements that map to a particular output element (Figure 5 (b)). Conveniently, the pushforward measure allocated to each output atomic subset is then given by summing the input atomic subset allocations in each the corresponding rows (Figure 5 (c), Figure 5 (c)). In other words, the pushforward allocations fit nicely into the margins of the table.

(a)

(b)

(c)

(d)

Figure 5: On finite spaces (a) functions f: X \rightarrow Y (b) are naturally organized into tables, with each row collecting all of the input points that map to each output point. (c) Given an input measure \mu, (d) we can compute the pushforward measure allocations by summing the \mu allocations in each row and then displaying the results in the margins of the table.

Historically, this kinds of graphical organization motivated the term marginal measure to describe pushforward measures, or marginal probability distribution in the case of an input probability distribution. Today this terminology is common even when the input and output spaces are not finite, and the tabular representation of functions isn’t quite as useful (Figure 6).

 

(a)