Probability theory defines how we can manipulate and utilize probability distributions, but it provides no guidance for which probability distributions we might want to utilize in a given application. Indeed the choices of probability distributions over a given space are immense, especially when the space is high-dimensional. Consequently probabilistic modeling is often overwhelming for practitioners trying to reason about how something is distributed across a given space.
Fortunately we can exploit conditional probability theory to build up high-dimensional probability distributions from many one-dimensional probability distributions about which we can more easily reason. With only a modest collection of probability distributions at their disposal a practitioner has to potential to construct a wide array of sophisticated probability distributions. A cache of interpretable, well-understood probability distributions is a critical piece of any practitioner’s statistical toolkit.
In this case study I review common probability density functions which specify some elementary probability distributions in the context of a particular parameterization of a given space, \(X\). Specifically I review families of probability density functions, \(\pi(x ; \theta)\) indexed by their own set of parameters, \(\theta \in \Theta\). As with the parameterization of the ambient space, \(X\), the parameterization of the family, \(\Theta\), is not unique and I will discuss alternative family parameterizations that provide useful interpretative or mathematical properties.
I will begin by introducing some important properties that motivate particularly useful building blocks. Then I will use these properties to populate our statistical palette, reviewing the elementary families of probability density functions that arise from these considerations. Finally I will discuss additional families that derive from pushing these elementary families forward along functions. In each review I will focus on the probabilistic behaviors of each family and try to build intuition as to how they can be be best deployed in practice.
Before that, however, a quick word on vocabulary. The precision of mathematics often requires ungainly or counterintuitive nomenclature. For example it quickly becomes annoying to repeatedly say, let alone type, “families of probability density functions” over and over again. Because of that most statisticians, including your humble author, will often get a little sloppy with their language. Instead of saying “family of probability density functions” we’ll often just say “probability density function” with the family implied, or even just “probability density” or “density”. Worse we’ll confound probability distributions with their density representations, saying for example “normal distribution” instead of “normal family of probability density functions in the context of a given parameterization of the ambient space”. The worst amongst us will hide behind the dreadful “random variable” abstraction. While this haphazard language is inevitable and relatively harmless once you’re comfortable with probability theory it can be a major impediment towards becoming comfortable in the first place. Consequently I will be extremely pedantic with my language here to facilitate the first steps into probabilistic proficiency.
Once that experience has been developed one can continue learning about families of probability density functions from the many available statistical resources, such as [1], [2], and [3]. Wikipedia is another fantastic reference for families of probability density functions and their properties.
A useful way to isolate particular families of probability distributions is to impose certain mathematical constraints that ensure convenient probabilistic properties. When these constraints are defined only in the context of a given parameterization of the ambient space they define not the probability distributions themselves but rather their probability density function representations. Care is needed to respect this context because reparameterizing the ambient space will modify these criteria, and then the families of probability density functions arising from them.
In this section I discuss two properties that motivate especially useful families of probability density functions. To demonstrate these properties, and provide some explicit context for their consequences, I will utilize the normal family of probability density functions that I introduced in my probability theory case study, \[ \pi(x; \mu, \sigma) = \frac{1}{ \sqrt{2 \pi \sigma^{2} } } \exp \left( - \frac{1}{2} \left( \frac{ x - \mu}{\sigma} \right)^{2} \right) \] A more thorough discussion of the normal family and its properties will be given in Section 2.3.
Sometimes a families might satisfy multiple motivating criteria, making them particularly powerful both in theory and in practice. Such overlapping motivations will help to explain why some families, like the normal family, end up so ubiquitous in statistical practice.
Often our goal is to probabilistically model the outcome of some complex system that aggregates many individual, component probability distributions together. As the component distributions are aggregated some behaviors of the component distributions might add incoherently, washing out in the distribution that emerges from this aggregation. Some behaviors in the component distributions, however, might add more coherently, reinforcing each other to persist into the final emergent distribution.
While modeling the entire system, and all of the individual components, will provide the most precise analysis, in many cases it is sufficient to directly model only the emergent distribution itself. How exactly, however, can we identify the kind of distribution that would arise from a given complex system? It turns out that the only distributions that can emerge are those specified within very special families of probability density functions that are closed under the fundamental operation that aggregates the component distributions.
In this section I build motivation for this result, introducing the exceptional stable families that exhibit the necessary closure and manifest powerful central limit theorems that we can exploit to model emergent behavior.
In mathematics closure refers to the circumstance where a common space serves as both the input and output of a given operation. For example addition will always map two real numbers to another real number so we can say that the real numbers are closed under addition. Alternatively division will in general map two integers not to another integer but rather to a rational number. Consequently the integers are not closed under division.
The operation that is relevant to understanding the emergent behavior of a complex system is one that maps a collection of component probability distributions to some outcome probability distribution through an intermediate joint probability distribution.
Consider, for example, modeling each component with replicates of a single one-dimensional space, \(X\), and modeling the complex system with the product space, \(X_{N} = \prod_{n = 1}^{N} X\). We can specify a probability distribution over each component using individual, and in general heterogenous, probability density functions, \(\pi_{n}(x_{n})\).
These component probability density functions then define an independent joint probability distribution over \(X_{N}\) through the joint probability density function, \[ \pi_{N}(x_{1}, \ldots, x_{N}) = \prod_{n = 1}^{N} \pi_{n}(x_{n}). \] In the special case where each component distribution is the same, \(\pi_{n}(x_{n}) = \pi(x_{n})\), we say that the components are independent and identically distributed, commonly condensed to IID.
Let’s now make the additional assumption that any emergent behavior takes place in another copy of our base space, \(X\). Given an independent joint distribution we can model the aggregration process with a projection operator that maps the product space to the outcome space, \[ \varpi: X_{N} \rightarrow X. \]
A common example, and the one that we will utilize most often is an additive projection, \[ \begin{alignat*}{6} \varpi_{+} :\; &\hspace{8mm}X_{N}& &\rightarrow& \; &X& \\ &(x_{1}, \ldots, x_{N})& &\mapsto& &\sum_{n = 1}^{N} x_{n}&. \end{alignat*} \] Keep in mind that these projection are defined only in the context of a particular parameterization of \(X\), and hence will change under reparameterization. For example if \(X = \mathbb{R}\) and we reparameterize the output space with the exponential map, \[ \begin{alignat*}{6} \exp :\; &\mathbb{R}& &\rightarrow& \; &\mathbb{R}^{+}& \\ &x& &\mapsto& &\exp(x)&, \end{alignat*} \] then the additive projection becomes \[ \begin{align*} y &= \exp(x) \\ y &= \exp(x_{1} + \ldots + x_{N}) \\ y &= \exp(x_{1}) \cdot \ldots \cdot \exp(x_{N}) \\ y &= y_{1} \cdot \ldots \cdot y_{N}. \end{align*} \] which corresponds to a multiplicative projection, \[ \begin{alignat*}{6} \varpi_{\times} :\; &\hspace{8mm}X_{N}& &\rightarrow& \; &X& \\ &(x_{1}, \ldots, x_{N})& &\mapsto& &\prod_{n = 1}^{N} x_{n}&, \end{alignat*} \]
Similarly if \(X = \mathbb{R}^{+}\) and we apply a logarithmic reparameterization, \[ \begin{alignat*}{6} \log :\; &\mathbb{R}^{+}& &\rightarrow& \; &\mathbb{R}& \\ &x& &\mapsto& &\log(x)&, \end{alignat*} \] then the multiplicative projection becomes an additive projection, \[ \begin{align*} y &= \log(x) \\ y &= \log(x_{1} \cdot \ldots \cdot x_{N}) \\ y &= \log(x_{1}) + \ldots + \log(x_{N}) \\ y &= y_{1} + \ldots + y_{N}. \end{align*} \]
Given a projection operator we can then push the joint probability density function forward through the projection, defining a new probability density function on \(X\), \(\pi_{\varpi}(x)\).
This construction of an independent joint probability density function and a resulting pushforward probability density function holds for any number of components, \(N\). Repeating the construction for increasing \(N\) defines a sequence of pushforward probability density functions starting with \(\pi_{\varpi, 1}(x) = \pi_{1}(x)\) and then continuing onto the pushforwards \(\pi_{\varpi, 2}(x), \ldots, \pi_{\varpi, N}(x)\).
Any such sequence of pushforward probability density functions then forms a path through the space of all possible probability density functions on \(X\),
Closure with respect to a given projection operator can then be defined with respect to the initial component probability density functions and the corresponding pushforward probability density function. For example we could talk about the closure of a particular probability density function, \(\pi(x)\), if the independent and identically distributed joint probability density function defined by \(\pi(x)\) on each component pushes forward to the same probability density function, \[ \pi_{\varpi}(x) = \pi(x). \]
More commonly, however, we talk about closure of a family of probability density functions. An entire family is closed under a given projection if the independent joint distribution defined by any component probability density functions within the family, \(\pi_{n}(x_{n}) = \pi(x_{n}; \theta_{n})\), pushes forward to a distribution on the outcome space specified by any member of the given family, \[ \pi_{\varpi}(x) = \pi(x; \theta_{\varpi}), \] for some \(\theta_{\varpi} \in \Theta\).
In other words any sequence of probability density functions will stay within the family.
When \(X\) is subset of the real numbers then we can also consider a slightly more general projection operator that introduces weights and an offset, \[ \begin{alignat*}{6} \varpi_{+} :\; &\hspace{15mm}X_{N} \times \mathbb{R}^{N} \times \mathbb{R}& &\rightarrow& \; &X& \\ &(x_{1}, \ldots, x_{N}), (w_{1}, \ldots, w_{N}), b & &\mapsto& &\sum_{n = 1}^{N} w_{n} \cdot x_{n} + b&. \end{alignat*} \] If an independent and identically distributed joint probability density function build up from a given family pushes forward to another member of that family for any choice of weights and offset then that family is denoted a stable family of probability density functions [4]. Stable families are exceedingly rare but have a myriad of interesting and useful mathematical properties, namely the closure introduced above.
Consider, for example, the normal family. Upon multiplication by a real-valued weight a normal probability density function with location \(\mu\) and scale \(\sigma\) pushes forward to a new normal probability density function with location \(w_{n} \cdot \mu\) and scale \(w_{n} \cdot \sigma\), \[ \mathcal{N}(x \mid w_{n} \cdot \mu, w_{n} \cdot \sigma). \]
The pushforward of an independent and identically distributed joint probability density function along the weighted projection then gives another member of the normal family, \[ \pi_{\varpi_{+}}(x) = \mathcal{N}(x \mid (w_{1} + w_{2}) \cdot \mu, (w_{1} + w_{2}) \cdot \sigma). \] We can apply this result recursively to give \[ \pi_{\varpi_{+}}(x) = \mathcal{N}( x \mid (\sum_{n = 1}^{N} w_{n}) \cdot \mu, (\sum_{n = 1}^{N} w_{n}) \cdot \sigma). \] Adding in the offset still doesn’t push us outside of our normal family, yielding the final normal probability density function \[ \pi_{\varpi_{+}}(x) = \mathcal{N}(x \mid \sum_{n = 1}^{N} w_{n} \cdot \mu + b, \sum_{n = 1}^{N} w_{n} \cdot \sigma). \] No matter the value of the weights or the offset, this construction will push one normal probability density function into another.
Moreover, consider decomposing the offset, \(b\), into a sum of \(N\) offsets, \(b = \sum_{n = 1}^{N} b_{n}\). We can then associate a weight and an offset to each of the component probability density functions, \[ w_{n}, b_{n}, \mathcal{N}(x_{n} \mid \mu, \sigma). \] Pushing each component probability density function through the local transformation \(x_{n} \mapsto w_{n} \cdot x_{n} + b_{n}\) yields a collection of non-identical normal probability density functions. The stability condition then implies that the additive projection of any heterogeneous collection of normal probability density functions is also a normal probability density function, which is exactly the closure property we introduced above.
The closure inherit to stable families is relevant only when each component is modeled by a probability distribution within a particular stable family. What use does that closure have, however, when the more realistic complex systems of applied interest would not be accurately modeled with only a single family, instead requiring heterogenous component probability density functions not contained within any single family?
If we begin within a stable family of probability density functions then any sequence of projective pushforward probability density functions will be contained within that family by construction. These stable families, however, also serve as attractors, probabilistic sirens that lull arbitrary sequences into their grasp.
The convergence of a particular sequence of probability density functions towards a stable family is known as a central limit theorem. A central limit theorem implies that the stable family exactly captures or exceedingly well approximates the aggregation modeled by the additive projection for large enough \(N\). Such large \(N\) behavior is known as asymptotics.