Probability theory defines how we can manipulate and utilize probability distributions, but it provides no guidance for which probability distributions we might want to utilize in a given application. Indeed the choices of probability distributions over a given space are immense, especially when the space is high-dimensional. Consequently probabilistic modeling is often overwhelming for practitioners trying to reason about how something is distributed across a given space.

Fortunately we can exploit conditional probability theory to build up high-dimensional probability distributions from many one-dimensional probability distributions about which we can more easily reason. With only a modest collection of probability distributions at their disposal a practitioner has to potential to construct a wide array of sophisticated probability distributions. A cache of interpretable, well-understood probability distributions is a critical piece of any practitioner’s statistical toolkit.

In this case study I review common probability density functions which specify some elementary probability distributions in the context of a particular parameterization of a given space, \(X\). Specifically I review families of probability density functions, \(\pi(x ; \theta)\) indexed by their own set of parameters, \(\theta \in \Theta\). As with the parameterization of the ambient space, \(X\), the parameterization of the family, \(\Theta\), is not unique and I will discuss alternative family parameterizations that provide useful interpretative or mathematical properties.

I will begin by introducing some important properties that motivate particularly useful building blocks. Then I will use these properties to populate our statistical palette, reviewing the elementary families of probability density functions that arise from these considerations. Finally I will discuss additional families that derive from pushing these elementary families forward along functions. In each review I will focus on the probabilistic behaviors of each family and try to build intuition as to how they can be be best deployed in practice.

Before that, however, a quick word on vocabulary. The precision of mathematics often requires ungainly or counterintuitive nomenclature. For example it quickly becomes annoying to repeatedly say, let alone type, “families of probability density functions” over and over again. Because of that most statisticians, including your humble author, will often get a little sloppy with their language. Instead of saying “family of probability density functions” we’ll often just say “probability density function” with the family implied, or even just “probability density” or “density”. Worse we’ll confound probability distributions with their density representations, saying for example “normal distribution” instead of “normal family of probability density functions in the context of a given parameterization of the ambient space”. The worst amongst us will hide behind the dreadful “random variable” abstraction. While this haphazard language is inevitable and relatively harmless once you’re comfortable with probability theory it can be a major impediment towards becoming comfortable in the first place. Consequently I will be extremely pedantic with my language here to facilitate the first steps into probabilistic proficiency.

Once that experience has been developed one can continue learning about families of probability density functions from the many available statistical resources, such as [1], [2], and [3]. Wikipedia is another fantastic reference for families of probability density functions and their properties.

1 Eye of the Tiger

A useful way to isolate particular families of probability distributions is to impose certain mathematical constraints that ensure convenient probabilistic properties. When these constraints are defined only in the context of a given parameterization of the ambient space they define not the probability distributions themselves but rather their probability density function representations. Care is needed to respect this context because reparameterizing the ambient space will modify these criteria, and then the families of probability density functions arising from them.

In this section I discuss two properties that motivate especially useful families of probability density functions. To demonstrate these properties, and provide some explicit context for their consequences, I will utilize the normal family of probability density functions that I introduced in my probability theory case study, \[ \pi(x; \mu, \sigma) = \frac{1}{ \sqrt{2 \pi \sigma^{2} } } \exp \left( - \frac{1}{2} \left( \frac{ x - \mu}{\sigma} \right)^{2} \right) \] A more thorough discussion of the normal family and its properties will be given in Section 2.3.

Sometimes a families might satisfy multiple motivating criteria, making them particularly powerful both in theory and in practice. Such overlapping motivations will help to explain why some families, like the normal family, end up so ubiquitous in statistical practice.