Probabilistic Modeling and Statistical Inference

Decision making is a critical task that spans not just science and industry but arguably the entire human experience. At the same time, decision making in complex applications is notoriously subtle and counterintuitive.

How do we make a principled selection between a set of actions, \(A\)? This decision becomes straightforward if each possible action is assigned a utility, \[ \begin{alignat*}{6} U :\; &A& &\rightarrow& \; &\mathbb{R}^{+}& \\ &a& &\mapsto& &U(x)&, \end{alignat*} \] that quantifies the overall benefit of that action minus any costs. Given such a utility assignment we could then simply select the action with the highest utility, \[ a^{*} = \underset{a \in A}{\text{argmax}} U(a). \] In practice, however, even the most rigorous utility assignments will depend on circumstances we do not know with perfect certainty. Here we must rely on inferences to inform those circumstances and motivate effective decision making.

In this case study we’ll review the foundations of statistical models and statistical inference that advise principled decision making. We’ll place a particular emphasis on Bayesian inference, which utilizes probability theory and statistical modeling to encapsulate information from observed data and our own domain expertise.

I take an observational perspective here that presumes the existence of a true data generating process that generates the observations from which we learn. This perspective is not universal and clashes with some philosophical perspectives on statistics that are not uncommon in popular introductory texts. Fortunately, in most cases the differences between these perspectives don’t have any consequences for how the resulting methodology is implemented in practice. That said, it helps to keep in mind the differing perspectives when comparing this material to other literature. Prevalent frequentist texts include [1], [2] and [3]. There are a wide array of introductory Bayesian texts from objective Bayesian perspectives [4], subjective Bayesian perspectives [5], and more applied perspectives [6].

This material will presume a familiarity of probability theory at the level introduced in my probability theory and conditional probability theory case studies.

1 Probabilistic Modeling

Models are approximations of the complex dynamics that drive the observable phenomena in the world around us. They provide the setting in which we can formalize learning and decision making and hence are a foundational aspect to any rigorous analysis. In this section we review latent observational processes, the data generating processes they induce, and the statistical models that offer a spectrum of possible data generating behaviors.

1.1 The Observational Process

A typical abstraction in applied analysis is to assume that we are making decisions about how to interact with some latent phenomenon. Our inferential target is then the phenomenon itself.

No material phenomenon, however, exists in isolation. Any phenomenon relevant to statistical inquiry instead manifests within some latent system that includes, at the very least, the mechanisms with which we can interact with it. In other words we can observe the phenomenon of interest only through its surrounding environment.

In order to make the desired inferences we need an observational probe that explores the environment and, hopefully, the phenomenon of interest.

The interaction of the probe with the environment and the phenomenon defines an observational process which examines the latent system and induces explicit observations that are sensitive to the phenomenon of interest.

An observational process encompasses any methodology that measures, collects, curates, or manipulates observations sensitive to the latent phenomenon. This includes, for example, not only dedicated experiments explicitly designed to interrogate the phenomenon of interest but also retrospective analyses of incidental experiments or previously conducted experiments designed for other purposes.

In encapsulating the phenomenon of interest our probe intersects only part of the surrounding environment. Consequently the observational process will be sensitive to only those elements of the environment relevant to the probe. Well-designed experiments offer the promise of better isolating the target phenomenon, yielding a much cleaner observational process and more informative observations.

In general there is no unique procedure for interrogating an environment and the phenomena within it. In other words, there is never a singular experimental probe. Instead each of the many possible procedures results in its own probe.

Each probes then defines its own observational process.

These distinct observational processes can become the basis of independent analyses, or even components in a single joint analyses that aggregates data from multiple sources.

Consequently observational processes are bespoke to not only a given phenomenon but also the specific details of the probe used to investigate it, especially the environment with which the probe intersects. A critical aspect of statistical modeling is capturing enough of these details to isolate inferences about the latent phenomenon that can then be fairly compared across different experiments. This is especially true when the specific behavior of the environment or the probe is transient or otherwise not exactly repeatable. Without a rich understanding of the observational process phenomenological inferences can poorly generalize outside of the exact context of the given experiment, promoting more confusion than comprehension.

1.2 The True Data Generating Process

By definition an observational processes is only a conceptual abstraction. In order for it to take form as an explicit mathematical object we need to define exactly how observations are realized from an observational process. This process, however, is subtle and results not in constant outcomes but rather a variation of possible outcomes within an observation space, \(Y\).

For example the realized observations might exhibit variation due to physical, or ontological, stochasticity inherent to the latent phenomenon, its environment, or the probe used to interrogate them. This variation is also known as aleatoric randomness of the observational process. Ontological variation, for example, is commonly posited when modeling atomic decays that produce radiation at varying intervals and in varying directions, or when sampling individuals from a population.

At the same time the realized observations might also exhibit epistemological variation due to limitations in how accurately we can observe the latent system. Lengths measured with a ruler, for example, are limited by the precision of the markings and the resolution of the human eye; they will not in general be consistent from person to person or even a single person from time to time. Similarly, increasingly common digital data collection is limited by the discretization of analog inputs and the numerical precision of intermediate computations.

Epistemological variation is particularly common in the study of dynamical systems. Chaos theory, for example, arises when infinitesimal uncertainty in initial conditions is evolved through nonlinear, but deterministic, dynamics. This evolution results in probabilistic behavior of the system despite its fundamental determinism. Likewise thermodynamics is a theory of probabilistic behavior in many-particle systems that emerges from deterministic microscopic dynamics when only certain macroscopic quantities can be observed.

Ultimately, however, the distinction between these two sources of variation is ill-defined. For example, one can argue that seemingly ontological variation is a result of the imprecise observation of complex but deterministic dynamics, as as in chaos theory. Likewise one can argue that epistemological variation defines an ensemble of possible but unobserved, or counterfactual, outcomes which manifests as ontological variation if the experiment were hypothetically repeated.

Fortunately the exact philosophical nature of this variation isn’t relevant in practice because these different perspectives all manifest in the same mathematical assumptions for the realization of the observational process. Regardless of its interpretation we presume that this variation is sufficiently regular that, while an observational process cannot be quantified by any single observation, it can be quantified with a distribution of observations. More formally we model the observational process with a probability distribution over the observation space, with realizations of the observation process modeled as exact samples from that distribution.

Probability distributions over the observation space are in general denoted data generating processes, and the probability distribution that exactly captures the observational process in a given application is denoted the true data generating process, \(\pi^{\dagger}\).

To distinguish possible outcomes of an observational process from explicitly realized observations I will decorate the latter with a tilde. In other words \(y\) denotes an arbitrary point in the observation space while \(\tilde{y}\) denotes a particular point.

If we have any uncertainty about the system that we’re analyzing then the true data generating process will be unknown. Inference is tasked with quantifying information about the true data generating process using observed data as its only expression. In particular, successful inferences advice decisions about how to interact with the latent phenomenon encompassed within the true data generating process.

Ignorance of the true data generating process also complicates the calibration of any decision making process informed by our inferences. Consider formalizing a decision making process as a map from the observation space to the action space, \[ \begin{alignat*}{6} a :\; &Y& &\rightarrow& \; &A& \\ &y& &\mapsto& &a(y)&. \end{alignat*} \] In other words, a decision making process consumes an observation and produces an explicit decision. Given knowledge of the latent system we could hypothetically construct a utility function for each possible action, \(U : A \rightarrow \mathbb{R}^{+}\), which would then quantify the benefit of the decision making process in light of a particular observation as \[ \begin{alignat*}{6} U \circ a :\; &Y& &\rightarrow& \; &\mathbb{R}^{+}& \\ &y& &\mapsto& &U(a(y))&. \end{alignat*} \]

If we knew the true data generating process then we could calibrate the decision making process by evaluating the outcome utilities corresponding to possible observations. More formally, we could push the true data generating process through this composite utility function to give a probability distribution over outcome utilities, \(\pi(U)\), and then use the shape of this utility distribution to determine the effectiveness of the decision making process.

For example, we could calibrate the decision making process based on its expected utility, \[ \overline{U} = \int \mathrm{d} y \, \pi^{\dagger} (y) \, U(a(y)), \]

Unfortunately this calibration is purely hypothetical because of our ignorance of the true data generating process. In any realistic application we will be able to calibrate a given decision making process only in the context of a statistical model.

1.3 The Observational Model

At this point we have presumed the existence of a true data generating process, but we are ignorant of its precise form in any given application. In order to find the appropriate data generating process we would ideally search through the space of all data generating processes, that is the collection, \(\mathcal{P}\), of all probability distributions over the observation space.

Unfortunately this big world of data generating processes is just too large and complex to be able to quantify in any practical application. Indeed the depths of this space are where pure mathematicians lurk, and that alone should be reason enough to stay away…

In any case, we have to limit our search to something more tractable in practical applications. That is we have to consider only a small world, \(\mathcal{S} \subset \mathcal{P}\), of data generating processes that is more manageable. This small world defines an observational model consisting of individual data generating process that each define a possible mathematical narrative of how the data could be generated.

As the term “small world” often brings with it connotations of theme park rides, I’m going to give this collection of data generating processes the alternative, slightly more sophisticated name of model configuration space. Each individual data generating process within the small world is similarly denoted a model configuration. In other words “small world”, “observational model”, and “model configuration space” can all be used interchangeably to denote the subset of data generating processes considered in a given analysis. I will use “observational model” when I want to emphasize that a given subset has been assumed for a given application.

In practice it helps to assign distinct, numerical labels to each data generating processes within an observational model. A parameterization of an observational model is the one-to-one mapping between the model configuration space and a numerical space \(\Theta\), such as the real numbers. In other words, a parameterization is the the assignment of a unique value, or parameter \(\theta \in \Theta\), to each model configuration, \(s \in \mathcal{S}\). Keep in mind that the parameterizations themselves are not unique; in general there will be many equivalent ways of labeling the model configurations.

Given a parameterization we can then specify the observational model with family of probability density functions over the observational space, \(\pi_{\mathcal{S}} (y ; s)\). In practice we index these densities a particular parameterization, \(\pi_{\mathcal{S}} (y ; \theta)\).

For example, let’s say that our observational space is the one-dimensional real numbers, \(Y = \mathbb{R}\), and our observational model consists of all distributions that admit Gaussian probability density functions over the observational space. We can parameterize this model configuration space with the two-dimensional real numbers, \(\Theta = \mathbb{R}^{2}\) consisting of location parameter, \(\mu\), that can take all values and a scale parameter, \(\sigma\), that can take only positive values. In this case our model configuration space is specified by \[ \pi_{\mathcal{S}} (y ; \mu, \sigma) = \frac{1}{ \sqrt{2 \pi \sigma^{2}} } \exp \left( - \frac{1}{2} \left( \frac{y - \mu}{\sigma} \right)^{2} \right). \] We can equivalently, however, parameterize the model configuration space with a location parameter, \(\mu\), and an unconstrained scale parameter, \(\lambda = \log (\sigma)\). In this case our model configuration space is specified by \[ \pi_{\mathcal{S}} (y ; \mu, \lambda) = \frac{1}{ \sqrt{2 \pi} } e^{-\lambda} \exp \left( - \frac{1}{2} \left( y - \mu \right)^{2} e^{-2 \, \lambda} \right). \] Under \(N\) repeated observations our observational space would become \(Y = \mathbb{R}^{N}\) with the corresponding probability density functions, \[ \pi_{\mathcal{S}} (y_{1}, \ldots, y_{N} ; \mu, \sigma) = \prod_{n = 1}^{N} \pi_{\mathcal{S}} (y_{n} ; \mu, \sigma). \]

Once we limit out consideration to a given observational model, our inferential goal becomes the quantification of those model configurations that are somehow similar to the true data generating process, and hence useful for informing our ultimate decisions. If the observational model contains the true data generating process then basing our inferences on the observational model incurs no loss; an exhaustive search through the observational model will always be able to find the true data generating process. We are not always, however, so fortunate. What happens when the true data generating process lies outside the scope of the observational model?

Indeed in practice this circumstance is the rule rather than the exception. In applications observational models are limited by the constraints of mathematical and computational feasibility, and will not completely capture the, frankly ridiculous, complexity of the natural world.

The inferences generated from an observational model that doesn’t contain the true data generating process may still be meaningful so long as the data generating processes it does contain capture the structure of the true data generating process relevant to our ultimate inferential goals. For example, an ecological model that neglects quantum mechanics and general relativity is too technically meager to capture the full reality of any observation of animal migration. Most ecological research, however, is sufficiently useful without having to consider what might happen if birds started migrating at the speed of light or quantum tunneling through geographical impediments. At the same time, research considering bird navigation guided by following physiological sensitivity to magnetic fields might need some consideration of quantum mechanics.

Ultimately inference within an observational model will at best approximate the true data generating process, a realization that resonates with the famous quote of George Box that “all models are wrong but some are useful” [9]. Although some use that quote to motivate carelessness model building, Box was arguing the opposite. Only with respect for the given application and careful work can we develop an observational model that contains model configurations that sufficiently well approximate the true data generating process, providing the foundation for useful inferences.

Principled model development strives for an observational process that captures the relevant structure of the phenomenon, environment, and probe that give rise to the true data generating process. We always have to be careful, to recognize the limitations of the even the most carefully designed observational model in any given analysis.

1.3.1 The Generative Structure of an Observational Model

A particularly effective way to build useful observational models is to design each model configuration to emulate the generative structure of the observational process, from the latent phenomenon through the surrounding environment to the final observational probe. The parameters of a model following the generative structure of the observational process typically decompose into parameters for each layer, \[ \theta = \{ \theta_{\text{phenom}}, \theta_{\text{envir}}, \theta_{\text{probe}} \}. \]

Keep in mind that the separation between environmental and probe parameters, need not always be so clean, especially for cruder approximations to the true data generating process. For example is the influence of the person collecting the data better classified as a part of the environment or a part of the probe? Taken together the non-phenomenological parameters are sometimes denoted systematic parameters.

In the Gaussian example introduced above we might consider the location parameter, \(\mu\), as a phenomenological parameter that quantifies some baseline effect. From this perspective the scale parameter, \(\sigma\), would capture more systematic structure relating to either the surrounding environment or the particulars of the experimental probe. Another example where this decomposition is natural is naive curve fitting, where the curve can be interpreted as implicitly modeling the behavior of the latent phenomenon while the residuals around that curve model the rest of the observational process.

In addition to facilitating the design of the observational model for a single probe, this generative decomposition also motivates principled observational models for multiple probes. For example two probes might interrogate the same phenomenon without experiencing exactly the same environment and probe circumstances. Consequently we might specify the observational model as \[ \begin{align*} \pi_{\mathcal{S}} (y ; \theta) =& \;\;\; \pi_{\mathcal{S}} (y_{1} ; \theta_{\text{phenom}}, \theta_{\text{envir}}, \theta_{\text{envir}, 1}, \theta_{\text{probe}}, \theta_{\text{probe}, 1}) \\ & \cdot \pi_{\mathcal{S}} (y_{2} ; \theta_{\text{phenom}}, \theta_{\text{envir}}, \theta_{\text{envir}, 2}, \theta_{\text{probe}}, \theta_{\text{probe}, 2}), \end{align*} \] where the subscripted parameters, such as \(\theta_{\text{envir}, 1}\) are unique to a specific context and the non-subscripted parameters, such as \(\theta_{\text{envir}}\), are shared across contexts. The better we can model the systematic structure of the observational process the better we can build analyses that study the same phenomenon across different experiments, or develop hypotheses about the outcomes of new experiments targeting the same phenomenon.

Consider, for example, a latent phenomenon that influences a binary response in the environment. We might model the phenomenological behavior with a probability, \(p\), of the binary response taking the value \(y = 1\) instead of \(y = 0\). The rest of the observational model depends on the particular structure of the observational probe.

If we probe the system once and record the binary outcome then we might utilize an observational model consisting of Bernoulli distributions parameterized by the phenomenological probability, \[ \pi_{\mathcal{S}} (y ; p) = \text{Ber} (y \mid p). \] Presuming that the system is memoryless, repeated probes might be modeled as a product of independent Bernoulli distributions that are all parameterized by the same phenomenological probability, \[ \pi_{\mathcal{S}} (y_{1}, \ldots, y_{N} ; p) = \prod_{n = 1}^{N} \text{Ber} (y_{n} \mid p). \] Alternatively we might probe the system \(N\) times but only record the total number of \(y = 1\) responses, \(k\). In this case we could model the observational process with binomial distributions parameterized with the phenomenological probability, \[ \pi_{\mathcal{S}} (k; p) = \text{Bin} (k \mid p). \] Finally, probing until \(k\) unity responses are observed could be modeled with negative Binomial distributions parameterized by the phenomenological probability, \[ \pi_{\mathcal{S}} (k; p) = \text{NegBin} (k \mid p). \]

A simple binary response quantified by a phenomenological probability, however, is a relatively crude approximation to any realistic observational process. The more deeply we resolve the observational process the richer our observational model will have to become.

That binary response, for example, might correspond to the capture of a particular species in an ecological experiment. In that case the probability of capture might not be a single, monolithic parameter but rather the outcome of the population dynamics between the species of interest, the species on which it preys, and the species which predate it.

It could also correspond to the alarm of a detector that has recorded enough incident radiation to pass a certain threshold. Here the latent probability would be a consequence of the possible sources of radiation, including radiation from particular sources of interest as well as the irrelevant background sources that permeate the environment. Depending on the precision required of the desired inferences we might also have to model the detector response itself with additional parameters, including those quantifying the calibration of the incident radiation flux and the behavior of the electronics which are ultimately integrated to trigger the detection.

Each deeper look into the observational process discerns more detail and a richer generative structure. At the same time, in a given application we might need only a crude approximation that blends together various aspects of the generative structure. Perhaps the biggest challenge in model building is determining exactly how sophisticated an observational model needs to be in order to achieve the desired inferential goals.

1.3.2 Limitations of Observational Models

Regardless of how sophisticated our observational model might be, it will always be an approximation to the complexity of the true data generating process. We have to be ever mindful of the limitations of the observational model in the context of any particular analysis.

For example, the observational model might approximate the true data generating process only within a certain regime of behaviors. At the same time the approximations might accurately capture only some behaviors but not others. A given observational model can express the large scale structure of the latent system but make no attempt to incorporate the structure at smaller scales. It might capture statistical relationships between phenomenological parameters but fail to accurately convey the physical dynamics or explicit causal structure of the phenomenon and its interaction with the environment.

Consequently we have to be extremely careful to employ model-based inferences only within the regimes where the approximations are reasonable and sufficient for the decisions that we want to inform. In other words, we have to develop a model that is sophisticated enough for the given application.

Another subtlety with the approximate nature of the observational model is that a given observational model might be a reasonable approximation for many true data generating processes. This is particularly common for observational models that focus on capturing the emergent behaviors of a complex observational process. At sufficiently coarse scales much of the latent structure washes out and only the most coherent of the latent behaviors survive. All true data generating process with similar emergent behavior can then be equally well approximated by the same observational model.

Linear regression, for example, can be interpreted as a model where phenomenological behavior manifests as a deterministic linear relationship between observed variates and covariates which is then convolved through Gaussian fluctuations. This linear relationship, however, can be considered as a first-order approximation to any latent phenomenological behavior. At the same time Gaussian variation well approximates a wide range of environmental and probe behaviors, especially within the context of the first-order behavior. Consequently any true data generating process with the same first-order phenomenological behavior, and emergent environmental and probe behaviors, will be reasonably approximated by the same linear regression model.

Emergent models cannot discriminate between the true data generating processes they approximate and they cannot express behaviors outside of the scope of the approximation. The meaningful interpretation of a model-based statistical analysis requires not just an explicit observational model but also the scope of the approximations it presumes. In other words, we need to at least conceptually understand the context of the model relative to the true data generating process.

1.3.3 Model-Based Calibration

While we cannot perfectly calibrate a decision-making process without knowing the true data generating process, we can define a proxy calibration within the context of a given observational model.

Firstly we assume that we can define utilities for all available actions within the context of each model configuration, \[ \begin{alignat*}{6} U:\; &A \times \Theta& &\rightarrow& \; &\mathbb{R}^{+}& \\ &(a, \theta)& &\mapsto& &U(a, \theta)&. \end{alignat*} \] Given a particular model configuration we can then compute the utility of an observation-informed decision-making process with the composition \[ \begin{alignat*}{6} U \circ a :\; &Y \times \Theta& &\rightarrow& \; &\mathbb{R}^{+}& \\ &(y, \theta)& &\mapsto& &U(a(y), \theta)&. \end{alignat*} \]

We can push each of the data generating processes in our observational model through the composite utility function to give a full distribution of possible utilities for each model configuration, \(\pi_{\mathcal{S}}(U; \theta)\).

The properties of this distribution then provide the foundation to define various calibrations. For example, for each model configuration we can quantify the expected utilities, \[ \overline{U}_{\mathcal{S}}(\theta) = \int \mathrm{d} y \, \pi_{\mathcal{S}}(y ; \theta) \, U(a(y), \theta) ]. \]

How we might exploit these calibrations in an analysis will then depend on how we handle their dependence on the unknown model configuration. No matter how we end up employing these model-based utilities we have to be vigilant in recognizing they are only as good as the observational model itself.

1.4 Model-Based Inferences

Once an observational model has been established, and its limitations acknowledged, we can consider model-based inferences that identify which model configurations are most consistent with the true data generating process. Unfortunately without knowing the true data generating process these inferences can be informed only by the information inherent in observed data and our own domain expertise. Within the context of an observational model, inference becomes the identification of how consistent model configurations are with the observed data and any domain expertise we introduce.

Consequently the observational model both plays two critical roles. Firstly it defines a context for constructing inferences from observations.

In particular, model-based inferences take the form of a quantification of which model configurations are consistent with the observed data.

Secondly those model-based inferences then inform decisions about how to interact with the latent phenomenon, bridging the gap between the observed data and the latent system.