Writing

Preprints of my work are posted on the arXiv as much as possible. Highlights include a long but comprehensive introduction to statistical computing and Hamiltonian Monte Carlo targeted at applied researches and a more theoretical treatment of the geometric foundations of Hamiltonian Monte Carlo.

Recently I have been taking advantage of the notebook environments knitr and Jupyter to develop case studies demonstrating important concepts in probabilistic modeling with Stan. The ultimate goal is a self-contained treatment demonstrating how to build, evaluate, and utilize probabilistic models that capture domain expertise in applied analyses.

Presuming a familiarity with calculus and linear algebra I recommend starting with an introduction to the important concepts in probability theory and concepts in conditional_probability theory followed by a review of model-based inference and decision making and a discussion of Bayesian computation in the first two sections of my introduction to Hamiltonian Monte Carlo. With this foundation one can move onto my recommended use of RStan and PyStan, and then a discretionary concentration on divergences in Hamiltonian Monte Carlo. Finally one can move onto an introduction to principled model building in Stan.

While relatively mature these papers and case studies are still very much dynamic and improving as I receive feedback from readers, so please don’t hesitate to send comments through email or pull requests on the case study GitHub repositories linked below.

Towards A Principled Bayesian Workflow (PyStan)

Given the posterior distribution derived from a probabilistic model and a particular observation, Bayesian inference is straightforward to implement: inferences, and any decisions based upon them, follow immediately in the form of posterior expectations. Building such a probabilistic model that is satisfactory in a given application, however, is a far more open-ended challenge. In order to ensure robust analyses we need a principled workflow that guides the development of a probabilistic model that is consistent with both our domain expertise and any observed data while also being amenable to accurate computation. In this case study I introduce a principled workflow for building and evaluating probabilistic models in Bayesian inference.

View (HTML)
betanalpha/jupyter_case_studies/principled_bayesian_workflow (GitHub)
Dependences: Python, Jupyter, PyStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Towards A Principled Bayesian Workflow (PyStan). Retrieved from https://github.com/betanalpha/jupyter_case_studies/blob/master/principled_bayesian_workflow/principled_bayesian_workflow.ipynb.

Towards A Principled Bayesian Workflow (RStan)

Given the posterior distribution derived from a probabilistic model and a particular observation, Bayesian inference is straightforward to implement: inferences, and any decisions based upon them, follow immediately in the form of posterior expectations. Building such a probabilistic model that is satisfactory in a given application, however, is a far more open-ended challenge. In order to ensure robust analyses we need a principled workflow that guides the development of a probabilistic model that is consistent with both our domain expertise and any observed data while also being amenable to accurate computation. In this case study I introduce a principled workflow for building and evaluating probabilistic models in Bayesian inference.

View (HTML)
betanalpha/knitr_case_studies/principled_bayesian_workflow (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Towards A Principled Bayesian Workflow (RStan). Retrieved from https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html.

Conditional Probability Theory (For Scientists and Engineers)

This case study will introduce a conceptual understanding of conditional probability theory and its applications. We’ll begin with a discussion of marginal probability distributions before introducing conditional probability distributions as their complement. Then we’ll examine how different conditional probability distributions can be related to each other through Bayes’ Theorem before considering how all of these objects manifest in probability mass function and probability density function representations. Finally we’ll review some of the important practical applications of the theory.

View (HTML)
betanalpha/knitr_case_studies/conditional_probability_theory (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Conditional Probability Theory (For Scientists and Engineers). Retrieved from https://betanalpha.github.io/assets/case_studies/conditional_probability_theory.html.

Probability Theory (For Scientists and Engineers)

Formal probability theory is a rich and complex field of mathematics with a reputation for being confusing if not outright impenetrable. Much of that intimidation, however, is due not to the abstract mathematics but rather how they are employed in practice. In this case study I attempt to untangle this pedagogical knot to illuminate the basic concepts and manipulations of probability theory and how they can be implemented in practice. Our ultimate goal is to demystify what we can calculate in probability theory and how we can perform those calculations in practice.

View (HTML)
betanalpha/knitr_case_studies/probability_theory (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Probability Theory (For Scientists and Engineers). Retrieved from https://betanalpha.github.io/assets/case_studies/probability_theory.html.

Bayes Sparse Regression

In this case study I’ll review how sparsity arises in frequentist and Bayesian analyses and discuss the often subtle challenges in implementing sparsity in practical Bayesian analyses.

View (HTML)
betanalpha/knitr_case_studies/bayes_sparse_regression (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Bayes Sparse Regression. Retrieved from https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html.

Fitting the Cauchy

In this case study I review various ways of implementing the Cauchy distribution, from the nominal implementation to alternative implementations aimed at ameliorating these difficulties, and demonstrate their relative performance.

View (HTML)
betanalpha/knitr_case_studies/fitting_the_cauchy (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2018). Fitting the Cauchy. Retrieved from https://betanalpha.github.io/assets/case_studies/fitting_the_cauchy.html.

Robust Gaussian Processes in Stan, Part 3

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View (HTML)
betanalpha/knitr_case_studies/gaussian_processes/gp_part3 (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 3. Retrieved from https://betanalpha.github.io/assets/case_studies/gp_part3/part3.html.

Robust Gaussian Processes in Stan, Part 2

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View (HTML)
betanalpha/knitr_case_studies/gaussian_processes/gp_part2 (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 2. Retrieved from https://betanalpha.github.io/assets/case_studies/gp_part2/part2.html.

Robust Gaussian Processes in Stan, Part 1

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View (HTML)
betanalpha/knitr_case_studies/gaussian_processes/gp_part1 (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 1. Retrieved from https://betanalpha.github.io/assets/case_studies/gp_part1/part1.html.

The QR Decomposition for Regression Models

This case study reviews the QR decomposition, a technique for decorrelating covariates and, consequently, the resulting posterior distribution in regression models.

View (HTML)
betanalpha/knitr_case_studies/qr_regression (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). The QR Decomposition for Regression Models. Retrieved from https://betanalpha.github.io/assets/case_studies/qr_regression.html.

Robust PyStan Workflow

This case study demonstrates a proper PyStan workflow that ensures robust inferences with the default dynamic Hamiltonian Monte Carlo algorithm.

View (HTML)
betanalpha/jupyter_case_studies/pystan_workflow (GitHub)
Dependences: Python, Jupyter, PyStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Robust PyStan Workflow. Retrieved from https://betanalpha.github.io/assets/case_studies/pystan_workflow.html.

Robust RStan Workflow

This case study demonstrates a proper RStan workflow that ensures robust inferences with the default dynamic Hamiltonian Monte Carlo algorithm.

View (HTML)
betanalpha/knitr_case_studies/rstan_workflow (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Robust RStan Workflow. Retrieved from https://betanalpha.github.io/assets/case_studies/rstan_workflow.html.

Diagnosing Biased Inference with Divergences

This case study discusses the subtleties of accurate Markov chain Monte Carlo estimation and how divergences can be used to identify biased estimation in practice.

View (HTML)
betanalpha/knitr_case_studies/divergences_and_bias (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Diagnosing Biased Inference with Divergences. Retrieved from https://betanalpha.github.io/assets/case_studies/divergences_and_bias.html.

Identifying Bayesian Mixture Models

This case study discusses the common pathologies of Bayesian mixture models as well as some strategies for identifying and overcoming them.

View (HTML)
betanalpha/knitr_case_studies/identifying_mixture_models (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). Identifying Bayesian Mixture Models. Retrieved from https://betanalpha.github.io/assets/case_studies/identifying_mixture_models.html.

How the Shape of a Weakly Informative Prior Affects Inferences

This case study reviews the basics of weakly-informative priors and how the choice of a specific shape of such a prior affects the resulting posterior distribution.

View (HTML)
betanalpha/knitr_case_studies/weakly_informative_shapes (GitHub)
Dependences: R, knitr, RStan
Code License: BSD (3 clause) Text License: CC BY-NC 4.0 Cite As: Betancourt, Michael (2017). How the Shape of a Weakly Informative Prior Affects Inferences. Retrieved from https://betanalpha.github.io/assets/case_studies/weakly_informative_shapes.html.