# Writing

Preprints of my work are posted on the arXiv as much as possible. Highlights include a long but comprehensive introduction to statistical computing and Hamiltonian Monte Carlo targeted at applied researches and a more theoretical treatment of the geometric foundations of Hamiltonian Monte Carlo.

Recently I have been taking advantage of the notebook environments knitr and Jupyter to develop case studies demonstrating important concepts in probabilistic modeling with Stan. The ultimate goal is a self-contained treatment demonstrating how to build, evaluate, and utilize probabilistic models that capture domain expertise in applied analyses.

Presuming a familiarity with calculus and linear algebra I recommend starting with an introduction to the important concepts in probability theory and concepts in conditional_probability theory followed by an introduction to modeling and inference and a discussion of Bayesian computation in the first two sections of my introduction to Hamiltonian Monte Carlo. With this foundation one can move onto my recommended use of RStan and PyStan, and then a discretionary concentration on divergences in Hamiltonian Monte Carlo. Finally one can move onto an introduction to principled model building in Stan.

While relatively mature these papers and case studies are still very much dynamic and improving as I receive feedback from readers, so please don’t hesitate to send comments through email or pull requests on the case study GitHub repositories linked below.

## Probabilistic Modeling and Statistical Inference

In this case study we’ll review the foundations of statistical models and statistical inference that advise principled decision making. We’ll place a particular emphasis on Bayesian inference, which utilizes probability theory and statistical modeling to encapsulate information from observed data and our own domain expertise.

View
(HTML)

betanalpha/knitr_case_studies/modeling_and_inference
(GitHub)

Dependences: `R, knitr`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2019). Probabilistic Modeling and Statistical Inference. Retrieved from
https://betanalpha.github.io/assets/case_studies/modeling_and_inference.html*.

## Underdetermined Linear Regression

A linear regression is underdetermined when there are fewer observations than parameters. In this case the likelihood function does not concentrate on a compact neighborhood but rather an underdetermined hyperplane of degenerate model configurations. The corresponding posterior density has a surprisingly interesting geometry, even with weakly-informative prior densities that ensure a well-defined fit. In this short note I walk through the nature of this geometry and why underdetermined regressions are so hard to fit.

View
(HTML)

betanalpha/knitr_case_studies/underdetermined_linear_regression
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Underdetermined Linear Regression. Retrieved from
https://betanalpha.github.io/assets/case_studies/underdetermined_linear_regression.html*.

## Towards A Principled Bayesian Workflow (PyStan)

Given the posterior distribution derived from a probabilistic model and a particular observation, Bayesian inference is straightforward to implement: inferences, and any decisions based upon them, follow immediately in the form of posterior expectations. Building such a probabilistic model that is satisfactory in a given application, however, is a far more open-ended challenge. In order to ensure robust analyses we need a principled workflow that guides the development of a probabilistic model that is consistent with both our domain expertise and any observed data while also being amenable to accurate computation. In this case study I introduce a principled workflow for building and evaluating probabilistic models in Bayesian inference.

View
(HTML)

betanalpha/jupyter_case_studies/principled_bayesian_workflow
(GitHub)

Dependences: `Python, Jupyter, PyStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Towards A Principled Bayesian Workflow (PyStan). Retrieved from
https://github.com/betanalpha/jupyter_case_studies/blob/master/principled_bayesian_workflow/principled_bayesian_workflow.ipynb*.

## Towards A Principled Bayesian Workflow (RStan)

Given the posterior distribution derived from a probabilistic model and a particular observation, Bayesian inference is straightforward to implement: inferences, and any decisions based upon them, follow immediately in the form of posterior expectations. Building such a probabilistic model that is satisfactory in a given application, however, is a far more open-ended challenge. In order to ensure robust analyses we need a principled workflow that guides the development of a probabilistic model that is consistent with both our domain expertise and any observed data while also being amenable to accurate computation. In this case study I introduce a principled workflow for building and evaluating probabilistic models in Bayesian inference.

View
(HTML)

betanalpha/knitr_case_studies/principled_bayesian_workflow
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Towards A Principled Bayesian Workflow (RStan). Retrieved from
https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html*.

## Conditional Probability Theory (For Scientists and Engineers)

This case study will introduce a conceptual understanding of conditional probability theory and its applications. We’ll begin with a discussion of marginal probability distributions before introducing conditional probability distributions as their complement. Then we’ll examine how different conditional probability distributions can be related to each other through Bayes’ Theorem before considering how all of these objects manifest in probability mass function and probability density function representations. Finally we’ll review some of the important practical applications of the theory.

View
(HTML)

betanalpha/knitr_case_studies/conditional_probability_theory
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Conditional Probability Theory (For Scientists and Engineers). Retrieved from
https://betanalpha.github.io/assets/case_studies/conditional_probability_theory.html*.

## Probability Theory (For Scientists and Engineers)

Formal probability theory is a rich and complex field of mathematics with a reputation for being confusing if not outright impenetrable. Much of that intimidation, however, is due not to the abstract mathematics but rather how they are employed in practice. In this case study I attempt to untangle this pedagogical knot to illuminate the basic concepts and manipulations of probability theory and how they can be implemented in practice. Our ultimate goal is to demystify what we can calculate in probability theory and how we can perform those calculations in practice.

View
(HTML)

betanalpha/knitr_case_studies/probability_theory
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Probability Theory (For Scientists and Engineers). Retrieved from
https://betanalpha.github.io/assets/case_studies/probability_theory.html*.

## Bayes Sparse Regression

In this case study I’ll review how sparsity arises in frequentist and Bayesian analyses and discuss the often subtle challenges in implementing sparsity in practical Bayesian analyses.

View
(HTML)

betanalpha/knitr_case_studies/bayes_sparse_regression
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Bayes Sparse Regression. Retrieved from
https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html*.

## Fitting the Cauchy

In this case study I review various ways of implementing the Cauchy distribution, from the nominal implementation to alternative implementations aimed at ameliorating these difficulties, and demonstrate their relative performance.

View
(HTML)

betanalpha/knitr_case_studies/fitting_the_cauchy
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2018). Fitting the Cauchy. Retrieved from
https://betanalpha.github.io/assets/case_studies/fitting_the_cauchy.html*.

## Robust Gaussian Processes in Stan, Part 3

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View
(HTML)

betanalpha/knitr_case_studies/gaussian_processes/gp_part3
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 3. Retrieved from
https://betanalpha.github.io/assets/case_studies/gp_part3/part3.html*.

## Robust Gaussian Processes in Stan, Part 2

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View
(HTML)

betanalpha/knitr_case_studies/gaussian_processes/gp_part2
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 2. Retrieved from
https://betanalpha.github.io/assets/case_studies/gp_part2/part2.html*.

## Robust Gaussian Processes in Stan, Part 1

This series of case studies introduces the very basics of Gaussian processes, how to implement them in Stan, and how they can be robustly incorporated into Bayesian models to avoid subtle but pathological behavior.

View
(HTML)

betanalpha/knitr_case_studies/gaussian_processes/gp_part1
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Robust Gaussian Processes in Stan, Part 1. Retrieved from
https://betanalpha.github.io/assets/case_studies/gp_part1/part1.html*.

## The QR Decomposition for Regression Models

This case study reviews the QR decomposition, a technique for decorrelating covariates and, consequently, the resulting posterior distribution in regression models.

View
(HTML)

betanalpha/knitr_case_studies/qr_regression
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). The QR Decomposition for Regression Models. Retrieved from
https://betanalpha.github.io/assets/case_studies/qr_regression.html*.

## Robust PyStan Workflow

This case study demonstrates a proper PyStan workflow that ensures robust inferences with the default dynamic Hamiltonian Monte Carlo algorithm.

View
(HTML)

betanalpha/jupyter_case_studies/pystan_workflow
(GitHub)

Dependences: `Python, Jupyter, PyStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Robust PyStan Workflow. Retrieved from
https://betanalpha.github.io/assets/case_studies/pystan_workflow.html*.

## Robust RStan Workflow

This case study demonstrates a proper RStan workflow that ensures robust inferences with the default dynamic Hamiltonian Monte Carlo algorithm.

View
(HTML)

betanalpha/knitr_case_studies/rstan_workflow
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Robust RStan Workflow. Retrieved from
https://betanalpha.github.io/assets/case_studies/rstan_workflow.html*.

## Diagnosing Biased Inference with Divergences

This case study discusses the subtleties of accurate Markov chain Monte Carlo estimation and how divergences can be used to identify biased estimation in practice.

View
(HTML)

betanalpha/knitr_case_studies/divergences_and_bias
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Diagnosing Biased Inference with Divergences. Retrieved from
https://betanalpha.github.io/assets/case_studies/divergences_and_bias.html*.

## Identifying Bayesian Mixture Models

This case study discusses the common pathologies of Bayesian mixture models as well as some strategies for identifying and overcoming them.

View
(HTML)

betanalpha/knitr_case_studies/identifying_mixture_models
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). Identifying Bayesian Mixture Models. Retrieved from
https://betanalpha.github.io/assets/case_studies/identifying_mixture_models.html*.

## How the Shape of a Weakly Informative Prior Affects Inferences

This case study reviews the basics of weakly-informative priors and how the choice of a specific shape of such a prior affects the resulting posterior distribution.

View
(HTML)

betanalpha/knitr_case_studies/weakly_informative_shapes
(GitHub)

Dependences: `R, knitr, RStan`

Code License: BSD (3 clause)
Text License: CC BY-NC 4.0
Cite As: *Betancourt, Michael (2017). How the Shape of a Weakly Informative Prior Affects Inferences. Retrieved from
https://betanalpha.github.io/assets/case_studies/weakly_informative_shapes.html*.