Bayesian nonparametric modelling

Dr. Dinos Perrakis

Description

Bayesian parametric models rely on specific assumptions, first, about the sampling distribution \(F(y\mid\theta)\) generating the observed data \(y\) and, second, about the prior distribution \(F(\theta)\) of \(\theta\). In many cases, however, it is desirable to relax parametric assumptions in order to allow greater modelling flexibility and robustness against misspecification of a parametric statistical model. In these cases, we may want to consider models where the class of distributions is so large that it can no longer be indexed by a finite dimensional parameter \(\theta\).

When the parameter lies in an infinite dimensional space, we have a Bayesian nonparametric model where the actual “parameter of interest” is a probability distribution, which we can denote by \(\Theta\). Working in such a setting within the Bayesian framework requires the definition of probability measures on distribution functions. Such random probability measures are known as stochastic processes (DP) and the most widely used stochastic process in Bayesian nonparametrics is the Dirichlet process whose realisations are probability distributions. In this case, the Bayesian model is as follows \[\begin{align} y_1, \ldots y_n \mid \Theta & \stackrel{\mathrm{i.i.d.}}{\sim} \Theta \nonumber \\ \Theta & \sim G, \label{eq4} \end{align}\] where \(G\) is defined as DP prior, a construction generally considered to be the cornerstone of modern Bayesian nonparametrics as the prior is conjugate to completely unknown distributions for i.i.d. random variables (discrete or continuous). An example of density estimation for continuous data is shown below.

Density estimation: ten posterior realisations arising from a DP prior (dotted lines in different colours) and the expected posterior density (solid line) overlaid on a histogram of continuous data.
Density estimation: ten posterior realisations arising from a DP prior (dotted lines in different colours) and the expected posterior density (solid line) overlaid on a histogram of continuous data.

 

Bayesian nonparametric models find numerous applications related to density estimation, clustering and regression analysis. In this project we will start by understanding the construction of the basic Bayesian DP model (briefly described here), its various representations - for instance, as a stick-breaking random process and as a hierarchical model - and how one can sample from the posterior. We will also see more advanced DP models which can be viewed as infinite mixture models. Further directions may include exploring the use of other stochastic processes. Emphasis will also be placed on practical implementation using modern R packages.

Prerequisites

Corequisite

Some resources

Note: The last two books are advanced and heavy in theory and are not necessarily needed for this project.

Feel free to email at if you have questions.