Deep Gaussian Processes

Combining the nonparametric flexibility of deep neural networks with the uncertainty quantification of Bayesian Gaussian processes.

What is a Gaussian Process?

A Gaussian Process (GP) is a Bayesian, nonparametric model over functions. Instead of learning fixed weights, a GP places a probability distribution directly over the space of functions that could explain the data, defined by a mean function and a covariance (kernel) function.

f(x) ~ GP( m(x), k(x, x′) )

This gives GPs two properties neural networks don't have natively: well-calibrated predictive uncertainty, and strong performance on small datasets without overfitting.

Why "Deep"?

A single-layer GP is limited by the expressiveness of its kernel — it struggles to model highly non-stationary functions, hierarchical structure, or representation learning. A Deep Gaussian Process (DGP) stacks multiple layers of GPs, where the output of one GP layer becomes the (latent, uncertain) input to the next — just like layers in a neural network, but every layer is a full probabilistic function.

Layer 1

Input → Latent space

Raw inputs pass through a GP mapping to a learned latent representation, capturing low-level structure.

Layer 2..N

Latent → Latent

Each subsequent GP layer transforms the previous (uncertain) latent representation, building hierarchical, compositional structure.

Output

Latent → Prediction

A final GP layer maps the last latent representation to the output, producing a full predictive distribution.

Key idea

Uncertainty propagates

Because every layer is probabilistic, uncertainty from early layers correctly propagates through to the final prediction.

Why It Matters

Property	Deep Neural Network	Single-layer GP	Deep GP
Representation learning	Yes	No	Yes
Calibrated uncertainty	No (needs add-ons)	Yes	Yes
Performs well on small data	Often no	Yes	Yes
Models non-stationary functions	Yes	Limited	Yes
Training cost	Low–moderate	Moderate	High

Training Challenges

Intractable inference: exact Bayesian inference through stacked, nonlinear GP layers has no closed form.
Variational approximation: practical DGPs (e.g., Damianou & Lawrence's original formulation, and Doubly Stochastic Variational Inference) use inducing points and variational distributions to make training scalable.
Vanishing/exploding signal: as with deep nets, depth can degrade gradient signal and requires careful initialization or identity-mean mappings between layers.
Compute cost: scales less favorably than standard deep learning, limiting DGPs mostly to small- and medium-scale data regimes today.

Typical Applications

Scientific and engineering domains with scarce, expensive-to-collect data (robotics, materials science, healthcare)
Bayesian optimization and active learning, where uncertainty estimates guide what to sample next
Time-series forecasting where confidence intervals matter as much as point predictions
Safety-critical systems that need to know when they don't know

Takeaway

Deep Gaussian Processes sit at the intersection of deep learning and Bayesian nonparametrics: they inherit the hierarchical, flexible feature learning of deep networks while retaining principled, propagated uncertainty at every layer — at the cost of heavier and more complex inference.