Title:Hamiltonian zigzag accelerates large-scale inference for conditional dependencies between complex biological traits

Abstract:Inferring dependencies between complex biological traits while accounting for evolutionary relationships between specimens is of great scientific interest yet remains infeasible when trait and specimen counts grow large. The phylogenetic multivariate probit model uses a latent variable framework to accommodate binary and continuous traits, but integrating many latent variables requires many computationally expensive draws from a high-dimensional truncated normal. The state-of-the-art approach, which combines the bouncy particle sampler (BPS) with dynamically programmed gradient evaluations, breaks down as the number of specimens grows and fails to reliably characterize conditional dependencies between traits. We develop an inference scheme that combines the recent Zigzag Hamiltonian Monte Carlo (Zigzag-HMC) with linear-time gradient evaluations and joint updates for highly correlated latent variables and correlation matrix elements. In an application exploring HIV-1 evolution from 535 viruses, the inference requires joint sampling from an 11,235-dimensional truncated normal and a 24-dimensional covariance matrix. Our method yields a 5-fold speedup compared to BPS and makes it possible to learn partial correlations between candidate viral mutations and virulence. Computational speedups now allow us to tackle larger problems: we study the evolution of influenza H1N1 glycosylations on around 900 viruses. For broader applicability, we extend the phylogenetic probit model to incorporate categorical traits, and demonstrate its use to study Aquilegia flower and pollinator co-evolution.