Title:Deep Boltzmann Machines with Fine Scalability

Abstract:We present a layered Boltzmann machine (BM) whose generative performance scales with the number of layers. Application of deep BMs (DBMs) is limited due to its poor scalability where deep stacking of layers does not largely improve the performance. It is widely believed that DBMs have huge representation power, and its poor empirical scalability is mainly due to inefficiency of optimization algorithms. In this paper, we theoretically show that that the representation power of DBMs is actually rather limited, and the inefficiency of the model can result in the poor scalability. Based on these observations, we propose an alternative BM architecture, which we dub soft-deep BMs (sDBMs). We theoretically show that sDBMs possess much greater representation power than DBMs. Experiments demonstrate that we are able to train sDBMs with up to 6 layers without pretraining, and sDBMs nicely compare state-of-the-art models on binarized MNIST and Caltech-101 silhouettes.