KIP publications

year 2016
author(s) Luziwei Leng*, Mihai A. Petrovici*, Roman Martel, Ilja Bytschok, Oliver Breitwieser, Johannes Bill, Johannes Schemmel, Karlheinz Meier
title Spiking neural networks as superior generative and discriminative models
KIP-Nummer HD-KIP 16-13
KIP-Gruppe(n) F9
document type Paper
Keywords spiking neural networks, deep learning, short-term plasticity, discriminative and generative models
source Cosyne Abstracts 2016, Salt Lake City USA
Abstract (en)

An increasing number of experiments suggest that the brain performs stochastic inference when dealing with incomplete and noisy sensory information. This, in turn, has led to the development of various theoretical models that attempt to explain how this could be achieved with spiking neural networks. One candidate theory interprets spiking activity as sampling from distributions over binary random variables (Buesing et al., 2011) and has been shown to be compatible with the ensemble dynamics of noise-driven LIF neurons in the high-conductance state (Petrovici et al., 2013, 2015; Probst et al., 2015). Based on this theory, we constructed hierarchical LIF networks that sample from restricted Boltzmann distributions and compared their performance with conventional restricted Boltzmann machines (RBMs) on a commonly used dataset (MNIST). An important result is that LIF networks can achieve similar classification rates (95.1 % with 1994 neurons) as their machine-learning counterparts of equal size (95.2 %). In classical RBMs however, statistics are typically gathered by Gibbs sampling. This algorithm has a distinct disadvantage when dealing with high-dimensional multimodal distributions, where it often gets trappedin a local minimum due to deep troughs in the energy landscape that appear during training. It is for this reason that conventional RBMs that may perform very well as discriminative models are, at the same time, rather poor generative models of the learned data. While various methods exist that alleviate this problem (such as AST, see Salakhutdinov, 2010) they usually come at a highly increased computational cost. In the second part of our study, we show how short-term plasticity enables LIF networks to travel efficiently through the energy landscape and
thereby attain a generative performance that far surpasses the one achievable by conventional Gibbs sampling. This distinct advantage of biological neural networks allows them to simultaneously become good generative and discriminative models of learned data.

  author   = {Luziwei Leng*, Mihai A. Petrovici*, Roman Martel, Ilja Bytschok, Oliver Breitwieser, Johannes Bill, Johannes Schemmel, Karlheinz Meier},
  title    = {Spiking neural networks as superior generative and discriminative models},
  booktitle = {},
  year     = {2016},
  volume   = {},
  pages    = {},
  month    = {February}
Datei abstract
Datei poster
KIP - Bibliothek
Im Neuenheimer Feld 227
Raum 3.402
69120 Heidelberg