×
you are viewing a single comment's thread.

view the rest of the comments →

[–]geoffhintonGoogle Brain[S] 26 points27 points  (2 children)

  1. Here are some of my beliefs about the brain that have made a big difference to the kinds of machine learning I have done:

The cortex is pretty much the same all over and if parts are lost early, other parts can take on the functions they would have implemented. This suggests its really worth taking a bet on there being a general purpose learning procedure.

The brain is clearly using distributed representations.

The brain does complex tasks like object recognition and sentence understanding with surprisingly little serial depth to the computation. So artificial neural nets should do the same.

The brain has about 1014 synapses and we only live for about 109 seconds. So we have a lot more parameters than data. This motivates the idea that we must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 105 dimensions of constraint per second.

Roughly speaking, spikes are noisy samples from an underlying Poisson rate. Over the short time periods involved in perception, this is an incredibly noisy code. One of the motivations for the idea of dropout was that very noisy spikes are a good way to get a very strong regularizer that can help the brain deal with the fact that it has thousands of times more parameters than experiences.

Over a short time period, a neuron really is a binary all-or-none device (so far as other neurons are concerned). This was one of the motivations behind Boltzmann machines. Another was the paper by Crick and Mitchison suggesting that we do unlearning during sleep. There now seems to be quite a lot of evidence for this.

[–]saguppa 1 point2 points  (1 child)

I'm not sure if I really understand this. What would happen if we lived for say 1014 seconds, would we be able to see the world the way we do now, for the entirety of our lifetime? Would the brain begin to overfit the data, so to speak? For example, suppose I grew up with a cat, would I not be able to recognize other cats as cats when I'm really old?

[–]visarga 0 points1 point  (0 children)

The impact of your old experiences is gradually reduced making space for new experiences. The question is, after how much time the original experiences become background noise? Maybe we could live for 3 million years but every 100 years or so, we would be tabula rasa. Maybe even sooner, if we were to judge how some adults seem to have forgotten all about being a child by the time they reach the second part of their life.