A couple weeks ago I spent an hour talking over video chat with Daniel Cantu, a UCLA neuroscience postdoc who I hired on Wyzant.com to spend an hour answering a variety of questions about neuroscience I had. (Thanks Daniel for reviewing this blog post for me! If you have neuroscience questions for him you can hire him here.)
The most interesting thing I learned is that I had quite substantially misunderstood the connection between convolutional neural nets and the human visual system. People claim that these are somewhat bio-inspired, and that if you look at early layers of the visual cortex you’ll find that it operates kind of like the early layers of a CNN, and so on.
The claim that the visual system works like a CNN didn’t quite make sense to me though. According to my extremely rough understanding, biological neurons operate kind of like the artificial neurons in a fully connected neural net layer–they have some input connections and a nonlinearity and some output connections, and they have some kind of mechanism for Hebbian learning or backpropagation or something. But that story doesn’t seem to have a mechanism for how neurons do weight tying, which to me is the key feature of CNNs.
Daniel claimed that indeed human brains don’t have weight tying, and we achieve the efficiency gains over dense neural nets by two other mechanisms instead:
Firstly, the early layers of the visual cortex are set up to recognize particular low-level visual features like edges and motion, but this is largely genetically encoded rather than learned with weight-sharing. One way that we know this is that mice develop a lot of these features before their eyes open. These low-level features can be reinforced by positive signals from later layers, like other neurons, but these updates aren’t done with weight-tying. So the weight-sharing and learning here is done at the genetic level.
Secondly, he thinks that we get around the need for weight-sharing at later levels by not trying to be able to recognize complicated details with different neurons. Our vision is way more detailed in the center of our field of view than around the edges, and if we need to look at something closely we move our eyes over it. He claims that this gets around the need to have weight tying, because we only need to be able to recognize images centered in one place.
I was pretty skeptical of this claim at first. I pointed out that I can in fact read letters that are a variety of distances from the center of my visual field; his guess is that I learned to read all of these separately. I’m also kind of confused by how this story fits in with the fact that humans seem to relatively quickly learn to adapt to inversion goggled. I would love to check what some other people who know neuroscience think about this.
I found this pretty mindblowing. I’ve heard people use CNNs as an example of how understanding brains helped us figure out how to do ML stuff better; people use this as an argument for why future AI advances will need to be based on improved neuroscience. This argument seems basically completely wrong if the story I presented here is correct.
See comments on Facebook here