6.803 | Spring 2019 | Undergraduate

The Human Intelligence Enterprise

Ajemian detangles neural nets

Ajemian Lecture Notes

If we are to capture human learning and cognition, we must envision the exotic architectures that make them possible.

  • New models of brain hardware can shed light on new models of brain software.
  • Design rule: Fit the approach to the problem you’re trying to solve.
  • The trendiest tools are not always the best tools for the job.
  • For example, modern computer systems are not the best metaphor for how brain hardware works. (6 key differences between computers and brains)
  • And artificial neural networks with backprop are not the best model of how brains learn. (6 unrealistic aspects of back prop)
    • In particular, backprop requires global lockstep coordination where brains are mostly asyncronous and local; and even recurrent neural networks pass information in one direction rather than many, as in the brain (compare recursive functions vs coroutines.)
  • And artificial neural networks with backprop are not the best engineering solution for every learning problem.
    • “Bulldozer computing” works well on object recognition, but fails on motor control (which has too-large state space) and medical diagnosis (which integrates many kinds of info.)
    • We should tailor new tools to such problems.
    • All of the recent ML performance improvements have come not from new ideas about learning, but rather from advances in computing power and data size.
  • Moore’s law will asymptote.
  • We must imagine new exotic architectures and learning mechanisms, tailored to the problems we’re trying to solve.
  • Surprise: There are other kinds of “neural networks” that are better brain models or better at solving ML problems:
    • Marr-Albus model of the cerebellum.
    • Hopfield networks
    • Self-organizing maps
    • Decision trees.

(C) 2019 Dylan Holmes. This work is not covered by MIT OpenCourseWare’s Creative Commons license but is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nd/4.0/.

Course Info

As Taught In
Spring 2019
Learning Resource Types
Written Assignments