Video: Exiting Flatland: Measuring, Modeling, and Synthesizing Animal Behavior in 3D
Taught by: Jesse Marshall, Harvard University (April 8, 2021)
Description: Mechanistic studies of complex, ethological animal behaviors are poised to define the next decade of neuroscience. Fully understanding the ontogeny, evolution, and neural basis of these behaviors requires precise 3D measurements of their underlying kinematics. While 2D convolutional networks have allowed for kinematic correlates to be monitored in repetitive behavioral tasks, they are ill-suited to track keypoints in 3D and across multiple behaviors. To address this, we developed a pair of tools: CAPTURE and DANNCE, that enable continuous whole-body 3D kinematic tracking across species, behaviors, and environments (Marshall et al. Neuron 2021; Dunn* Marshall* et al. Nature Methods 2021). CAPTURE uses chronically attached retroreflective markers and motion capture to continuously track the head, trunk, and limbs of rats in 3D. DANNCE generalizes this tracking capacity to animals not bearing markers by leveraging projective geometry to construct inputs to a 3D CNN that learns to perform 3D geometric reasoning and identify body keypoints in 3D. Together these approaches enable new lines of inquiry into computational ethology, the neural basis of behavior, and artificial models of behavioral production using deep imitation learning. I will discuss the technical details of these approaches and demonstrate how to use them to analyze the structure of animal behavior across multiple timescales.
Speaker Bio: Jesse Marshall is currently a K99/R00 postdoctoral fellow with Bence Ölveczky in the Harvard Department of Organismal and Evolutionary Behavior, where he invents new techniques for behavioral tracking and uses them to investigate the neural basis of movement. He completed his PhD in Physics in 2016 with Mark Schnitzer at Stanford University, where he developed new optical approaches for recording neural activity and applied them to elucidate the neural basis of movement disorders. He received his undergraduate degrees in physics and mathematics from the University of Chicago in 2009.
