The ability to reason accurately about a surrounding driving scene and anticipate the behavior of other road users is a vital task required for safe and effective autonomous driving. Experienced human drivers learn to predict that a neighboring vehicle with a turn signal will likely change lanes, or that a pedestrian walking at the edge of a road may be about to jaywalk.