You Can't Tell Elephants Where to Go