Asteroids: Should we really be concerned?

Please take a look at the following simulation.  Play it in HD so you can see the full effect.

This animation shows the number of known asteroids in the solar system that have been detected, along with their corresponding orbits.  The simulation starts in 1980 and moves to current day.

Watching Earth, you may notice that it looks like we already should have gotten hit many times over the past 30 years, however, this is a 2-D representation.  The asteroids that look like they are hitting the Earth are actually passing in front of, or behind our planet.

I'll let you make up your own mind...