Do I really need an exhaust heatshield?

[mrvillicus]

Ive got a really annoying and loud rattle coming from either the heatshield or the exhaust system but before removing the exhaust I want to remove the heatshield, after looking at it tonight I can see most of the securing nuts have worn away at the metal so nothing is actually holding the heatshield on safely, I've removed the rear section over the end can but I also want to remove the centre box and the rest if I need too, will this be safe?

Thanks Ben