There is an additional factor to this.
Google and probably the other major search engines are checking the IP numbers on links and it is well known that by linking to a "bad neighbourhood" can trigger filters which can cause your site to go down on the rankings.
Now this is when you link to a bad neighbourhood, although here, imagine being in a bad neighbourhood. That is, your host is hosting other bad web sites and they have the same or same class of IP like your innocent site. Guess what Google will think? Right.