Intermediate Mass Black Holes: fact or fiction?