Going off of what Anthroid said, it seems possible to me that a search committee could choose to ignore a negative reference as sour grapes. I've heard some pretty outlandish accusations (both on the fora and in real life) that if said in the context of a job search would reflect far more poorly on the person making the accusation than on the candidate.
And, indeed, this is why (in a situation with no reference from the department chair) we would call
several people at the applicant's campus, not one; and we'd have at least some of those calls made by faculty in other departments here who have, for example, a degree from that school and a dissertation supervisor still on the faculty. (I'm just remembering that when I was Director of Women's Studies, I more than once had calls out of the blue from women committee members asking if I "knew anything" about someone here who I may or may not ever have met because their university school was interviewing him (always "him" in this case) for a position as a dean or a department chair.) "Research," as you know, doesn't mean just taking someone's word for something, it means gathering all the evidence you can when something that doesn't fit the other data turns up.
In every case I can remember (maybe one every 7 years or so in my department), we've heard enough different versions of the supposed charges (the most recent involved "being mean to grad students") from enough reasonable people to decide to move ahead with the offer.