When you suspect a conflict of interest for one of your references, you should ask that person if he or she is comfortable providing the reference.
The above is fine, but I prefer Frenchdoctor's take on this.
Personally, I'd ask another person to write a reference. This way, you kill two birds with a stone :
- it will appear more professional, honest, in a "treat me just like any other candidate" way.
- since the director already knows you, his opinion will be a fourth reference anyway.