I also had something similar occur early in my pre-tenure years -- and only later, as I gained more understanding of the process ahead, could I see that my chair's (and others') behaviors came from tenured folk who darn well knew that such vague concerns would not surface in the tenure packet. The intent, instead, was to so unsettle me as to change my behavior to become less confident in the classroom . . . which, of course, can lead to actual (filed) complaints, lower evals, and the like.
What is wrong with people? It's ok to not like someone with whom you work. Just don't fvck with them.
Well, for a clue for what was wrong with that department, this and more became more evident when I came up for tenure.
And it became even worse, when I won my appeal of the turndown, and I became the first tenured woman in the department.
I hope that the OP is dealing with "only" incompetence. But if not, OP, you may be able to look back and congratulate yourself on catching on so early in your pre-tenure years. Your antennae are activated now. Monitor -- and trust -- your radar.