I'm not sure whether I trust my sense of what's acceptable. I haven't put together the committee yet and I'm considering as potential members two scholars that I'd like to also be invited speakers, hence the question.
I don't think people really pay a ton of attention to this dimension of conference planning unless . . . it somehow borders on unethical questions regarding funding. Are all the invited speakers getting a nice speaker's fee? If so, then you are correct that you don't want them on the AC.
In that case, you can invite them, and then just informally ask for their advice about other speakers they would recommend for the program. From a networking standpoint, that's probably also a good approach.