One good reason:
You would have some control over who is hired. That may help your own work/career situation. Just imagine not being on the committee and, after the search, being introduced to your new colleague, who is hostile to either you or your teaching, or wants to take the department in a direction you don't agree with?
The V in VAP stands for
visiting. As in, not going to be there forever. What stake do most VAPs have in the "direction" of the department? And the extent of control over the process gained from service on the hiring committee is not all that great, in my experience. We can read all the files, and if the committee advances a candidate the other faculty don't like, we vote and sometimes drop candidates.
Again, more forcefully claimed: departments who put VAPs on search committees are accommodating shirkers. If the department has less than or equal to two regular faculty members, then assigning a VAP may make sense. But with three or more? Then there's plenty of faculty to form a proper search committee.