Maybe it's different in academia, but my business background makes me suspect that if you really have a body of heavy hitters in the field, but don't already have a specialized academic group in place, there is a reason. That reason is likely because all of those heavy hitters know how much work putting something like this together would be and they chose (and continue to choose) to expend their time and effort on more lucrative (personally and professionally) endeavors. My guess is that, if approached about the idea, many will be quite encouraging--after all, once it's in place it will likely benefit them, too. Some might even agree to "help". But, beware. They didn't get to be big names by spending their time benefitting others; they saw to their own careers, first. You need to see to yours, especially as you're just starting out.
Is the time you will spend on this really going to be more beneficial to you than spending time on your own research, service to your own university, and teaching obligations? It won't run itself once it's up. At least in the beginning, if you spearhead it, you're "it". What happens if you get a little further along and need to step back for the sake of your own career? While your professional profile might go up if it's all a resounding success, your name will also be linked to it if it fails. Think, also, about the politics involved in something of this nature. Right now, it sounds like you're on good terms with the senior members of your field. Will that continue to be true if you end up in the middle of two of them who disagree about policies or direction? If one or more of them agree to help and fail at follow-through, you're probably not in a position to do much of anything about it--and, if others expecting it to happen look to you for reasons why it hasn't, pointing fingers at a senior colleague isn't really a recommended option.
As I said, maybe none of these are concerns in academic circles, but they would be in the business/tech circles I ran in. From reading here, I suspect academia isn't that much different--in fact, I think the stakes might even be higher. It sounds like you already know and talk with each other. I'd think seriously about the suggestions you've received about less formalized ways to meet as a sub-group, whether that be sub-conferencing at a larger meeting (formally, or at a sub-group dinner), hosting the occasional sub-group conference of your own, or starting a FB page or listserv. Seems like a lot less work (for everyone), with fewer pitfalls and potential downside consequences (for you).