Can I go somewhere that isnt prestigous because I believe in the research they are pursuing there?
Yes. However, you better be presenting, publishing, and patenting at the non-prestigious place as well as networking.
However, Lurkingfear is right that some very high-profile people are in some low-profile places. Usually, though, how those people continue to be high-profile is publishing and presenting.
A hiring committee may not know Big Name in the sub-sub-sub field to be impressed, but I doubt any but the biggest snobs would hold any kind of productive post-doc against you. However, be sure to sell it in your cover letter to search committee as this fabulous opportunity you took to work in the area of whatever, which you will now bring to their school as expert you with several publications to back up your claims of expertise.
The primary benefits of a prestigious post-doc are networking and access to generally better resources. Be sure to ramp up your networking via conferences and visits to past or potential collaborators if you take a non-prestigious post-doc as well as publishing.
In summary:
Publish and/or patent while keeping up a reasonable presentation profile at conferences and no one will care where you did the research.
Don't publish or patent and that fancy degree from a top-tier program will be ignored by everyone.