That is a really hard call. I'd say go with your heart on it. The mills is a very big down point but then on the other hand once you get in there and get to know everyone there may always be the chance to educate and influence change also
I'm sure if you did get the job that any one who got a hedgie would have a well socialized one because it would be too hard not to interact with them at work. I think you will make a great division either way and I know I won't judge if you decide it is something you would love to do and took the position
There may be good that comes out of that way too.