And it turns out, there might be a clear winner after all. Yes, wearing tighty-whities can lower your sperm count. You read that right: Wearing tight underwear can lower your sperm count.