You're NOT the only one experiencing this issue. It's actually a pretty wide spread issue.
Do you have Time Warner Cable for your internet? That seems to be the common issue, with maybe the modem being the cause.
People have tried lots of things like port forwarding, changing DNS, reinstalling, etc. It seems to be a problem that neither Time Warner or Microsoft wants to admit or really address.
However, if you bought it digitally from Xbox then they seem to be willing to give you refund because it is essentially a broken product. But if you're like me and bought it from amazon, with a gift card, then you are out of luck. No matter what Amazon will not return any digital game.
So my suggestion is to just return it. That's unfortunate because it's a good game.