The first question I have is are you playing in the same house? The second question is which version of X-Box 360 do you both have?
If the answer to the first question is no, then it could be connection issues on your end. Call your ISP (internet service provider) and have a chat with them to see what they can do to boost your connection. If that fails, call Microsoft support to see if there are any issues on their end in your area. Also, something that might help, if you are using Wifi try connecting both X-Box 360s with ethernet cables. You may have to get a router to do this, depending on your type of modem.
If the answer to the first question is yes, then you should try a system link, and have your brother host. You can look up how to do this using a search engine such as Google or Bing. If you are both running off of the same internet connection, it's possible that it's giving his X-Box priority over your own, thus causing the issues you are describing.
If the answer to the last question is that you both have the same X-Box 360, then ignore the question.
If the answer to the last question is that you both have different types of 360, then that may be the issue. They may be fighting each other for access to the internet, and if your brother's may be winning out. If you are both using Wifi, try connecting an ethernet cable to your X-Box and leave his on Wifi. Or you could upgrade to a newer X-Box version (whoever is using the older version, that is).
If none of that works, then I'm sorry but I can't think of anything else