I've been to weddings with one person on each side and weddings with (literally!) 18 on each side. There are so many ways to accommodate large wedding parties (by having most people sit, for example), so you should include who you want to include. It's your day, and you want to be surrounded by the people most special to you. Don't worry about how it may look to have a big or small party.