Picking what to wear in your family photos can be a bit overwhelming. The most important thing is, something you LIKE and feel comfortable in.
But, in more detail, coordinating and layered works best! Everybody matching exactly doesn't work so well, but coordinating is great. Same color pallet, like oranges, reds, and browns, or greys, blacks, and a pop of color.
Logos, graphics, and writing doesn't work well. But, solids, small patterns, and textures are great. Layer, layer, layer. Layer more than you would normally. It really looks fantastic in photos, and adds dimension.
One person is always welcome to wear plaid, but, 2 people in different plaids, don't quite work! But, there are always exceptions, and I've had some great shots with multiple plaids.
Button down shirts can be hard, as the buttons can buckle out, and are unflattering. So if you would like a button down shirt, I recommend wearing a sweater over the shirt, or make sure it fits you and isn't too snug.