As with any part of the brewing process, sanitation is the key. Anything that touches your beer must be sanitary. That includes the bottles and caps. I won't tell you what to use, everyone has their opinion, But, use something.
I've always been big on saving money. You don't have to use actual brewing equipment to brew. I do fine with stuff I get cheap, or free. The one piece I find essential, though, is a bottle filler. You can bottle without one. But, since it only costs about $2.50, it won't break the bank.