I've had a similar problem with my bell peppers...see piccies here:
uh-oh-potato-blight-holey-peppers
My chillies are still hole free, although there has been some nibbling on the leaves....I've still not managed to track down the perpetrator as yet, I'm hoping that when my peppers are actually ripe that they'll still be ok
I did at one stage find a few aphids on the plant & also a couple of very small caterpillars - but it didn't seem enough to be causing the amount of damage that I was seeing.....which now seems to have calmed down.