Some wasp species do build nests underground or in walls. You may need to get a professional in to get rid of them, since it can be dangerous to try to destroy the nest yourself if you don't know exactly what you're doing (the wasps can swarm out and attack, stinging hundreds of times.)
This Wikipedia article may help you identify the insects.
Paul