Audience targeting in the search network can completely reduce the performance of your ads.
If you're targeting a smaller region with lower population, there's no need to target by audience.
Instead, you can refine your keyword targeting and also block bad queries with negative keywords.