Despite their importance as a reservoir of biodiversity, the factors shaping soil microbial communities are still poorly understood. In particular, the extent by which cultivation impacts the structure of bacterial and fungal communities in soil is not completely known. Using 16S rRNA gene and ITS sequencing, we characterized the soil microbiota of vineyards and of neighbouring permanent grassland soils in the Italian province of Trentino. We found that the soil bacterial communities have a core of conserved taxa that account for more than 60% of each sample, and this core is only modestly affected by cultivation. Differently, the fungal microbiota from the different locations had a much smaller set of conserved taxa. In addition, the fungal communities of vineyards were significantly altered by the presence of grapevine-associated fungi. Cultivation had a site-specific effect on the diversity of the soil microbiota, that in several cases was significantly different from what found in corresponding permanent grassland. We also found that in most locations there was a negative correlation between fungal and bacterial diversity, suggesting that these two component of the soil microbiota compete for resources. Our results highlight the different response of bacterial and fungal communities to environmental factors.