Ibiza Town is a good base for a holiday on the island, but, to me, not if you want it quiet as you seem to suggest.
Can you provide more information, such as how many bedrooms you need and, when exactly you are thinking of going? This summer, for instance, will already be well booked up in any reasonably priced villa.
I would think that Santa Eulalia would be a good bet for you. It's the third largest town on the island, so has a good selection of restaurants and bars, is not too noisy / raucus, and is within easy reach of most other towns you might want to visit.
You can also get a catamaran direct to Formentera from there, although the beach at Ses Illetes is very busy by the time it gets there. An early ferry from Ibiza Town would be the only way to guarantee a bed on that beautiful beach, unless you want to book a table at one of the two, (fairly expensive), beach restaurants in advance. They both have reserved sunbeds and parasols for diners.