Hi there,
We visited the Dordogne with a 3 year old a few years ago. We enjoyed the following
Rocamadour - village and castle clinging to side of cliff - fantastic views. Park at the top or bottom and then use the different modes of transport between each level.
Domme - bastide walled town
Le Gouffre de Padirac - caves with river at the bottom - boat ride
Quercyland - water park (we didn't go here)
We went to another park north of Le Bugue that had animals, rides and waterpark.
Don't worry about not speaking French - you can still go round the chateaux without a guide and I'm sure the guide spoke english in the caves.
Enjoy!!
Where are you staying?