Preheat the oven to 375°F and line a baking sheet with foil.
Pierce the sweet potatoes in several places with the tip of a knife and place them on the baking sheet.
Bake for 1 hour or until very soft, and the skin wrinkles when pressed. Remove from the oven and let cool for at least 30 minutes. Peel, discard the skins, and either mash the sweet potatoes in a large bowl or puree them in a food processor for a smoother filling. (Should yield about 2½ cups.)
Increase the oven temperature to 400°F and position the oven rack in the bottom third.
In a large mixing bowl, combine the roasted sweet potato, sugar, milk, eggs, vanilla, ginger, cinnamon, and nutmeg. Whisk until well combined. Pour the mixture into the pie crust.
Bake for 15 minutes. Reduce the oven temperature to 350°F and continue baking for 40 minutes or until the filling is puffed, and the center barely wobbles when gently shaken.
Let the pie cool completely on a wire rack. It can be served at room temperature or refrigerated for up to 24 hours before serving.
Serve with whipped cream if desired. Cover and refrigerate leftovers for up to 5 days.