I think that memories can stick from a very young age, especially if they're from important events.
Like I've said, I can remember stuff from the mobile home and the nursery school at old Coley primary school in Reading.
I can also remember going to sort out our new brick house with dad just before we moved in - I must have been about 4yrs old because I hadn't started primary school yet