Although I'm not exactly a fan of civilisation based RTSs, there's something about AofE that does interest me, it's by far my favourite of this genre. It has that thrill of creating something unique, and it gaining in power, size and wealth - the same feeling you'd get from Roller Coaster Tycoon when you've made a successful park. The historical themes also give it that war like 'Camp A Versus Camp B' thing, and the fun comes from the success of beating and destroying the other teams; the fact that these are whole populations from an actual historical era (and who you can watch develop over time) give it that "I'm God and I'll do whatever I like" sense - if you see what I mean.
I do see it as a little, perhaps not nerdy (that belongs to the RPGers), but a geeky game - if you look at it in wider terms these types of games are essentially based on History, Strategy and Planning. Though unless you are one of those folks who plays it hours at a time, I wouldn't label anyone who plays them, as they can be very fun and enjoyable. As for the Expansion packs - I think they can all be justified and just offer something a bit newer to someone who's played the bulk of the game - it adds a bit more life to them.
The last AofE I played was II with the Conquerors expansion - after that I found that I didn't really have time for the others, considering my other tastes in games and the fact that AofE is very time consuming.