This article on Code Project (found via Steve Pietrek) might be an excellent stepping-stone for someone trying to learn language-design and compiler technologies. While code-generation and templates are good first steps and are easy to do easy things, but you should be aware that as the semantics of what you're trying to accomplish increases, the difficulty typically inverts. That is, at some point the verboseness but flexibility of generating assembly-language or IL becomes less painful than subverting the semantics of C# or VB.NET (or whatever other language you use).

This actually touches on a broader point: one thing we've seen with the shift towards agile processes is an emphasis on refactoring. This requires a faith in the prospect that a program can be incrementally changed from one form to another. In practice, this is generally true, but there are certainly cases, and DSLs might be a good example, where there may be a discontinuity of architecture significant enough to foreclose evolution and force you back to viewing your initial work as "build one to throw away."

That still doesn't mean that one ought to regret the initial work. "The simplest thing that could possibly work," is still a solid principle for approaching a project. But the devil is in the word "possibly."