Jason Bock and Ted Neward did a good job summarizing specific talks, so I won't duplicate that effort.

Overall, I loved the conference, it was revitalizing for me. The speakers varied greatly in their presentation skills, but I actually liked the "texture" that gave the conference; you definitely didn't have the feeling of homogenized content. There were only two talks that struck me as near-useless and, from me, that's high praise.

Most impressive technologies:

• Newspeak by Gilad Bracha. I was utterly blown away by this. I never thought I'd think "Wow, that looks better than LISP for building languages!"
• Intentional Software as presented (and explained at the bar) by Magnus Christerson. I think I finally "get" intentional programming and, with that epiphany, reverse from being extraordinarily skeptical of its value to being extraordinarily impatient about getting my hands on it.

Most impressive application:

• ResolverOne from Giles Thomas and Resolver Systems. Hands-down brilliant. The top half of the screen is a spreadsheet, something familiar to 100,000,000 users. The bottom half of the screen is a programming editor. Click in the programming editor, type:

def foo(v) : 
   return v + 1
 

Click on the spreadsheet, put "1" in A1, and in A2 "=foo(A1)" and see the results of your Python function. Look back in the programming editor and see your spreadsheet expressed programmatically. I intend to write more about Resolver soon. Like OneNote, this is one of those applications that give you a shock of recognition -- "Yes! This is what I want!" -- and makes you lament how much effort we spend gilding lilies in our industry when we ought to be spending our time out trying to come up with this kind of real innovation. (Plus: it's a fascinating development story... More tk.)

Stupidest Thing I Said: "Why did the OLPC choose to emphasize Python when it ships with Smalltalk and Smalltalk is clearly a better--" At which point I was struck by a sock filled with quarters. Now, in defense, the sentence I'd intended to complete was not, as my assailants later asserted, "--language" (Hey, I'm the one wrote wrote "The myth of better programming languages") but "-- learning environment?" But even so, after Jim Hugunin's talk (in which it took him three lines of code to import the XNA framework, XBox controller library, and Microsoft Robotics Studio connection) and Giles Thomas' talk (Resolver), I withdraw from the field.

Second-Stupidest Thing I Said: "Intentional Software? Simonyi's galley slaves? Those guys are the Duke Nukem 3D of programming -- they'll never ship!" "Well... actually we're in beta now, and have you seen the boat? It's quite nice."

P.S. I had to leave at mid-day Friday, meaning that I missed several talks that might have been highlights -- I was very much hoping to hear Chuck Esterbrook's talk on Cobra, which takes Eiffel-style Contracts and extends them with integrated test-specification (finally!).

Yesterday I went swimming on a coral reef in 100' visibility with whales singing so loud that I almost expected to see them underwater. For the next 3 days, I'll be in Seattle, where I assume they have this thing I hear about called "heating."

Actually, after canceling the trip because I had such a non-productive, unlucky, and generally lousy January I decided at the last minute that the best thing in the world for me was to be in a room filled with people who totally out-class me.

My plan had been to spend, like, 6 weeks developing something and then saying "Well, I threw this together on the flight over..." and then have people say "well, it sucks of course, but for a plane flight, it's not bad." Now I'll just sit quietly in the corner. I'll be the guy wearing three layers of sweater over an aloha shirt....

Definitely looks worth checking out:

I used to live several miles from Harvard Square, which was (a) a destination in and of itself and (b) the last stop on the Red Line (back in the day). I used to live about 5 minutes from the Belmont Center route, which ran occasionally, and about 15 minutes from the Waverly Square route, which ran more frequently. However, if I went to the Belmont Center route and walked another 20 minutes, I could get to an intersection served by the Arlington buses as well as the Belmont Center bus. However, doing so involved abandoning the Belmont Center route for about 5 minutes, during which, of course, the bus for which I had been waiting might very well drive by... 

Aside from trying to figure out if you got more, less, or identically wet by running or walking through the rain, the question of how best to get to Harvard Square was a central preoccupation of my teenage years. Mathematicians have concluded that I should have waited for the bus.

Mitch Barnett responds to No Silver Programmers with a discussion of visual programming in BizTalk:

[Y]ou drag n dropped onto the canvas (kinda like Visio) and then you would set a bunch of design time properties and sometimes call out to some C# assemblies, or web services or adapters, but most often it was a matter of using the design surface, which included a designer for mapping one file format to another and a rules engine design surface, etc.

If you thought of the designers as “raising the level of abstraction” or a visual Domain Specific Language, in a lot of cases, not much real C# code was written – you spent most of your time in this abstract kind of world.  Point being, when I was hiring programmers, like in the traditional sense of yeah, I can write lots of C#, I found that some programmers found using the designers, very easy and others, just could not get their heads wrapped around, to the point, that they could not actually do the job – period. 

Meaning that in this specialized instance, I found programmer productivity was totally dependant on how well your head dealt with abstractions and not traditional programming constructs – to the point where some of my guys could whip out a BizTalk solution in a matter of days, others maybe a week or so and others, never.

[Emphasis Added]

Just to be a skeptic, I'm not sure that there's just 1 thing (raising the abstraction level) going on when you're developing on a design surface. You're engaging the visual and spatial and pattern-recognition components of a your mind in a way that, perhaps, leads to different problem-solving skills (whether inherent or learned). I dunno'. The interesting thing would be to study the problem-solving ability of developers using a visual design surface with an equivalently high-level abstraction that used a text representation (block_1 = new Foo; block_2 = new Bar; block_1.outputPlug1 -> block_2.inputPlug1; etc...)

A related point, and an important one, is that different languages "engage" your mind in different ways and part of the joy of being a computer programmer is finding a good match (or better yet, good matches) for your particular wiring. Personally, I quite like pictures in some development tasks, but have never found a visual design surface that actually delivered a high-enough level of abstraction (in terms of "things you can do with code," i.e., meta-programming, reflection, code generation).

Philip Johnson tells me by email of Hackystat:

[A]n open source framework for collection, analysis, visualization, interpretation, annotation, and dissemination of software development process and product data....

Hackystat aspires to be the "Apache" of software engineering measurement systems: open source, standalone, scalable, extensible, and platform and technology neutral.

Our current thrust with Hackystat is "Collective Intelligence for Software Developers", and we are thinking about things like annotable Simile/Timeline representations, mashups with Facebook and Twitter, and other ways to create a kind of collaborative sense-making between man and machine so that metrics can be reliably and efficiently refined into actionable knowledge.

I've long been a fan of metrics, but traditionally programmers have resisted the introduction of metrics due to a (probably reasonable) fear that simplistic measurements would be tied to performance reviews, e.g., "Gee, you didn't add nearly as many lines of code to the system as Bob. No raise for you!"

Hopefully, the time of such fears has passed and Hackystat or a similar project will start giving us some of that missing data.

This is a photo by Wayne Levin, an incredible photographer who stars in a new video for Hawaiian tourism

While I'm at it, this is one Wayne took of Tina, bluewater freediving near a fish-aggregating buoy:

In email, Charles Gallo makes a point regarding "No Silver Programmers":

I think the HUGE difference between the mean programmer, and the silver bullet programmer is NOT how fast they can code "x" function points. The HUGE difference is when they sit down to solve a problem, and they can come up with an "elegant" solution to the problem that takes 1/2 the number of function points (or less), and therefore even the average programmer writing the design that these super programmers came up with would do it in 1/2 the time and with 1/2 the bugs (because there is 1/2 as much code).

This is definitely a fair point -- good programmers sometimes hit home runs. I had a case in point last year, when I had a client that had a problematic partner. The partner had a solution that they believed to involve such a huge barrier to entry that they could charge whatever they want, not respond to user needs, etc. Well, sadly for the partner, the barrier to entry was "write a parser."

Parsers are hard if you don't know what you're doing. Given their adamant refusal to make requested changes (not "that will take time" or "that will take money" but "no, that cannot be done") and confidence in their position, I strongly suspect that their parsing code was a very big, very complex hairball of hand-crafted code. On the other hand, I had Antlr, the Visitor pattern, and a couple weeks.

But the question is not "do homeruns exist?" but "To what extent is overall productivity affected by homeruns?"

This is a hard question to answer. On the one hand, you can say "Homeruns are infrequent -- even the best programmers spend a lot of their time doing mundane work." On the other hand, you can say "Homeruns, though infrequent, have ongoing effects on the system. Implementing a new Visitor over a nice AST is easy, if not entirely trivial." And, sometimes, a homerun can be something like Google's PageRank and have consequences vastly greater than "who has above-average productivity?"

I'm going to try to wiggle out of the conundrum by saying that homeruns (or even "extra-base hits") more often come from design and architectural decisions than from actual coding techniques. Not always, but more often. And once you raise the question of the distribution or impact of architectural talent, I don't think we have any data. And since we don't have any data about it, I think it's very hard to reason about it. I have my biases (I strongly distrust architects who don't code) but I respect the opposite belief (architects are less effective if they are busy "in the weeds").

I'm now reading Dan Simmons' The Terror on my Kindle. He's a very competent writer, and perhaps it's his very slow, very claustrophobic build-up (which he'd d*** well better pay off) that makes it so noticeable, but I have to say that I'm very aware of a certain "running on a treadmill" sensation when reading on the Kindle.

I turn pages, click click click, and the story progresses, but the only token of my progress is a bar at the bottom (the same length for all material, no matter the word count) that occasionally deigns to darken another pip. Like the animated plane on the in-flight display, this is almost worse than no indicator at all ("We still aren't past Nebraska?" Wait for it... wait for it ... tick it moves a single pixel...).

Especially with thrillers, the book-reading experience includes the sensation of the story moving from right-hand to left. It includes the canny appraisal of the upper-right corner, when the remaining pages become individualized -- "An hour more, then! I can miss the sleep!" The force of will to read every clause as the thumb holds down only the last 3 pages...

Andrew Dalke reminded me of an essay by Stephen J. Gould (discussed at:

http://www.michaelshermer.com/1996/10/bicycles-baseball-bacteria-and-bach/ and http://www.motherjones.com/commentary/columns/1997/01/outspoken.html) about the decreasing deviation in performance as a field matures. Relating it to the old studies on programming productivity, Dalke wonders:

If that applies to programming, and I expect that's a strong confounding effect, then those 1980-era studies have another problem - the field was too new. Many of the good programmers my age have been programming since high school or earlier, so about 20 years. Who in the 1980s had that background?

Haskell is a language that is pretty hard to "just pick up" (especially if you are mostly familiar with mainstream, C-derived languages). Perhaps "Real World Haskell" by Bryan O'Sullivan, Don Stewart, and John Goerzen will help the language (much beloved in academia) increase in popularity.

In reaction to No Silver Programmers, several people have spoken to the compounding benefits of good programmers (or the compounding pain of bad programmers). This is an excellent point. I think it's best put in this blog post, which takes the focus away from the developers and speaks of the quality of the codebase and the ease with which changes can be made.

[N]o one ever talks about code base productivity. If we do, where does it lead us?...[D]oing many common tasks in this code easily took 3x longer than necessary....[S]o bad that I was almost giddy whenever I could work on the new-style code. I'm sure you know what kind of code I'm talking about. Doing anything in this code easily took 3x even longer....A good code base makes any programmer more productive....

Maybe silver programmers don't exist. But is there "silver code"?

[Valued Lessons]

The author thinks "yes" and I agree. I alluded to this in my original post, but consider this curve:

This curve -- the cost of a change in software increases exponentially over time -- was gospel in the world of software development until less than a decade ago. It was absolutely accepted that, essentially, as the artifacts of software development (requirements, documentation, and, of course, source code) grew in size and complexity, the cost of making a change became ever-higher. In other words, as value grew in more-or-less linear form (one more feature), complexity grew in a non-linear form (like compounded interest).

About a decade ago, Kent Beck said "What if that assumption is wrong? What if we could have a curve like:

How would our processes and artifacts have to change to make that come about?"

And his conclusion was "Yes, we can achieve that second curve," by the processes that he called "Extreme Programming."

(One of the reasons why I was so heated in "No Silver Programmers" is precisely because the canard of "5% 20x productivity" not only accepts but exaggerates received wisdom. Just as Beck changed the industry by questioning the received wisdom of exponential cost increases, we cannot afford to perpetuate the myth of super-programmers.)

The way to achieve that second curve (which we can call the "agile cost curve") is, ultimately, to have a "silver codebase." (Well, a silver "artifact" base, because you have to do documentation and training and all of that stuff.) If the complexity of your codebase is growing non-linearly, no programmer (no matter their linear multiple of base productivity) can give you that second curve.

The point is that no heroic performance can change the shape of the exponential rise in cost over time. Tool choices don't change the shape either. What changes the shape of the rise in cost over time are practices.

So, the question becomes "How do you develop and maintain a silver codebase?" And it's widely believed that we have some answers to that:

• Refactoring (continual rewrites of the codebase rather than simple accretion)
• Incremental delivery with shorter timeframes (feedback and validation rather than the risk of diverging concepts)
• An emphasis on quality during development (increased cohesion, decreased coupling, improved project guidance)
• Preference for working code over documentation

Of course, practices are dependent on people -- all the people on the team. To me, a good programmer is one who contributes to the practices that maintain a "silver codebase" -- no matter how quickly they write code! Bad programmers are those who don't care about the quality of the codebase, either due to:

• Arrogance ("Oh, that's not a problem. I can fix that in a couple minutes." "I just pulled an all-nighter!")
• Lack of discipline ("Works on my machine!" "I'm done when I see my feature run the first time")
• The wrong temperament (delete a loop to hoist a variable)

In addition, the quality of the programmers contribute additionally to the interpersonal dynamics of the team, likelihood of a disastrous miscommunication, etc.

So, the interesting question then becomes "How do we maintain that second curve in the face of bringing in new developers?" In recent posts relating to programming talent, I know I've come across as a hard-ass (even though I'm a sweet guy, love my dog, wave people in at intersections, etc.). That's because I've reluctantly concluded that the "agile cost curve" is not stable in the face of poor hires.

In an agile team, there is a great emphasis on individual responsibility. We all learn new practices. We all make mistakes. We all break the build. In such a team, an irresponsible developer (not a slow developer, not an inexperienced developer) can subvert the entire process. And the result, in my experience, is that the "agile cost curve" can jump right back to the classic, exponential cost curve.

Bad programmers are not good programmers who are slow. They are actively counter-productive to the team.

She's got to get a CT to confirm, but basically the Doctor told her "no problem." So ends probably the least-productive week of the past several years...

My sincere thanks to everyone who got in touch and kept us in your thoughts. It really helped.

Sometimes I move a penny around on a poster, but this tutorial of creating a Popfly mashup takes things up a notch.

asdfasdasdf

My initial reaction to this is fear that Sun will try to force things a little too hard. When I think of MySQL, I think of ease-of-use and reliability. When I think of SQL Server and Oracle, I think of the ancillary tools and the fierceness with which DBAs cling to their specialized knowledge. DBAs don't have the programmers predilection to throw aside their tools for The Next Big Thing.

Actually, just a brief post. LINQ has injected into the mainstream a whole range of functional programming topics previously seen as esoteric. Putting aside the merits of each of them, the interesting dynamic to watch will be if these approaches generate a new sub-niche of programming. For instance, C++ templates led to template metaprogramming, just such a sub-niche. To what extent will awareness of lambda functions and monads be integrated into mainstream awareness and to what extent will they exist as a self-perpetuating "eddy" in the mainstream? (Or will functional programming actually spread and come to be a dominant way of thinking about software construction in the same way that objects did? I still view it as unlikely, but not nearly as low-chance as I did a few years ago.)

In response to "No Silver Programmers," Reuben Fries points out:

Say I'm a pretty good developer and there's this guy who is 5x worse than me, meaning it takes him a full work week to finish what i will finish in a day.

but then, what happens when it has 3x as many bugs, and it takes him 5x the time to fix each one?   suddenly we're at 15x for the bug fixing phase.  and actually, it should take him more than 5x as long to fix each one -- fixing bugs is much harder the more there are in the same code, and harder the longer it the time between writing them and finding them.  (and think about how much more time occurs between writing a bug and fixing one for this guy compared to me -- he has 3x as many bugs, often blocking other bugs from being found and always blocking other bugs from being fixed, and he takes 5x as long to fix each one...)

and it doesn't scale linearly when we look at code that will take me a month.  that's going to take him more than 5 months.  his design mistakes will pile up and his buggy code will pile up.  something that takes me a year he may simply be incapable of doing.

Yes. Further, a team with him on it will require more coordination than one made of average-or-better developers. A 90% finished system will suddenly reveal a crucial weakness because of his work violating encapsulation or somesuch.

Bad programmers are not good programmers who are slow. They are actively counter-productive to the team.

This is another great point why, to improve your team's productivity, it's important to focus not on the hope that you can discover a magical super-programmer but on rooting out incompetence.

Someone who should know better gave a commencement address based on the premise "5% of programmers are 20x more productive than the other 95%." This is utter BS and it's important to say so.

First, as boring as it may be to say "we don't have the data," go to scholar.google.com and find me a peer-reviewed study of individual programmer productivity among professionals not students. I'll wait.

What did you find? 1992's Watts Humphrey study? DeMarco's 1989 references to Coding Wars (the source for Peopleware and the likely source of the misquote in the commencement)? Maybe 1981's work by Barry Boehm? The literature sucks. Anyone, including me, who tells you anything about software development productivity is telling you beliefs, not science. Understanding that is vital: the entire agile revolution is based on revisiting cost assertions taken as gospel. "What if the cost of a change didn't increase over time?" That question is why the practices followed by the best software developers today have little in common with the practices of ten years ago.

So, accepting that everything anyone says about this is anecdotal...

Do programmers vary in their productivity? Absolutely. Is there a small percentage of programmers who are very much more productive than average? Absolutely. Are they 20x more productive than those not in their ranks? Absolutely not. The numbers quoted in Peopleware strike me as most realistic: the best are near 3x median and 10x worst. In other words, the significant thing is not that some professional programmers are awesome, it's that some professional programmers suck.

Doesn't that explain your world better? Have you ever met someone 20x better than you? Seriously, a day to do what you could produce in a month of undistracted, phones-off, heads-down programming sessions? Of course not. But have you met someone who's 5x worse than you? Who you say "It took you a week to do this?" I'm going to guess yes.

That incompetents can stay in the profession is not nearly as mysterious and fun as a secret society of magical programmers. It's a lot more gratifying to think "I've never met anyone 20x better than me, so therefore I'm part of the elite," than "What does it mean that Bob and I have similar titles?"

Programmer productivity is essentially code creation: develop a function that satisfies specification X. I love code creation; I used to run a newsletter called "Those Who Can, Code." I write software to help me deal with stress. But code creation is not what spells the difference in software development productivity, for which I absolutely believe that teams can achieve a significant multiple (perhaps 10x) over median productivity.

Software development productivity is the rate at which you deliver value to the client. If good data on the productivity of code creation is lacking, good data on software development productivity is essentially non-existent. It's extraordinarily difficult to measure. Measuring satisfaction is an insufficient proxy, because satisfaction will tend to be a delta from the last experience, not an absolute.

There are only a few things we can say with certainty about software development productivity:

• We don't generally do a great job at it
• Processes can improve it, if the processes are a good fit to the team
• Tools can improve it, if the tools are a good fit to the team
• It is always a people issue

Now I've come full circle to agree with the conclusion of the commencement address.

--

What does the actual sparse data say? This is the graph from DeMarco's Coding Wars paper, the graph that is generalized in Peopleware (where the probably-significant shading of COBOL entries is missing):

(Remember "good" is to the left, not the right!)

Pretty shaky foundation for the whole damn super-programmer myth, wouldn't you say?

Tina was diagnosed with leukemia when she was 29 and we were newlyweds. Two years ago, she was diagnosed with breast cancer. I got home from a business trip to find out that her routine dental x-rays had led to a referral to a dental surgeon because two things might be cancerous. One a shadow, the other white like another tooth. "With your history, just to be sure," the dentist said, but didn't offer a "but there are lots of other things that are more likely."

You Google "jaw cancer" and you hear about oral cancers in general -- bad -- or osteosarcoma -- bad.

She sees the surgeon on Friday.