Magpie Developers

I hadn’t seen this post before, but it sums up my own feelings on a subject so well that I feel relieved knowing somebody has already done a better job of writing it than I could have done, and I don’t have to do anything.

[http://www.codinghorror.com/blog/2008/01/the-magpie-developer.html]

This reference is also superb:

[http://quoderat.megginson.com/2006/03/06/programming-languages-of-distinction/]

A Grumpy Guy Complains About Complaining

Over the past couple of weeks I’ve seen a handful of “Why Golang Sucks” articles. A couple were reasoned, the rest were copy-cat, none of them make any sense to me.

It would be easy to get the impression from pop-software news that the last line of C was written about 5 or 6 years back, and the last small band of C++ programmers are holed up in a cave someplace metaprogramming. I can assure you, this is not the case. If nothing else, one need only consult the sources for the various language-of-the-month compilers, or maybe the source debs for the last 100 or so updates applied to your Ubuntu machine.

I get that C and C++ also “suck” in many ways, I have certainly spent a significant portion of my life scratching my head staring at gdb and emacs. I think of these languages, these tools, as something akin to violins and cellos. They’ve been around, virtually unchanged, for several hundreds years. They are an absolutely gigantic pain in the ass to play- they haven’t got any frets, they have tiny fingerboards designed to be played by scrawny, malnourished 15th century musicians. They’ve got tuning pegs designed for strings made out of animal guts and which rely on friction alone to stay in tune. They’re freaking baroque. And yet, they get played a lot. You might argue that their design actually encourages mistakes.

So here we find ourselves in a world with various better choices than C and C++, yet they still get used for a pretty significant chunk of all the coding that goes on in the world. Why is this?

I believe you can sum up the reason that programmers still use these languages as something along the lines of “because they knew what they were getting into”. The language, including the flaws, are known, and the developers go with it anyway. Maybe there’s some platform/library lock-in of course, some inertia. But we kind of know, disregarding the annoyances and frustrations of using C and C++, there isn’t much you can’t write in those languages, and the shortcomings are widely known.

Which brings me back to these Go articles. The entire nature of the Go language is documented. Exhaustively. It is perhaps the most true specimen of a post-internet computer programming language. It was birthed on the web in the wide open, the specs, the docs, the compiler and the library are all there for you to investigate. How is it then, that this number (well at least these 4 or 5 I’ve complaining about), managed to understand their own needs so poorly that they devoted a big chunk of time to learning and hacking on Golang, only to find that the language was missing some crucial feature that they can’t do without?

I’ll give you a hint, which is that there’s nothing wrong with the language. The problem is that we do a piss-poor job of understanding what it is we expect from programming languages right now, at least members of the pop-software culture (myself included). There’s a kind of tacit understanding, with regard to quality in a programming language that “I’ll know it when I see it”. In other words, I’ll learn enough to start writing a handful of programs in some new language, and then if the language is any good, it’s quality will gradually reveal itself to me as beautiful arrangements of code in my emacs buffer.

This is utter horse shit. This is like finding a new alloy of steel and then building a bridge out of it to see if its any good. If Go doesn’t have generics, and you know you need generics, why the fuck did you bother with Go? These details are widely, exhaustively documented; they are not surprises.

I’m not a golang enthusiast, but I am a lover of programming languages, and as a long time (but not brilliant) C and C++ programmer, I see a lot to love in Go. However, I’m thinking of Go in terms of its ability to replace programs and methodologies that I’d probably otherwise solve with the classic GNU autoconf/make/gcc conglomerate (avert your eyes!). It is important for people to understand not the language itself, but also where the language is coming from. When Rob Pike mentions that one of the initial motivations for Go was that it was taking too long to compile giant C programs, that should give you some intuition into the problems they were trying to solve with the language.

Go introduces some new kinds of problems, but nobody in the project is keeping them a secret. It’s the programmers responsibility to know what is required of the materials.

Wanting to Be Something vs. Wanting to Do Something

Julia Evan’s article on becoming a kernel hacker reminds me of the the early 90s, when earnest young hackers dutifully ftp’d down the latest Linux kernel patch file and excitedly ran make config to see what new stuff had been added, the rebooted immediately to check out the new uname sweetness.

Rarely was there any real reason for this other than bragging rights on IRC, although occasionally there would be some excitement when a previously unsupported sound card showed up as supported. Mostly it was just enthusiasm, for running one’s own server, making one’s own choices about how the machine would be maintained. We’d all been living under the unspeakable tyranny of root on the university shared UNIX machine, so this kind of freedom was a big deal.

The credits file did not go unnoticed, and the names contained in it were (at least to me) like stars on baseball cards; Ted Ts’o, Alan Cox, Dave Miller. These guys were my heroes, hard-hacking, hard-flaming intellectual monsters. These guys weren’t writing software, they were writing a freaking kernel. No libc for these guys, just a compiler and a bunch of registers and interrupts and all that good stuff. Hard core.

This was in 1993, and although my secret wish was to hack on the kernel myself, I didn’t dare commit the embarrassment of asking on the Linux kernel list the obvious question of “how can I become a kernel hacker”, although it got asked weekly at least by shameless newbies. It is still being asked today.

The problem I encountered is that, every time I opened up the editor to “hack on the kernel”, I was faced with the immediate and overwhelming problem of “what am I going to hack?”. I could start by fixing some typos, although I’m not really sure what I would do with the patch once I figured out how to generate it in the right format. I could try to find some bugs and fix those. There wasn’t kernel Bugzilla at the time though, so it was tricky to find them. I could add a new driver for some unsupported piece of hardware, only, I didn’t really have any money to buy unsupported hardware, and I didn’t have the foggiest clue how one would go about writing a driver for a piece of hardware.

I never really got off the ground as a kernel hacker, until one sweet day years later, when I was working as a system administrator for the university. I had installed (probably Redhat) Linux on a re-purposed server that had a DEC FDDI card in it. Somebody had ported the driver written by Matt Thomas, who did a bunch of drivers for some of the Tulip and various other DEC DEFPA/DEFEA Ethernet/fddi cards, to Linux. I went to build a piece of software, Samba I think, on this machine, but it crashed immediately on start-up with a segfault. After spending a few hours trying to figure out gdb, I managed to find that one of the structs that contained the hardware address for the NIC was the wrong length in the kernel header (Ethernet uses a 48-bit MAC but FDDI was something else).

I was a kernel hacker! I fixed it, and it worked, and I excitedly sent off my one-line patch to anyone and everyone. I got word back from the driver maintainer that he’d gotten the same patch from a few other people and that he would get it into the next kernel revision. Eventually, need have necessitated hacking on the kernel, but I wasn’t special; at least three or four other people had found the same need.

Over the years I have wanted to be all kinds of things. A musician, a kernel hacker, a crack C programmer, a Erlang programmer, a builder of fine musical instruments, an amateur radio licensee, a system administrator, a manager of software developers, a mixing engineer. I’ve lost count of the things in which I have held short-term avocational interest.

Now that I’m older, I’ve had an opportunity to work with at least one instance of a genuine professional example of each one of these jobs that I’ve lusted over. One thing has been true for every single one of them; none of them, not a single one, wound up in that position intentionally. In other words, the kind of lust that I had to “be” a certain kind of person did not get a single one of those people into the perhaps enviable jobs they were in. I remember reading Robert Love’s story about how he got started working on Linux; he’s probably an exception to my rule, but he was also self-motivated in an extraordinary way.

What puts them there, in general, is need. At my last job we had a team of guys whose jobs were to write code (some open-sourced, mostly not) that ran in kernel modules. Over the years these guys had gone from writing C, to writing IP routing code in C, to eventually being called up to have some of the routing code run in kernel-space, etc. They’d been thrown into the trenches by need.

It’s a gradual transition built upon layers and layers of accumulated skill. The reason that people like Alan Cox and Ted Ts’o show up in the kernel credits file is that they have invested a lifetime of study and effort in the design and operation of microprocessors, so transitioning to a relatively big Linux kernel was a natural step, not necessarily a first step.

It is folly to give young people advice about how to be something, because you cannot become something by will alone. We become things in response to need, not solely as a consequence of desire. Many of the things I’ve wanted to be, like a Erlang programmer, I eventually got the opportunity to become, but only after paying my dues. Programmers are in a kind of partnership with the people who pay them; my employer is willing to take a risk on paying me while I am learning a new language and not producing, but only because I’ve demonstrated a skill for learning new languages. It’s a good tradeoff for each of us.

Although we have to be careful to protect these wishes in young people, I think we also have a duty to disambiguate to young people that prestige is usually the product of an enormous volume of work. If a young person wants “to be a kernel hacker” but doesn’t know how to run an assembler, how can he or she know whether kernel hacking is something enjoyable? Perhaps writing a few small programs in x86 assembly is a better way to test the waters than struggling to read the LKML. It’s harmless enough to give some token advice, but finding a way to help these kids capitalize on their enthusiasm working on fundamentals would be better.

Esteem and Work

I’m a little unusual in the software development profession in that I have a “serious” side hobby. I play an instrument in what you’d probably call a semi-professional capacity. On a good year I probably play 20-30 gigs, play on a record or two, and make a couple thousand bucks (then squander all of it on instruments).

I’m not in it for the money; I’d starve to death if I was. I got a huge thrill out of playing in front of audiences in the beginning, and still do when they’re gracious. Playing in clubs is usually a pain in the ass though, so the joy in playing now comes from being able to impress people I respect with my playing and having them call me back for gigs.

There are two wins for me when this happens. First, it is very exciting to play with musicians who are really good. If the chemistry is right you will hear this incredible sounding music and it will be hard to believe you are partly responsible for it. The other win is that my self-esteem is boosted each time one of these calls happens, since it is a validation of the work that I have been doing.

Music and Software have a couple of similarities, most obviously that there is a lot of room for creativity. Both have rules, but those rules depend on context and there is a playful disregard for them among musicians.

The other similarity is that individuals and very small teams can win enormous amounts of attention while doing music or software. The knowledge that it is possible to hit it big as a programmer can lead to weird situations in a workplace where there are a large number of achievers for whom their work is the primary source of esteem.

I think that people who aren’t involved in the music business really believe in the idea of rock stardom, whether they are fans or outright wannabees:

  • Get an instrument
  • Learn the instrument
  • Join a band
  • Make an album
  • Get famous

It almost never actually works that way. Everybody knows this trope (hell people have written songs about it) because the only musicians they know are stars, and that’s the story of a star. Unless you’re a musician yourself, you probably don’t know about the millions of people who want to make a career out of music but never get anywhere close to stardom.

For every famous guitar player in the world, for example, I could find you a hundred guys who have the same, or often way more, talent and discipline. Most will have recorded 4 or 5 albums that you’ve never heard, play 2-10 gigs per week with people you’ve never heard of (and a few you have), and are getting by if not prospering. They teach all day at least one or two days per week, to boot.

For these people, taking as many gigs as they can get is not a means of increasing their esteem in the music community, its simply that unless you play your ass off, you can’t make a living. Very often there is real risk in terms of the personal resources that must be wagered when pursuing an opportunity.

When they show up to play these gigs, things are different than people outside the business expect. Even though a music director or bandleader may hire a musician on the basis of their enormous talent and improvisational creativity, many gigs have very formal requirements to play a certain way at a certain time, to endure outrageously long rehearsals, and to get fined for being a minute late. If you can play by the rules, yet be powerfully creative when called upon to do so, you can make a living at it.

I see lots of problems with the idea of passionate creativity in the software business, and mostly they are the same problems that strong willed people run into in the music business.

If you’re an entrepeneur you get the choose the rules, but the caveat is that only certain kinds of people are cut out for entrepeneurial life; they’re like the bandleaders of the music world. If you can handle being a one-man-show, fantastic. If you can attract talent and run a team, then you’re really cooking.

If you’re a working stiff, a gun for hire, you are typically not getting paid to be creative, you’re getting paid to do work, to put on the show. The performance may require enormous talent, and you may take a couple of solos in the spotlight, but the show is

The bulk of professional musicians are not simply playing gigs; a career player might be arranging written music, producing recorded music, directing performances, in addition to the teaching and performing that is required. This is work. It’s music, but the tasks are very clearly defined, they require a lot of skill, and have real value.

When musicians show up to play these gigs, there are a lot of brown M&Ms. Although a music director or bandleader may hire a musician on the basis of improvisational creativity, many gigs include very formal requirements that one should play a certain way at a certain time, endure outrageously long rehearsals, and to risk getting fined for being a minute late. If you can play by the rules, yet be powerfully creative when called upon to do so, you can make a living at it.

Outsiders, I think, see performers on stage and work out a naive equation in which releasing and performing records leads to fame (i.e. esteem) and in turn to crowds of adoring fans. Gaining esteem this way is akin to becoming rich by playing the stock market. There’s skill involved, but also in many cases a lot of luck. It is not a very sound strategy with which to pursue a career.

I find the process of building a bridge or an aircraft carrier way more mind-blowing than writing one web-app or recording an album, but it would be difficult to look at a giant engineering problem and find a small number of people who deserved glory for it. This happens all the time in Software though; a programmer writes a piece of software that becomes wildly popular, and before you know it, that person is a household-name among nerds. Hero worship happens way more in the software development field than any science or technology field I’ve seen. The pursuit of stardom in the field has become embarrassingly obvious, and it comes with problems.

A good example of this is an article I read several weeks ago which treats with a somewhat condescending tone (I believe it was unintentional, but illustrates the principle I am getting at) toward the segment of programmers who work in languages other than the latest hotness and don’t blog about it or give talks at conferences. This article, and gobs of others on Hacker News and its ilk, are built on a tacit understanding that to be somebody in this business requires a certain avant-garde approach. It would not do, for example, to be caught programming in Java or, god forbid, PHP.

There’s also a strong sense in this crowd that it represents today’s creative class, the leading edge of intellectual enlightenment. There’s a dearth of blogging, with elegant eastern design sensibilities and expensive-looking fonts. A lot of self-written bios in the third person, replete with professional looking photos of the author on-stage at a conference, hands folded Merkel-style delivering a serious looking talk to a rapt audience while wearing one of those almost (but not quite) invisible head-mounted wireless microphones. It is positively mortifying.

I see lots of problems with software engineers identifying so strongly with passionate creativity, mostly the same problems that strong-willed people run into in the music business.

If you’re an entrepreneur you can decide on your own rules and do whatever the hell you want, however, only certain kinds of people are cut out for entrepreneurial life. They’re like the bandleaders of the music world. If you can book gigs as a one-man-show, fantastic. If you can attract talent and run a team, then you’re really cooking.

If you’re a working stiff, a gun for hire, you are typically not getting paid to be creative, you’re getting paid to do work. You are playing other people’s gigs. The bulk of professional musicians are not simply playing gigs; a career player might be arranging written music, producing recorded music, directing performances, in addition to the teaching and performing that is required. That is work. It’s music, but the tasks are very clearly defined, they require a lot of skill, and have real value.

The main thing here is that these people are able to make a living because they get the calls. They get the call because they have a track record of showing up and doing the right thing, regardless of whether it is “their thing”. A pro might play Top 40 Friday night, a Wedding on Saturday, and a country dance hall Sunday. This life may not appear to contain glory, but there’s something to be said for having one’s esteem delivered in bite-sized portions over a lengthy career rather than dropped from the cargo bay from altitude.

The real problem at the ground level is that these esteem-oriented programmers are preoccupied with attention in the same way Kim Kardashian is, or the way a cocaine enthusiast is always working on a score. It is a compulsion, a mildly pathological delusion in which they imagine the eyes of the community upon them. Worse yet, it is distraction from their work which, curiously, could be a source of esteem if they chose to accept it as such.

To coin a phrase from Steinbeck, many of these programmers see ordinary software work not as a source of wealth and satisfaction, but as a temporary embarassment on their path to acclaim. They just need to find the new langyage for that project in that will get them there.

Esteem-Oriented Programming

I enjoyed reading an article from a fellow named Ben Northrop. About 90% of the posts I find by way Hacker News, et al, are crap; in a roundabout way that’s actually the topic of this post.

I am vastly more cynical than Mr. Northrop. I stormed through his essay and was annoyed that the programmer motivation I’d been spending the past few months thinking about was missing from his diagram. I stewed overnight, ruminating about my own post, the re-read Ben’s post, and realized my mistake.

I felt Ben’s diagram was missing a corner- the one I’d been thinking about myself. In my first read of the post though, I’d skipped the first sentence where he describes being technological motivation as being for the love of learning. A cynic might call this adopting technology for the sake of adopting technology, although that’s not always a bad idea. I think this idea makes more sense in the hardware world, where it’s very easier to measure improvement in terms of cost or efficiency. In software, I usually think of technology-for-its-own-sake in the context of a suggestion that we change a language or a library along with a vague promise of improved productivity or reliability. It’s usually hard to prove.

Anyway, my first take on Ben’s essay was that he meant that technological motivation was a sort of righteous march out of one paradigm-age into the next. I’d been thinking a lot about the current that is driving the adoption of Functional languages in industry. The functional idiom is one of my favorite aspects of programming, but having witnessed the rise and fall of Object-purism during my college years, a lot of the software pop-culture surrounding FP is I think being promoted by same breed of hot-house flower that aimed to encode the world as an object or a design pattern.

At this point, I should pause and offer you my sincere promise that this is not a slag piece on the functional paradigm, or it’s languages. At my new gig I’m knee deep in Scala and Erlang, and I’m enjoying them tremendously. Also, I don’t have to remind you that FP is absolutely not new, nor has it remained unpopular until today.

What has bothered me though is the trend where developers acknowledge the shortcomings of the paradigm (chiefly the learning curve) vs the strengths (ease of concurrency, or whatever your favorite benefit is) but then ignore those factors and go ahead and use the functional tools anyway. There’s a kind of tacit acknowledgement that these tools represent the new thing, and that settling for the old thing would be embarassing.

The argument that I am going to make is not advanced very readily down this avenue, so I’ll apologize and take an abrupt turn.

The other habit that I notice now is programmers talking a lot about whether the code they are writing is idiomatic or not. Whether code is idiomatic or not is subjective, and programmers who go to any length to achieve idiomatically-correct constructs in their programming are, ultimately, striving to satisfy someone else, whether they recognize it or not. It is the same as trying to please your conversation partner by speaking a foreign language with a correct accent.

Who could fault a person for trying to speak with the local accent? At worst, the person might be trying to blend in to the culture. More practically, he or she might simply be trying their best to be understood. Really, it doesn’t matter, because people do this all the time and nobody thinks its wrong. Regardless, its still a thing that people do for their esteem; we want to be understood, and to fit in.

The same is true for idiomatic style in programming. Although people like to claim that programming “idiomatically” has advantages unrelated to esteem, reading code is nothing like participating in a conversation. In order for a programmer to recognize that a construct in a program could be simplified, he must have to understand it first, otherwise he couldn’t possibly know how to substitute a more idiomatically-correct improvement. You might argue that time is wasted by not using the most familiar constructs from the idiom, but I think you’d be wasting your time, because programmers delight in using constructs that trade obviousness for conciseness, and other programmers are willing to accept these clever constructs as a kind of cost-of-doing-business. And then they repeat them.

This is not a habit that programmers adopt in an attempt to be more clear; it has everything to do with being seen as clever by other programmers. It is purely an effort to boost one’s esteem in the community of programmers who read the code. I can’t think of a single instance in which I saw a Perl programmer use the so-called Schwartzian Transform in a program where there wasn’t a corresponding comment above the block that pointed out the fact, lest it go un-noticed. I think of this as esteem-oriented programming, and I feel that it is the missing vertex of Ben’s polygon.

Alright now I’ve said it. I can hear the cogs whirring, the arguments being formulated, outlines of lengthy comments clattering into emacs. The jokes on you since I haven’t bothered to add comments to my blog. The thing is, there’s nothing wrong with doing things for your esteem. I do it constantly. I tell stories in order to show how much I know about obscure subjects. I comment in meetings to make it clear that I understand a thing. I make jokes to make it clear that I can draw analogies from the topic at hand. I’m fine at it, but I understand that I’m far from being the most clever person on earth but that there is still value in managing my esteem as a programmer, including trying to adopt the idiom of a given language when writing code.

What programmers need to understand, however, is that in the workplace the decisions that they make, whether by trying to find opportunities to substitute idiomatically-correct but perhaps less than intuitive constructs into a program, or when they select Haskell to write a program that might as easily have been a shell script, is that their choice comes with a cost. If the program is very short but difficult to understand, years later a junior developer might be tasked with fixing it, decide the program was unreadable, and rewrite it. If your program is written in a language that attracts other esteem-oriented programmers, they may be unable to fight the urge to rewrite it in the latest popular language. When you write intensely clever programs, other esteem-minded programmers often look upon the program as a challenge; how can I improve upon this?

In every decision you make as a programmer, you must acknowledge the role that esteem plays in your life, and consider that your own esteem and the esteem of others may play in the future of your work, and the work of others. Do you want your work to endure strictly because it was clever, or because it was thoughtful to the organization you wrote it for, and to the people who might be tasked with supporting it? Now consider these same values in the context of selecting languages, tools, or any other piece of technology. The point I’m trying to make is that there’s no shame in considering one’s own esteem. You just need to treat it like any other cost when evaluating options.

Ultimately, the way you make a lot of money in this business is by making money for the people you work for. It is difficult to to measure the amount of joy esteem brings a person, but the number of programmers that I’ve personally observed to have become wealthy by getting things done, as opposed to getting things done in a clever fashion, is lopsided. Ultimately, I think that a programmer who can get things done efficiently, and siezes upon the opportunities to be clever in the process, is the most successful.

Emacs-nox

In the past, passing –without-x-toolkit to the configure script for emacs was sufficient to get it to omit window system support from the build. Maybe this still works elsewhere but on my new work box running Wheezy, I still get a build with X. This works though:

./configure --prefix=$HOME \
    --with-xpm=no \
    --with-jpeg=no \
    --with-png=no \
    --with-tiff=no \
    --with-gif=no \
    --with-x-toolkit=no \
    --without-x

Foreword

I used to keep a blog where I would write a combination of stories I thought were funny, along with occasional long rambling pieces about politics or technology. After doing this for a few years I realized I’m ok at writing funny stories, but that when I wrote about politics or technology it was boring as ass to read, mostly because I tried to make it sound like I was some kind of expert on politics or history. I labored under the fantasy that impressionable people would wander across these posts and be moved by them in some way. In reality, the readers were almost all friends of mine. They already knew how I felt about stuff. They liked my funny stories and would endure the other stuff.

I am a compulsive reader of sites like Hacker News, lobste.rs, and Lambda The Ultimate, as well as an awful troll on the programming and Unix subreddits. Playing music is my other major hobby, and the people I know who play music for a living all acknowledge how blogging (I guess non-stop Tweeting also qualifies) is a good way to raise your own share price, so to speak. The problem is, I am neither an entrepeneur nor a genius; I had planned to title this site “Average Programmer”, after I read an annoying article about the “Unseen 99%” of developers who toil in anonymity sans blogs or tweets. I love the social part of being a programmer, but I despise the overabundance of elitist groupthink in the field; if nothing else, it bores me to read. So I figured, what the hell, I’ll write my own stuff.

So here’s my disclaimer; I am not an expert, but I’ve been doing this a long time and I have some opinions. My main goal here is entertainment, not enlightenment.

By the way, if I never get around to adding credits to the template I’m using, it comes from here. If you’re that person and you’re reading this, I really appreciate that you made the template available.