My comments on the MS OSS/Halloween paper

From jeske@... Mon Nov  2 18:53:21 1998
Date: Mon, 2 Nov 1998 18:53:21 -0800
From: David Jeske 
To: esr@thyrsus.com
Subject: halloween comments...
Message-ID: <19981102185321.Q22075@home.chat.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Mailer: Mutt 0.94.13i
Status: RO
Content-Length: 8700
Lines: 174

1) Project Direction 

>  Having historical, 20:20 hindsight provides a strong, implicit
> structure. In more forward looking organizations, this structure is
> provided by strong, visionary leadership.
>
>  { At first glance, this just reads like a brown-nose-Bill comment by
> someone expecting that Gates will read the memo -- you can almost see
> the author genuflecting before an icon of the Fearless Leader.
> 
>    More generally, it suggests a serious and potentially exploitable
> underestimation of the open-source community's ability to enable its
> own visionary leaders. We didn't get Emacs or Perl or the World Wide
> Web from ``20:20 hindsight'' -- nor is it correct to view even the
> relatively conservative Linux kernel design as a backward-looking
> recreation of past models.
> 
>    Accordingly, it suggests that Microsoft's response to open source
> can be wrong-footed by emphasizing innovation in both our actions and
> the way we represent what we're doing to the rest of the world. }

I think you are missing an important part of his point, which I have
seen in OSS before. I'll go out here on a limb and blindly state that
"Nearly all successful OSS projects are jumpstarted by a single
individual (or at least co-located group)."  It's very difficult to
communicate visionary ideas about a piece of software which is not yet
fully understood via the OSS model. I contend that in these kinds of
projects, OSS is victem of a lack of face time between
developers/designers.

Until we either (a) admit that we need face time, or (b) come up with
colaborative tools which rival face time in expressiveness, OSS
projects will be bounded by the ability for a developer or group of
co-located developers to jumpstart the project. 

Counter-examples are welcome.

2) OSS = `perfect' API evangelization / documentation

>  OSS's API evangelization / developer education is basically
>  providing the developer with the underlying code. Whereas
>  evangelization of API's in a closed source model basically defaults
>  to trust, OSS API evangelization lets the developer make up his own
>  mind.
>
>   NatBro and Ckindel point out a split in developer capabilities
>   here. Whereas the "enthusiast developer" is comforted by OSS
>   evangelization, novice/intermediate developers --the bulk of the
>   development community -- prefer the trust model + organizational
>   credibility (e.g. "Microsoft says API X looks this way")
>
>   { Whether it's really true that most developers prefer the `trust'
>   model or not is an extremely interesting question.

I think the thing that developers 'trust' is incorrectly explained in
his description above. I think with a better description it becomes
obvious why more developers prefer the trust model.

In order to be productive, a developer wants to reuse as much code as
he can. This means leveraging available code or APIs. The less you
have to learn about the guts of these apis, the quicker you can move
onto the next piece of the project. If you have documentation to an
API, the most productive assumption you can make is that the library
works like the documentation. When you find bugs in the library, your
assumption is proven wrong. Your willingness to 'trust' someone else
to make the library correct is inversely proportional to your
confidence that you yourself could understand and fix the
library. 

Excellent developers who have learned through experience to trust
themselves first will prefer open source, because they trust their
ability to fix the code more than anything else.

Most developers are not excellent. Most developers are average, and
those developers prefer to trust the idea that there are developers on
the other side of the fence (the API in this case) who understand that
code base and will fix it. These developers do not want to know the
details of how the API works, either because they won't understand it,
or they fear they won't understand it, or they just plain don't have
the time.

Interestingly, this also explains why so many excellent developers are
drawn to open source. They prefer to be in a world where they only
have to rely on their own skills to fix a problem.

>   Twenty years of experience in the field tells me not; that, in
>   general, developers prefer code even when their non-technical
>   bosses are naive enough to prefer `trust'. Microsoft, obviously,
>   wants to believe that its `organizational credibility' counts -- I
>   detect some wishful thinking here.

Even more insterestingly, my explanation above may identify another
reason managers may prefer commercial software to open source. They
trust their people skills, and in turn, their ability to get another
company to fix something they need fixed, more than they trust their own
ability to fix that problem. Or perhaps they trust their ability to
use power, leverage, and politice to motivate the other company more
than they trust their ability to motivate a group of non-controlled
developers.

>   On the other hand, they may be right. We in the open-source
>   community can't afford to dismiss that possibility. I think we can
>   meet it by developing high-quality documentation. In this way,
>   `trust' in name (or in publishers of good repute such as O'Reilly
>   or Addison-Wesley) can substitute for `trust' in an API-defining
>   organization.) }

I think there is an added aspect to this trust equation which OSS
needs to address, and to some degree has been trying to address. If a
company relies on a product from another company, they are both
financially involved in the transaction. I can trust that MS will try
their best to fix and release better versions of DirectX, because if
they don't, then nobody will be able to continue using it, and MS will
(presumably) lose as much money as the reliant company did.

However, if a group of OSS developers decides that they just don't
want to do their API anymore, they lose no money (instead they lose
their homestead). Bussiness care about money, not homesteads, so they
don't understand the incentive the OSS developer has for continuing
work on the project. Because they don't understand it, they don't
trust it. More importantly, if a business needs the development to
continue, they understand how to provide money, but the OSS developer
isn't doing it for money. A business may feel powerless.

2) Post-Parity Development

> [most of section deleted]
>
> This is possibly the single most interesting hurdle to face the Linux
> community now that they've achieved parity with the state of the art
> in UNIX in many respects.
> 
>  { The Linux community has not merely lept this hurdle, but utterly
> demolished it. This fact is at the core of open-source's long-term
> advantage over closed-source development. }

I don't agree at all. Linux has perhaps created the most full featured
UNIX ever. However, it is still UNIX. Linux has achieved stability,
and a complete UNIX tools set, but is still victem to all of UNIX's
problems. Linux dosn't even take advantage of last years' hardware
technologies like AGP, video overlay, newer PCI sound cards. It's
based on applications and display technology written before many of
it's users were born. (http://editorials.freshmeat.net/jim981031/)

I think inadvertantly you explained the problem best right here:

> In the open-source world, innovators get to try anything, and the
> only test is whether users will volunteer to experiment with the
> innovation and like it once they have. The Internet facilitates this
> process, and the conventions of the open-source community are
> specifically designed to promote it.

This implies that an innovator (presumably a single person or very
small group) can write the innovation themselves. When something only
comes from coordination among tons of developers working on different
projects (for example, interapplication communication, drivers, API
definition) the OSS model dosn't fare so well. How do you convince
developers to agree (and trust) an unproven system? 

OSS is good at creating more code to do more work, but it isn't very
good at creating less code to do more work.

The answers to OSS problems (IMO) always come down to writing some
more code, or some script, or something to bind dissimilar systems
together. If someone wants to apply conformance to a collection of OSS
tools, they end up confronted with the 'fork' problem. 

In other words, OSS is challenged by the chicken-and-egg problem of
new technologies. How do we demonstrate to OSS developers how much
nicer apps could be with better interapplication communication without
them helping to put it in?

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + jeske@...