Thanks to Janet Lefkowitz, a librarian at the Hebrew University of Jerusalem, the LibraryLookup project has added support for a fifteenth class of OPAC (online public access catalog) system: Ex Libris Aleph. This was an interesting collaboration. I'd looked at a few different Aleph systems, and found that their URLs varied from one implementation to the next in ways that I didn't have time to unravel. But Janet was willing to do this research, and she presented me with a set of Aleph URLs that illustrated the variations. I updated the Build your own Bookmarklet (BYOB) script accordingly.

A technical note: the BYOB script falls back on one of those guilty pleasures of scripting, eval. The fifteen URL templates are classified in a JavaScript array, like so:

queries['eosQ'] = '/VAR1/search/AdvancedSearch.asp?selectField1=IS&txtSearch1=\'+isbn,';

<input name="eosQ" value="WEBOPAC"/>

document.forms['byo'].eosQ.value
document.forms['byo'].aleph.value

var1 = eval ( "document.forms['byo']." + vendor + ".value" ); 

A historical note: Aleph, the first letter of the alphabet for more than three centuries, is a leading character in a book I just happened to pick up at the library last night: Language Visible: Unraveling the Mystery of the Alphabet from A to Z, by David Sacks. The name 'aleph' meant 'ox' to the Phonecians, and the letterform evolved from a picture of an ox's head.

The book is full of fascinating bits of trivia like that. What really grabbed me, though, was this discussion of the portability of alphabets:

Even if two languages are totally unlike, letters often can make the transition. Because their core selection of sounds (inherited from the alphabet's earliest stages) is close to being universal, letters usually can be adapted to a different tongue through only a few changes: three or four letters revalued to new sounds, a letter or two invented, unneeded letters discarded.
...
The newly independent countries of Azerbaijan, Turkmenistan, and Uzbekistan have not altered their spoken languages, which are Turkish tongues. But the governments have moved to replace Cyrillic street signs, textbooks, tax forms, etc., with new ones printed in a modified, 29-letter Roman alphabet. Elementary schools now teach Roman letters. The massive, disruptive changeover -- inspired by westward trade ambitions and hatred of the Soviet memory -- was declared officially complete in Azerbaijan, at least, in 2001. The new alphabet is modeled on that of modern Turkey, which switched from Arabic to Roman letters in 1928, under the westernizing regime of Kemal Atatürk.

Prior to 1940, Azerbaijan, Turkmenistan, and Uzbekistan used the Arabic alphabet, until the early Soviets imposed the Roman one in the 1920s. Thus the three regions have seen all three major alphabets in the last 80 years: Arabic, Cyrillic, and (twice) Roman. Although the languages of the three countries are unrelated to Arabic, Russian, or Latin, each alphabet has taken hold in turn.

Many folks wouldn't want to be reminded how easy it is to convert sparse input into a detailed profile that includes a phone number, a street address, a satellite photo, and driving directions. Re-entering the basic facts each time perpetuates an illusion of privacy. Yet the reality, for many of us, is that these facts are public.

Since I haven't told Google (or any other directories) to delete my records, I've implicitly given permission for Web applications to use that data. Let me now make that permission explicit. I'd be happy if a Web form made intelligent use of public information about me.

I'd be even happier if I could control the source of that data. Public information is a poorly defined concept, after all. There are online directories that still remember an address I vacated five years ago. I'd like to maintain the facts about me that I deem public. When applications need those facts, I'd like to refer them to a service that dispenses them. [Full story at InfoWorld.com]

When I previewed this column last week, it occurred to me that FOAF is an example of a mechanism that empowers users to assert facts about themselves. I don't expect earth-shattering results from the publication of my own FOAF file. But if for now it does nothing more than neatly encapsulate certain facts I'm sometimes asked to produce -- my picture, my bio -- that's useful.

In theory, it would be straightforward for business homepages to adopt a similar approach. They all do the same stuff: About, Company, Products, News, Contact. There's an obvious XML format for News -- RSS -- but not for these other things. It's easy to imagine a virtuous cycle. Companies publish their facts in a structured form. As a result, more directories list them -- and do so more correctly. As a result, more companies are incented to publish XML facts. And yet in practice, this hasn't happened.

Agreeing on a format is, of course, always a huge obstacle. But I suspect the Web design reflexes that we carry forward from the 90s are also getting in the way. It's been a very long time since I visited a company's home page and thought: "Wow, get a load of those DHTML menu effects!" Or: "Nice font!" I'm there for the information, and I'll shred the site trying to find it, grumbling the whole time. I know I'm not the only one who feels this way.

Of course I'm not wholly insensitive to aesthetics. In fact, I worship CSS wizards who can dress a skeleton of structured information in beautiful clothing. But people really hate looking at, or thinking about, that skeleton. Steve Jobs' demonstration of Safari RSS at Apple's recent developer conference was a great example. At one point, he flipped back and forth between the skeletal (RSS) and clothed (Web) views of a page. It was the least compelling moment of the keynote. Jobs himself sounded unconvinced, and the audience responded with silence.

It'd be great if business websites formed a FOAF-like "web of machine-readable home pages." But I don't expect that'll happen anytime soon. When people look at websites through X-ray glasses, they don't like what they see.

The Longhorn cover story ran this week. It includes a main story, an interview with Quentin Clark, and an interview with Miguel de Icaza and Brendan Eich. Here are some outtakes from my interview with Quentin Clark, director of program management for WinFS.

On XML datatypes

We see being able to store structured data (like contacts), semi-structured data (XML), and unstructured data. In the case of a Word document, the XML isn't described by the doc type in WinFS, but the WinFS type defines an XML datatype, you can stick in the XML there, and we can reason over that. A JPEG, when you pull off the excess headers, is just a series of 1s and 0s you feed into an algorithm, that will never be structured. But even within a WinFS type, like a doctype, we allow for all three components to be within an instance of that type. A photo is a good example. We have a picture/photo type, things like what camera model, where the picture was taken, plus the unstructured bitstream. With respect to metadata handling -- and property promotion is only part of that -- we talk about picking the author out of a Word doc, putting it into a WinFS property. We also have property demotion, so coming through the APIs you can reprogram the title of some item, and it finds its way back into the filestream.

Just ignore WinFS types for a second, let's say I don't need no stinking types, I'm gonna build my own. You can make it from scalars, the XML datatype, and a binary field. We've defined the Windows doctype to have the ability to have a filestream as well as an XML datatype. That gives you a lot of power. You can walk up to WinFS, create a scope -- all documents, the whole store -- and then issue XPath queries into items that have XML datatypes, then we can go reason over those things.

On sharing

We want to use synch as a way to enable people to share stuff with each other, and to enable offline experiences. It's the Outlook 11 idea that I'm working always locally -- we're bringing that model to all data. You'll have ability to use any scoping mechanisms -- querying, or explicit wrangling where you drop 16 things into a list -- and say, hey, I want to share this with whatever, another machine or with another person.

Relationships that exist within the scoping are no problem, we know how to rehydrate them on the other end. For relationships outside the boundary, there's a couple of different mechanisms. So if you got the document from me, but I didn't give you contacts, then if you do have that contact, Sarah Wiley, on your end, we'll reconstitute the relationship. If it's not a thing we can positively identify, then it would dangle. But after the PDC we changed the data model to make dangling references not really dangle any more. There was a point where we tried to work through the user experience of finding and showing danglers where we realized hmmm, that sucks, people won't know why are they even doing this. Can we change how we store and model the data so that's not an issue? So you'll have a document, and an author, and the system knows nothing about that author because you don't.

On things being in more than one place

Consider three scenarios. First, I want to keep a list of stuff I need to do today -- a piece of mail, the notes preparing me for this call. Let's assume there's no query-based way to do this, it requires explicit wrangling.

Second: in Outlook 11 I have search folders -- stuff from direct staff, stuff where I'm on the To line, the Cc line -- I use these every day as part of my reasoning over my life. A lot of the reasons you want to do things in that query way imply that you don't want to manually intervene. You wouldn't want to inform the system manually about the To line, although you tell the system that my staff is the following people.

The third case is about where you want things to live, physically. Where they are contained. In Outlook, I have a PST and the OST, which is the reflection of my online Exchange mailbox. No big surprise, the 200MB Microsoft gives me is not big enough to contain all my mail, so I have PSTs to keep stuff. That's a containment thing, where do I want it to actually live? I have a removable hard drive at home, at some point I decide this photo will live there.

So those are the three axes. The limitation of Outlook 11 is that it doesn't allow you to put an item in more than one user lassooing. We want to allow multiple lists, or folders, where you can put the same thing in both. We're removing that Outlook limitation.

We encountered significant design challenges around user experience and expectations, and also problems around the DAG (directed acyclic graph). Consider security. I take an item, it lives in a bunch of folders, what is the security on that thing? Folder 2 has it too, then moves to folder 3. All the way back on folder 1, does the owner have any way to know what's happened? Then there's naming. If I have a doc, call it "jon's doc," created in a single folder, then I want to have it appear twice in that same folder, what is it called? If it's in a second folder, and I delete it from folder 1, then at some point I rename folders and put the doc back, calculating namespaces becomes complex.

On the object/relational/XML "trinity"

Why do you need all three? I take it that it's obvious why you need objects: you program to them. The CLR has given us some language independence, and we've done a fairly good job building an object universe better than we had before, we're strong believers in that. As for XML, there's no argument there either. The big thing isn't turning out to be industry schemas, but the fact that you have this self-describing thing, this is what I can learn about it, and I can reason on it in a programmatic way by pulling it up into an object.

Then there's relational, this is harder to describe. I will observe that nobody has built an XML store that has the level of scale, performance, or capabilities of today's relational stores. It's just true that the relational model has a set of design characteristics that give it performance characteristics that are are just inherent. Doing things in the XML store doesn't give you the same benefit -- and that's not even accounting for the fact that there's so much data in relational stores today.

But we're taking the start of the Yukon XML work, and bringing it into WinFS. Our vision is a marriage of these worlds, this is why they did so much work around SQL/CLR in Yukon. And we've done more since then in the WinFS part of the code.

Yukon doesn't reach the end of the journey, WinFS the client does not, but that's where they're headed. In terms of a data platform, that's what we want. This couples with the discussion of structured, unstructured, and semi-structured data. XPath doesn't make sense with the JPEG bitstream. Having that object/relational/XML trinity over a breadth of datatypes, that's the holy grail, that's completeness.

On WinFS benefits

First, having datatypes in Windows, so you can do things like program around a contact. The shared data is a huge benefit, though admittedly it's tricky to get it right. If you are Eudora or Act, how do we make sure you can plug in and own the contacts that are yours, while ensuring that Amazon can still query into a contact and pull out an address?

Second, the Outlook 11 experience of being always local. So an ISV builds a specialized app for architectural firms. Thanks to WinFS synch, the user can go offline and online. Using rules, if meeting notes come in that talk about changes in plans, he gets an action item.

Third, customized experience. If I drill into docs by Jon about WinFS, I can name that query, reuse it, write rules about it.

Fourth, finding things. How many times do you get the call: "I created this doc, help me find it." You'll never get that call again. If their field of view is too broad, they can narrow it easily. Fulltext search is there. Life will become a lot easier for end users. We had to turn off indexing in XP by default, it was too slow and chatty. When you're chasing the truth, it's hard to do. Being the truth is easier to do. When people come in and make changes, we know about it, we're built on a database.

Feedster's Scott Rafer wrote to point out that there is a URL syntax for assembling the conversation around a blog post:

http:\//www.feedster.com/links.php?url=\
  http%3A%2F%2Fweblog.infoworld.com%2Fudell%2F2004%2F07%2F16.html%23a1041

You need to escape the target URL, which isn't easy to do while copying and pasting it, but is easy for a bookmarklet to do. So, drag this link -- Feedster Citations -- to your toolbar, and you can have one-click access to the conversations around any blog post you're currently viewing.

The Bloglines equivalent, by the way, is:

http:\//www.bloglines.com/citations?url=\
  http%3A%2F%2Fweblog.infoworld.com%2Fudell%2F2004%2F07%2F16.html%23a1041

and here is the Bloglines Citations bookmarklet. Again, drag it to your toolbar for one-click access to Bloglines citation lookups.

Currently, this pair of queries (Feedster, Bloglines) yields the same set of items referring to Friday's Feedster reloaded item. This suggests to me that conversation tracking is becoming more deterministic. Excellent!

Is it really necessary, by the way, to escape the target URLs? If not, the use of these query mechanisms would be able to spread more virally.

Congratulations to Scott Johnson and the rest of the Feedster for the launch of Feedster version 2. There are lots of new features to digest, but the ones that most interest me are those that enhance cross-blog conversation. At this URL, for example, I can find a tidy summary of the reaction to this item:

Excellent! Here's a suggestion, for what it's worth. The URL that produces that summary looks like this:

http://about.feedster.com/?id=39&epoch=1089217495

The equivalent Technorati query looks like this:

http://www.technorati.com/cosmos/search.html?url=\
  http://weblog.infoworld.com/udell/2004/07/07.html#a1035

Of course there's a supercomputer out there that hasn't yet been applied to this problem. I'm not the only one who wonders when, and how, it will.

BPEL (business process execution language) is the XML-based language of Web services "orchestration" -- that is, a means to connect multiple Web services to create end-to-end business processes. Recently, InfoWorld Test Center Lead Analyst Jon Udell interviewed BPEL expert Edwin Khodabakchian about the future of this language. Khodabakchian is CEO of Collaxa, a pure-play BPM startup whose BPM orchestration product has supported BPEL for more than a year. Collaxa was acquired by Oracle earlier this month, and its BPEL Server product is now marketed as Oracle BPEL Process Manager. Full story at [InfoWorld.com]

In this outtake from our interview, Edwin pushes back against the notion that BPEL is overly complex. A lot of the complexity, he argues, has to do with XML Schema, not BPEL itself. He goes on to describe how alternate bindings -- based on WSIF and JSR 208, as well as Indigo -- will extend BPEL's reach beyond SOAP Web services to the full range of legacy protocols.

Congratulations to Scott Johnson and the rest of the Feedster gang for the launch of Feedster version 2. There are lots of new features to digest, but the ones that most interest me are those that enhance cross-blog conversation. At this URL, for example, I can find a tidy summary of the reaction to this item:

Excellent! Here's a suggestion, for what it's worth. The URL that produces that summary looks like this:

http://about.feedster.com/?id=39&epoch=1089217495

It's opaque with respect to the item that it summarizes. The reason is that Feedster summaries are by day, not by item. This means, among other things, that I can't make a Feedster version of my Technorati trackback bookmarklet that I could use to generate this kind of view with a single click, from any blog post I happen to be reading.

An example of that kind of Technorati query looks like this:

http://www.technorati.com/cosmos/search.html?url=\
  http://weblog.infoworld.com/udell/2004/07/07.html#a1035

This seems preferable to me. Notice, though, that the Technorati query yields the dreaded Ouch! No results found. It's not news that the blog world's exponential growth has been challenging to keep up with.

Of course there's a supercomputer out there that hasn't yet applied itself to this problem. I'm not the only one who wonders when, and how, it will.

Update: http://www.feedster.com/links.php?url=http%3A%2F%2Fweblog.infoworld.com%2Fudell%2F2004%2F07%2F07.html%23a1035

Here's a scenario that I've come to call "the coffee-shop problem" because it pertains to a local coffee shop, though it also applies to a home office that might receive visitors. You have a single DSL or cable connection. The challenge: offer Wi-Fi to visitors without exposing your connected computer (or LAN).

I haven't yet found a low-end (sub-$100) appliance that can do this. If you're willing to spend closer to $1000, the solution I'm testing here in my home office at the moment, Fortinet's FortiWiFi-60, solves the problem handily. You can establish firewall, anti-virus, intrusion-detection, content-filtering, and traffic-shaping policies between any pair of its WAN, LAN, DMZ, and 802.11b/802.11g interfaces. That's overkill for the coffee shop scenario, of course. And while it's entertaining for me to fiddle around with the various policies, that's very much an administrative thing, not something a non-technical user would want to do. For the coffee shop and home office scenario, I think there might be a market for a cheap appliance that would isolate a WLAN from its host LAN in a turnkey way.

For the enterprise, of course, this is a more complicated problem. In some cases, you want to isolate visitors from the local network. In other cases, you'd like to be able to collaborate with visitors and share intranet resources with them. Solutions to this problem tend to require administrative support. But that often won't correspond to the way we delegate trust in the physical world. For example, on a recent visit to a corporate campus, I signed in at the visitor center. But when the meeting moved to another building, it wasn't my visitor credentials that gave me access to that building. Rather, I piggybacked on the authorization of the employee who unlocked the door with his card. But since there was no analogous way to delegate network access (isolated or not), I spent the day out of contact with the world.

At the PKI Unlocked summit at Dartmouth College, I saw an interesting approach to solving this problem. Greenpass, one of the projects being directed by Sean Smith, is a prototype system that enables a trusted insider to delegate certificate-based access to a guest. It was set up and running in the seminar room, and it works like this:

1. On connecting to the access point, the guest is bounced to a registration page.

2. The guest uploads his or her digital certificate to Greenpass.

3. Greenpass produces an image based on the guest's public key, and displays it on the guest's laptop.

4. The guest shows the image to the delegator.

5. The delegator compares it to an image based on the key associated with the access request, and if the images match, accepts the request.

6. A SPKI/SDSI certificate is issued for the guest. (Pronounced "spooky/sudsy", these technologies support a decentralized, peer-to-peer approach. The design of Groove was influenced by SPKI/SDSI.)

7. A modfied RADIUS server accepts the SPKI/SDSI certificate.

There's plenty of rocket science under the covers, but the parts that people do -- compare images, vouch for guests -- are easy and natural. Nice!

Update: Here are some suggested solutions to the coffee-shop problem;

From Seairth Jacobs: D-Link's DSA-3100 Public/Private Hot Spot Gateway. Seairth writes: "I admit it's not sub-$100 and it doesn't provide some of the features as the Fortinet product, but it may be a good compromise. Also, because it doesn't have the wireless built in, it is possible to keep up with the latest and greatest (wireless features) without having to replace the entire device."

From Dave Megginson, Will Glass-Husain, and Eddy Carroll: A 3-box solution, two Linksys (or equivalent) routers connected in a Y configuration with a third that talks to the cable/DSL box. "I have a vague feeling there might be gremlins in this double network address translation," writes Will, "but can't think of a concrete reason it wouldn't work." (Brian Jepson also suggested this to me, a while ago.)

Today (July 14) I'll be attending PKI Unlocked, a seminar on PKI deployment at Dartmouth College.

From July 28 - 30 I'll be at OSCON 2004. And from July 31 - August 1 I'll be at the VanPy (Vancouver Python) Workshop, where I'm filling in as keynoter for a guy who knows a lot more about Python than me (though he pretends otherwise), and sharing the stage with the guy who invented the language. Gulp.

My recent mangling of Diego Doval's name in a print column was a harsh reminder that I neglect one tradition of print journalism at my peril. That tradition is a fact-checking mechanism called CQ. The idea is that an author, when writing the name of a person, company, or product, should CQ it to indicate that the spelling has been double-checked. (The acronym "CQ" is itself unCQ-able, since nobody owns the term or seems to know what it stands for.) Of course a copy editor shouldn't automatically trust an author's CQ. But it's one layer of a defense-in-depth strategy.

Here's a real-life example. In next week's column I mention a certain Microsoft initiative, now mothballed. I wasn't sure about "Hailstorm" versus "HailStorm", but found some examples (via Google) that convinced me to go with the former. Having double-checked in this way, I should have written "Hailstorm [CQ]," but didn't. My editor, who had a clear memory of the Hailstorm spelling, did CQ it that way. But in fact, the correct spelling -- thankfully caught at the last minute by an eagle-eyed copy editor -- appears to be "HailStorm."

HailStorm was originally described in a Microsoft whitepaper, now 404. The original press release, still online, uses both spellings. If you search Google or even Microsoft.com, you'll also find examples of both spellings. About the best that can be said, as my editor pointed out, is that spellings with the cap S are more frequent.

I've long been fascinated with the way in which Google can perpetuate misspellings. Compare, for example, the count of results for embarrass (count: 401,000) and embarass (count: 41,400). Obviously you shouldn't use Google as a dictionary, you should instead go here or here. But I'll bet a lot of people do look up "embarass" on Google, find evidence to support their misspellings, and thus perpetuate them. I've even wondered if there's a feedback loop here that will increase the ratio of incorrect to correct spellings over time.

Although you shouldn't use Google as a dictionary, note the difference between looking up the wrong and right spellings there:

In the latter case, Google refers you to an authoritative source -- in this case, dictionary.com. Of course, CQ-able facts usually can't be found in a dictionary. The authority that governs them is the person who owns the name in question, or the company that owns the name or product. At least, that's how it ought to be. But look at what really happens:

I own the spelling of my name. InfoWorld, as my employer, has some ownership interest in that fact too. Microsoft, even though it has 404'd the HailStorm whitepaper, still owns that piece of its institutional history. Shouldn't these responsible parties control such facts about themselves?

HailStorm, of course, was based on a mechanism for publishing machine-readable facts. There are other ways to skin the cat. FOAF, for example, is a way for individuals to assert facts about themselves. Currently Google sees 14,700 foaf.rdf files and 416 foaf.xml files -- not including mine, which I just added today. I resisted FOAF until now because I've worried about asserting things which can't be asserted, such as relationships. But the core concept of FOAF, as captured in the tagline "a Web of machine-readable homepages," is indisputably valid.

If you removed FOAF's "friend-of-a-friend" branding, the concept might make more sense to organizations. For example, the homepage of infoworld.com or microsoft.com might contain:

<link rel="dictionary" type="tbd" href="dictionary.xml">

The dictionary.xml file would assert public facts: names of employees, organizational units, products. These would reflect internal records. How would an organization mark facts in its internal databases as being both correct and releasable? In my mind's eye, I see a Web form. On the form there is a button. And the button says:

WHATWG's home page asks rhetorically: "Shouldn't this work be done at the W3C or the IETF?" And it answers: "Many of the members of this working group are active supporters and members of the W3C and other standardization bodies. We plan to submit our work for standardization to a standards body when it has reached an appropriate level of maturity." Bingo. That's how things used to work a decade ago when Web standards, and the applications built on them, formed a virtuous cycle of co-evolution.

Another sign of forward motion came from the Mozilla Foundation, which announced last week that it will modernize the long-stagnant Netscape plug-in API in collaboration with Adobe, Apple, Macromedia, Opera, and Sun Microsystems. In other words, everyone but Microsoft. While Internet Explorer sits on the sidelines, benched by Avalon, the rest of the players are creating some excitement on the field. Go, team! [Full story at InfoWorld.com]

There's more history and passion wrapped up in all this than I can begin to understand, though Tim Bray's comments at the end of last week offer some glimpses. Tim was, however, relieved to see that Safari may encapsulate its Dashboard-related extensions in a "pseudo"-namespace -- which seems entirely reasonable to me as well. And despite misgivings, he was listening to Brendan and "not hearing much to disagree with."

The bottom line, for me, is that the browser is the most powerful engine for creating and distributing software that the world has ever seen. Its birth was a messy affair, and its adolescent growth spurt -- if that's what this is -- might not be pretty either. But I'd really like to see it reach for its full potential.

The current issue of Digital ID World just arrived. While reading an article about CoreStreet, a company whose identity technologies have intrigued ¹ me for a while, I noticed something strangely missing from the article: a byline.

Flipping through the magazine, I found several bylined columns and one bylined feature, but most of the features -- hefty four-to-six-page articles on a range of identity-related topics -- are anonymous.

This would be odd in any case, but for a magazine with the tagline Identity is Center it seems downright surreal. What's up with that?

Update: Here is the explanation: anything not bylined is written by Phil Becker, the magazine's founder and editor-in-chief, who is also its most prolific author. Phil worried that it would seem egomaniacal to print his name so many times. For what it's worth, I think it's pretty cool that the chief cook can also wash all those bottles.

¹ Another identity-related bit of news, as you'll discover if you follow that link, is that older InfoWorld content (pre-2004, I believe) now requires (free) registration.

Tim Bray has thrown down the warning flag with respect to the Dashboard-related HTML extensions in the next version of Safari. "I'd be really happy if someone explained to me how this is different from what Netscape and Microsoft did to each other so irritatingly back in 1996," he writes.

Well, here's how it looks to me. In this post about Dashboard, Dave Hyatt mentions that extensions are being done "in a way that is designed to be compatible with other browsers." The linked site belongs to the Web Hypertext Application Technology Working Group, just formed last month. From the WHATWG's home page:

Shouldn't this work be done at the W3C or IETF?

Many of the members of this working group are active supporters and members of the W3C and other standardization bodies. We plan to submit our work for standardization to a standards body when it has reached an appropriate level of maturity. The current focus is rapid, open development and iteration to reach that level.

Several members of this working group attended The W3C Workshop on Web Applications and Compound Documents. The position paper submitted by Opera and Mozilla represents the fundamental principles upon which the WHAT working group intends to operate. [WHATWG]

That document, which enumerates a whole bunch of practical ways in which browsers could support better Web applications, resonates powerfully for me. Unlike in 1996, Microsoft today sees Web applications as a dead end; Internet Explorer is frozen; the wholly proprietary Avalon is their future. Meanwhile Mozilla, Safari, and Opera think they can create forward motion on Web apps, within a cooperative framework. My $0.02: go for it.

Every now and then I find myself playing the Howard Beale (Peter Finch) role in Network, across from Arthur Jensen (Ned Beatty), the capitalist visionary who explains how things really work. At Digital ID World in 2002, it was Phil Becker who channeled Arthur Jensen. During last week's Gillmor Gang show, it was Sun's Jonathan Schwartz. In the wake of JavaOne I'd been thinking tactically about technical aspects of Java. For Schwartz, though, it's all strategic and economic -- as in this sermonette on leasing and net present value. I've thought a lot about subscription businesses, and I've even helped create one, so I have a basic appreciation for the model. But I'm not qualified to evaluate Schwartz's plans for turning Sun's various assets into recurring revenue -- at least, not on financial terms.

I can, however, make some observations about the fitness of those assets for the stated purpose. Java, in particular, has always delivered the right mix of ingredients -- at least in theory. It's portable. It scales up to the cloud and down to the handset. It's a robust substrate for network services. And it can project a rich user interface onto any device. In practice, though, it's been a challenge to exploit all this goodness with the Java layer decoupled from its OS substrates. I can think of a couple of ways in which tighter integration could be useful:

• A better Java Web Start. "It's a great idea," Pito Salas blogged last week, "but disappointingly implemented." With Java applets, Sun had the first mover advantage in deploying code on demand. Lately Microsoft has been iterating toward a viable .NET solution. It's true that the ClickOnce technology in the 2.0 ("Whidbey") version of the .NET Framework won't play on as many devices as Java can. But ClickOnce will reach a lot of desktops. Java Web Start needs to do better there, and sooner rather than later.

• Stronger Solaris/Java synergy. In Solaris 10, as Sun has been pointing out recently, a single instance of the OS will be able to be virtualized into many isolated partitions. I haven't yet seen an explanation of how Java-based workloads map to those partitions, but I presume the model will be one or more JVMs per partition. Will superior intra-JVM communication be a Solaris 10 differentiator? Will a more granular mapping of applications (rather than JVMs) to partitions be possible? In the latter case, you'd need a process-like abstraction in Java -- and in fact, one is forthcoming.

As Schwartz notes, Sun's plans for the Windows desktop ran afoul of tactics for which Microsoft wound up making a "two billion dollar apology." OK, but what makes Java special on Solaris servers, or on Sun's Linux desktops for that matter? Sun asserts that Solaris is the best Java substrate, but doesn't marshall a lot of evidence. I haven't seen the same kind of argument made for Sun's version of Linux, in comparison to other Linuxes, but since the Java Desktop System doesn't run much in the way of Java software, the point's kind of moot.

It was inevitable that Java would grow more operating-system-like over time. One example is application isolation, specified in JSR 121. Another is dynamic management of the Java stack on J2ME devices, described in JSR 232 and demonstrated by Nokia at JavaOne. In this Real clip ¹ from the twenty-minute concept video, we see Java components deployed to a network of Nokia Communicators, and then remotely managed. This is no doubt a great thing for the world of handsets. But desktop and server operating systems have their own highly-evolved management methods, to which Java is somewhat orthogonal. If Sun's own operating systems are going help create the new economic world order that Schwartz envisions, maybe they and Java should find ways to work more closely together.

¹ Accessing this clip, by the way, was no mean feat. Sun's video URLs are even more elusive than Microsoft's.

In next week's InfoWorld column, I quote Diego Doval, CTO of clevercactus. Or rather, I meant to quote him. For reasons that escape me, I attributed Diego's remarks to Diego Rivera, the famous Mexican muralist. I have no earthly idea how I managed to transpose the living computer scientist and the dead artist. Since Diego Rivera won't be reading this, I'll direct my apology to Diego Doval. For the record:

Diego Doval I am co-founder and CTO of clevercactus ltd. I submitted my PhD thesis (in the area of self-organizing networks) last year to Trinity College Dublin, Ireland. I was previously a teaching assistant at TCD's Computer Science department. I graduated from Drexel University in Philadelphia, PA where I did research in the Software Engineering Research Group. I worked at Fuego Corp. before going to the US, then after graduation I was a Research Associate in the Personal Systems Group at IBM's TJ Watson Research Center in Yorktown Heights, New York, and later at Mindstech International in Silicon Valley.

Diego Rivera DIEGO RIVERA (1886-l957), muralist painter, was one of the greatest artists in the XXth century. Born in Guanajuato Mexico, in 1892 he moved to Mexico City with his family. He studied in the San Carlos Academy and in the carving workshop of artist José Guadalupe Posada, whose influence was decisive.

A couple of months ago I spoke with Jeffrey Snover, who is the architect of MSH (aka Monad), Microsoft's new object-oriented command shell. At the time, I didn't get to see a demo. Yesterday, Chris Sells pointed to the episode of the .NET show that includes a Monad demo by Snover and Jim Truher. Sells also notes that the beta of Monad is available for XP and Server 2003, so I've registered for the download. The concept is wonderful: a Unix-like shell where the stuff that gets piped around is self-describing, either in the form of .NET objects or their XML serializations. Although it targets "the Longhorn wave," I'll be curious to see what Monad can do on current Windows OSs.

I was hoping to use Rich Persaud's AV clipping service to point to some interesting parts of that Monad demo. That service, which inspired the experimental MP3 clipping service I tried on Tuesday, also enabled me to quote from the Apple WWDC keynote on Wednesday. To form URLs that quote from Real, QuickTime, and Windows Media streams, you just need the URL of the stream. Which, in the case of MSDN broadcasts, is either hard or impossible to find.

Here's the home page for the episode of the .NET show that includes the Monad demo. Here's the URL behind the "Watch it now!" button. Here's the XML manifest embedded in that URL. And here, from deep inside that file, is a reference to the actual .WMV file:

<mediaVideo identifier="060EDE76_49D9_423B_8DA3_D6DB5039745E" 
  xlinkHref="netsow43_mbr.wmv" xlinkActuate="onLoad" xlinkRole="ecrs">

Now, MSDN does an awesome job with its webcasts. If you access them from IE -- which, unfortunately, is the only way you can access them -- you'll find that the transcripts are linked to the video with exquisite care, like so:

<div id="p01:12:23"><img src="/seminar/shared/images/playsync_stat.gif" 
 alt="Jump to #85" hspace="2" border="0" align="absmiddle" class="PrintNever"
 onclick="callSeek('01:12:23');" onmousedown="downPlaySync(this)" 
 onmouseout="resetPlaySync(this)" onmouseover="this.style.cursor='hand'; 
 togglePlaySync(this);" />
<b>JEFFREY SNOVER:</b>  Yeah.  So now let's focus in on even a more 
sophisticated example where again you have the MSH do more work for you.  
This is the code to stop a process.  Again it's a class, again you put 
the commandlet attribute on top of it and here we have a public int, 
array of integers, called ID.  So we're going to kill processes by 
their process ID and we've got attributes on top of it.  
<div class="code"><pre>
[Cmdlet("stop", "ps1")]
public class StopPs1: Cmdlet
{
	[Parameter( 
		Mandatory = true, 
		PipelineInput = PipelineInput.ByMatchingProperty, 
		Position = 0)]
	[Prompt("Where's the ID dude?")]
	public int [] Id;
	public override void ProcessRecord()
	{
...
</pre></div>

This is incredibly well done. And yet, the entire presentation is hermetically sealed. From the outside, there's only a single IE-accessible entry point. Conspiracy theorists will doubtless find evil here. I don't. If MSDN wanted to assert total control over this content, it wouldn't offer downloads:

Offline Viewing Download: For those of you who want to download a copy of this episode to your local hard drive for off-line viewing, we provide this as a separate file (self-extracting .exe) that you can download. We now offer two file size choices, depending on the bandwidth of your Internet connection and a third one especially for mobile devices.

300 KB version (246 MB file)
100 KB version (87 MB file)
Mobile devices version (78 MB file)

What is evident, though, is a cultural reluctance to work with the Web on its own terms. MSDN's predilection for publishing URLS that point to self-extracting .EXEs, rather than (in this case) to .WMV files, is really quite odd. My advice: point to the .WMVs too. You've already invested a huge amount of effort in this stuff. The content is intended to be public, and its purpose is to evangelize. So, why not trust the Web and let it help you do that? If you make the URLs directly available, here are some of the positive effects that can ensue:

• A blogger could point directly to one of your timecoded fragments, or use an AV clipping service to point to a newly-constructed fragment.

• A Firefox user on Mac OS X could access the content. You don't want to just preach to the converted, do you?

• A transcoding service could (in theory) make the video accessible in non-Windows-Media formats.

Update: Ace detective Alf Eaton took up the challenge, and has extracted the URL I was looking for. The trick is to hit the "Watch this video" URL with a user agent that pretends to be IE but isn't. Alf did that using the debug feature of Safari. Armed with this insight, I was able to do the same thing with the command-line tool curl:

curl -A "Mozilla/4.0 (compatible; MSIE 5.5)" \
  http://msdn.microsoft.com/seminar/shared/asp/view.asp?\
  url=/theshow/en/episode043/manifest.xml 

which, as Alf discovered, yields:

/theshow/en/episode043/netsow43_mbr.asx

which in turn yields:

mms://wm.microsoft.com/ms/seminar/en/Episode043/netsow43_mbr.wmv

Thanks Alf! Now, where was I? Oh, yeah, I wanted to highlight a couple of things:

• navigating from objects to subobjects

• piping output to Excel

• data coercion in the pipeline

• errors as collections of first-class objects

• globbing and wildcarding alternate namespaces, with tab completion

Scalable vector graphics and animation are two of the hallmark features of Macromedia's nearly ubiquitous multimedia player. Yet the company has done a poor job of creating -- or convincing third-party developers to create -- components that make it routine for people to work with spatial and temporal data. And in the recent push to legitimize Flash as a rich-client platform, the company has de-emphasized what is at the core of every Flash movie: its timeline.

It's a hard sell, admittedly. Microsoft is also having a tough time articulating the business case for the scalable vector graphics, 3-D, and animation capabilities it's building into Avalon, the next-generation Windows graphics subsystem. My advice? Stop worshipping the raw power of next year's graphics processing unit, and start showing developers concrete ways to help users deal with their four-dimensional data. [Full story at InfoWorld.com]

We live in an age of innumeracy. The "chartoon" style of graphics that Nigel Holmes invented at Time has now, to my dismay, begun to cheapen the editorial page of the New York Times. Holmes' arch-nemesis Edward Tufte, whom Salon aptly describes as a data artist, sets the bar for precise, intelligent, meaningful visualization of data. We don't get nearly enough of that, and it's not for lack of GPU horsepower or elegant APIs. The gating factor is that you can't bottle and sell the Tuftean sensibility. Still, we can try. I'd like to see Apple or Macromedia or Microsoft put Tufte (or someone who thinks like him) in charge of a Manhattan program to produce a new breed of display widgets and data-wrangling wizards.

I was recently shown a stunning visualization of sales data based on the open source Java Treemap Viewer (background). Like the DateLens viewer I mention in this week's column, the Treemap viewer derives from pioneering work at the University of Maryland's Human-Computer Interaction Lab. I can't show you the actual visualization I saw because it's proprietary, but here's a demo. This technique has been around for years. Of the real-life data sets that could be productively visualized this way, though, I'll wager that few are. I've got a hunch there are a bunch of other techniques that are languishing in research labs too. The industry's challenge is to dig them up, refine them, and deliver them to developers and end users in ways that will really improve our data-driven communication.

Update: Andrew Duncan wrote to remind me that I omitted another WWDC graphics demo: Aran Anderson's stunning Orbit satellite simulator. "If you need a non-trivial justification for all that GPU goodness," he asked, "wouldn't Orbit qualify?" That's a great point, thanks Andrew. An awesome app, indeed. As I mentioned in my column, scientific visualization has always been a voracious consumer of GPU cycles, but it has also tended to live in its own sci-viz ghetto. Now it's time for this stuff to break out into the world of mainstream business data.

During the June 18 Gillmor Gang show, I asked Hummer Winblad's Mitchell Kertzman about open source business models. Kertzman said ¹ that the key factor, from his perspective, is the way in which the open source stack frees commercial software companies from the burden of "dragging around an expensive platform." He also questioned the need ² for the JVM, citing two reasons. First, that Java's portability has become a non-issue now that there are only two platforms that matter: .NET and Linux. Second, that the rise of XML Web services has given a boost to the text-savvy scripting languages: Perl/Python/PHP, the "P" in LAMP.

At that point something clicked in my head, and I proposed ³ a software taxonomy based entirely on virtual machines -- the VB runtime, the CLR, the JVM, the Perl and Python VMs. Some of these are bound more tightly to operating systems than others, some are bound more tightly to programming languages than others, but they all share a set of common characteristics. The definition of a modern "software platform," I would say, is a VM and its associated class libraries. And a bunch of implications flow from that.

Here's one. In last Friday's item on automated code analysis, I forgot to mention that the growing reliance on VMs is become a key enabler of a new breed of tools that enhance software quality. From Greg Wilson's blog:

One of the most important features of the "New Standard Model" of programming is its emphasis on unit testing. Just five years after the first version of JUnit was written, an ever-increasing number of programmers actually create and run tests as a matter of course.

But writing tests by hand is still tedious, and still requires a fair degree of programming skill. Enter Li and Wu's new book. Over the course of twelve detailed (and sometimes rather intense) chapters, the authors explain how to build a higher-level testing tool for .NET programs using:

• reflection, to find and call the methods being tested;
• CodeDOM, to generate testing code from specifications; and
• Excel, as a user interface.

[Helium: Greg Wilson]

VMs still aren't completely viable on the client side, so a lot of what's becoming possible hasn't really sunk in, but that's about to change. Eclipse runs on the JVM, Chandler runs on Python, various things run on the CLR (and Mono), Longhorn apps will run on the CLR. One way or another, your platform will be a VM. Its capabilities, class libraries, OS bindings, and language bindings will matter more to you than the underlying OS or language.

¹ This is an experimental MP3 clipping service. Alternatively (i.e., if I break it) you can just go to the broadcast and play 9:21 to 10:25.

² 15:08 - 15:59

³ 21:13 - 23:35

Back in January I mentioned Agitar Software in a column on software testing. The backstory was that Agitar got in touch with me after reading my review of Compuware's DevPartner Studio. I had used NLucene, the .NET port of the Java-based Lucene search engine, as a benchmark to explore that product's debugging and source-code analysis features. Agitar's development lead, Kent Mitchell, picked up on the idea. He fed Lucene's Java sources into his test automation tool, Agitator, and used Lucene to demonstrate his product.

Today Agitar's Mark de Visser pointed me to this interesting experiment. It's a set of test coverage reports for Agitar's own product plus some open source Java projects including Ant, Berkeley DB, Cocoon, and Lucene. What exactly these reports mean is open to interpretation, as Agitar points out. Note also that the Agitar is a special case, since the company has been dogfooding its own tool. While "agitation" of arbitrary code can automatically produce a bunch of tests, they're not really mean to be used without human oversight. CTO Alberto Savoia puts it this way:

Agitator can greatly accelerate the development and thoroughness of unit tests by automating most of the activities that don't require human understanding, intelligence, and creativity, but you still need to invest time and thought to direct the automation and to make sure the results the results are correct, robust, and maintainable. [Developer Testing: Eating our own dogfood]

The meta-theme I find interesting here is the virtuous cycle involving open source codebases and a new breed of static and dynamic code analysis tools. Another example: Coverity's Linux bugs database (registration required, see this Linux Magazine article by Benjamin Chelf for background).

To Eric Raymond's famous dictum -- "Given enough eyeballs, all bugs are shallow" -- perhaps we should now add: "Given enough code to study, the eyeballs will be fitted with increasingly powerful spectacles."

Somebody asked me today why Windows XP doesn't have something like Mac OS X's Keychain: a secure, systemwide store for names and passwords. And then I remembered, dimly, that it does -- sort of. When XP came out, all the feature lists mentioned Credential Manager, which uses the Windows Data Protection API (DPAPI) to do something that sounds just like what the OS X Keychain does. On XP, you get to the Credential Manager like so: Control Panel -> User Accounts -> Manage my Network Passwords.

It seems bizarre that I could have forgotten all about this. But then again, perhaps not. When I looked at the Stored Usernames and Passwords list, I found nothing there except for my Passport account. No FTP sites, websites, email accounts, or SSH accounts. Nor was I able to add such accounts using the GUI. Digging a bit deeper, I learned that Windows Server 2003's cmdkey can be transplanted to XP, where it can be used to list and add credentials. Using cmdkey I was able to add a Web account by specifying the "generic" type -- as opposed to the default, which is the domain. But IE still paid no attention. Its credential memory is apparently unrelated to Credential Manager. Who knew? Not me, anyway.

Next I went back to double-check the OS X situation. In Keychain Access, I found FTP sites, SSH accounts, and certificates, but no websites. How come? Oh, Firefox. I haven't used Safari in ages. Firefox evidently talks neither to Credential Manager on Windows nor to Keychain on OS X. But while Windows' native browser, IE, doesn't talk to the systemwide credential store, OS X's native browser, Safari, does. When I told Safari to remember credentials for a secure website, they showed up in Keychain Access. (Apparently Camino supports the Keychain too.)

Weird, eh? Some parting questions:

1. Does IE really not use DPAPI to store non-Passport Internet credentials, and if not, why not?

2. Will XP SP2 make any changes in this area?

3. What would it take for a cross-platform app, say Firefox, to support both Credential Manager on Windows and Keychain on OS X?

4. Do any existing apps do both?

Update: Ari Pernick spells out the situation, which is a bit complex. Briefly, WinInet uses DPAPI for NTLM/Kerberos, but uses PStore for basic and digest authentication. He writes:

Pstore doesn't do as good of a job of protecting credentials as the Data Protection and Credential Management APIs do and as the warning on the API documentation suggests, it is likely to change or go away in Longhorn. In that timeframe WinInet will switch to use the better APIs for those types of credentials. As for Udell's question #1, which asks why we don't use the better APIs to store basic and digest authentication, my best guess is that the credential manager wasn't really made to hold that type of credential well (you can't input them from the GUI UI). And to answer question #2, this hasn't changed in Windows XP SP2.

Even with the planned changes I referred to, you are still a far cry from centralized credential management that includes all web credentials. The credentials in the better store may still not show up in the GUI and forms based authentication is a completely different beast altogether. Sounds like a nice feature to integrate all of those in one GUI for a user, and maybe an IE or a security pm will hear the call and make it so, especially if the users ask for it. [Ari Pernick: WebTransports: Where to put the credentials?]

On the Google PC, you wouldn't need third-party add-ons to index and search your local files, e-mail, and instant messages. It would just happen. The voracious spider wouldn't stop there, though. The next piece of low-hanging fruit would be the Web pages you visit. These too would be stored, indexed, and made searchable. More ambitiously, the spider would record all your screen activity along with the underlying event streams. Even more ambitiously, it would record phone conversations, convert speech to text, and index that text. Although speech-to-text is a notoriously imperfect art, even imperfect results can support useful search. [InfoWorld.com]

It seems clear, though, that whatever can be accomplished by means of what I've come to call "managed metadata," we'll always want that Google effect to be happening in parallel. When asked about the Semantic Web and RDF at InfoWorld's 2002 CTO Forum, Sergey Brin said:

Look, putting angle brackets around things is not a technology, by itself. I'd rather make progress by having computers understand what humans write, than by forcing humans to write in ways computers can understand.

Yesterday, one of my DSL providers ran afoul of a backhoe which severed its OC3. The bad news was that a bunch of customers, me included, learned that we had no redundant path to the backbone -- at least not through this provider. (This is one reason why I maintain a separate circuit through a different provider; that one was unaffected.) The good news was that the fiber got spliced together very quickly, and the provider was really, really sorry and really, really proactive. I got calls from three people alerting me to the outage, and calls from four other people notifying me that it was cleared. In a situation like that, there's no such thing as overcommunicating.

Hence this note. If you haven't heard from me in a few days but think you should have, it's not because of that fiber cut. Apparently my home mail server, to which my InfoWorld mail is forwarded, tightened up its reverse DNS lookup policy. It could resolve the domain, but not the specific hostname/domain. That's been corrected now on our end (thanks, Kevin), and I hope the queued messages will transfer today.

Last week Tim Bray wrote about his (and Sun's) involvement in the European Commission's investigation into the OpenOffice and Microsoft flavors of XML office documents. The upshot:

You can find the Committee's conclusions here; they're short, readable, and defy summarization. [ongoing]

Therefore, it is recommended that:

1. The OASIS Technical Committee considers whether there is a need and opportunity for extending the emerging OASIS Open Document Format to allow for custom-defined schemas;
2. Industry actors not currently involved with the OASIS Open Document Format consider participating in the standardisation process in order to encourage a wider industry consensus around the format;
3. Submission of the emerging OASIS Open Document Format to an official standardisation organisation such as ISO is considered;
4. Microsoft considers issuing a public commitment to publish and provide non-discriminatory access to future versions of its WordML specifications;
5. Microsoft should consider the merits of submitting XML formats to an international standards body of their choice;
6. Microsoft assesses the possibility of excluding non-XML formatted components from WordML documents;
7. Industry is encouraged to provide filters that allow documents based on the WordML specifications and the emerging OASIS Open Document Format to be read and written to other applications whilst maintaining a maximum degree of faithfulness to content, structure and presentation. These filters should be made available for all products;
8. Industry is encouraged to provide the appropriate tools and services to allow the public sector to consider feasibility and costs of a transformation of its documents to XML-based formats;
9. The public sector is encouraged to provide its information through several formats. Where by choice or circumstance only a single revisable document format can be used this should be for a format around which there is industry consensus, as demonstrated by the format's adoption as a standard.

The next day I received a note from somebody at Waggener-Edstrom, Microsoft's public relations firm, pointing to and summarizing this open letter from Jean Paoli. Both notes -- that is, the PR rep's and Paoli's -- stress point #1: that support for user-defined schemas, which Office 2003 alone offers, is a big deal. I agree. Neither note directly addresses points #4 ¹, #5, or #6. And neither cites the original report, though the Office XML home page, which the Paoli letter points to, does point to the European Commission's wrapper page. And it, in turn, points to:

• the recommendations
• the full "Valoris" report (78-page PDF)
• Microsoft's comments
• Sun's comments

I'm citing those URLs here partly for my own future reference, and partly to try to attract attention to a subject that's important, complex, and warrants a lot more discussion and commentary. Just now, with the Valoris report loaded into my browser, I clicked my Technorati talkback bookmarklet -- which in this case resolves to this lookup, and found only this comment from Stephen McGibbon. Meanwhile, Feedster comes up blank for Valoris report.

Open document formats are a big deal. Here's hoping that the next time I issue those queries, more will turn up.

¹ Note, however, that the Office XML home page calls out the FAQ which "has been recently updated with information regarding the perpetual nature of the program, patent grants, and more."

Jeremy Mazner is asking some great questions:

• Q: Does a quick blog entry meet the same standards and go through the same background and vetting process as a "real" story?

  A: Many (though not all) of the items I post here are as carefully written as what goes into print. None are is ¹ edited by anybody but me. None are vetted by anyone at InfoWorld, but all can be vetted by everybody who chooses to comment.

• Q: Is a blog entry equally as obligated to represent both sides of a controversy, or is it expected to only represent the journalist's point of view?

  A: For the magazine, I write features and reviews and columns. All are expected to be fair. The story types exist along a spectrum ranging from less to more personal. The blog lives at the personal end of the spectrum.

• Q: Are blogs supposed to be more of a conversation -- and if so, should they always have comments enabled?

  A: I think blogs can't help but be a conversation. As to comments, after years of doing Web forums and discussions, I'm experimenting with taking a break from flames and spam. I'd like to think that the blogosphere's less tightly-coupled "discussions" -- mediated by logs and search engines -- delivers better signal-to-noise with less psychic strain. That said, I do miss direct comments, I do use them selectively, and I may try renabling them.

Jeremy's questions were motivated by a series of questions I've been asking about Longhorn. This is part of a strategy I've been using -- since the pre-blog era, in fact, when my medium of choice was NNTP -- to deepen the stories I research for magazines. When the subject is not a secret, I find it extremely helpful to raise some issues publicly and invite a range of interested parties to react to them. A recent example was this entry in support of this story.

In that spirit, I owe Jeremy a response to his questions about my take on WinFS. He asks: "What is an 'XML-centric database' anyway?" A good example of the basic idea -- and the one I've been working with -- is Berkeley DB XML (which has also been adopted by the Chandler project). DB XML supports indexed XPath search, a poweful capability that's now being woven into both RDBMSs with XML support, and "native" XML databases. An even more powerful standard is XQuery, which though not a final recommendation is implemented provisionally in both conventional RDBMSs and native-XML dbs.

We have standard query languages (XPath, XQuery), and standard ways of writing schemas (XSD, Relax), and applications (Office 2003) that with herculean effort have been adapted to work with these query and schema languages, and free-text search further enhancing all this goodness. Strategically, why not build directly on top of these foundations?

Tactically, why do I want to write code like this:

public class Person
  {
  [XmlAttribute()] public string Title;
  [XmlAttribute()] public string FirstName;
  [XmlAttribute()] public string MiddleName;
  [XmlAttribute()] public string LastName;
  ....

<People>
  <Person
    DisplayName="Woodgrove Bank"
    IMAddress="Support@woodgrovebank.com"
    UserTile=".\user_tiles\Adventure Works.jpg">
    <EmailAddresses>
        <EmailAddress
            Type="Work"
            Address="mortgage@woodgrovebank.com"/>
        <EmailAddress
            Type="Primary"
            Address="Support@woodgrovebank.com"/>
   </EmailAddresses>

I believe two things to be true. First, we have some great XML-oriented data management technologies. Second, the ambitious goals of WinFS cannot be met solely with those technologies. I'm trying to spell out where the line is being drawn between interop and functionality, and why, and what that will mean for users, developers, and enterprises.

¹ David Clarke, of CapeClear, points out that "this statement, precisely by virtue of its obvious lack of sub-editing ('are', not 'is'), re-inforces the very point it seeks to make!" Delightful! As David mentioned to me in email, there ought to be a word for this reflexive case.

Current approaches to taking browsers offline typically enqueue messages that later update a server-based data model. An Alchemy application, though, always works with a genuine local data model that it stores as sets of XML fragments and navigates in a relational style. Bosworth's hunch is that a Web-style thin client, driven by a rich data model intelligently synchronized with the services cloud, could do most of what we really need -- both offline and online. Nothing prevents Java, .Net, and Flash clients from adopting the same strategy, by the way. But if Bosworth is right, the universal client that we know and love could get a new lease on life. [Full story at InfoWorld.com]

In the story as printed, this sentence:

BEA's Alchemy relies on a server component for the same reason that Macromedia's Flex does: both companies want to sell servers.

BEA's Alchemy relies on a server component for the same reason that Macromedia's Flex does.

Clustered People who build services tend to assume they don't know who's going to use them, and how often.

Metadata-driven So you can change the behavior without recompiling and redeploying the code.

Asynchronous We're still trying to convince the industry of this, but it's a lot better if you do this asynchronously, because a lot of time the thing you're trying to talk to can't respond right away, either because it wasn't written to handle the load or because the thing you're asking it to do takes time.

Intermediated The problem is that your credit approval service got bought by BFA and they consolidated the thing, so now it's a different address with a different message, and you don't want to redeploy your app. So you want everything to go through some fabric that is essentially modifiable. Call these things intermediaries, enterprise service buses, fabrics, I don't care, but you need one of these things. We've announced one called QuickSilver, BlueTitan does a nice job with this, there's Confluent...

Nothing controversial here. But there are wildly different approaches to the construction of the client-side systems that we'll attach to this infrastructure. Microsoft's Longhorn must try to extend the Windows franchise. BEA's Alchemy is free to extend the Web. This isn't an either-or deal, of course. Both strategies can succeed and co-exist. It helps, though, that the Web has found a powerful new ally.

Tim Bray points to Sun's John Clingan who asks the important question (in English, not Latin): Who analyzes the analysts? This bit caught my eye:

I remember back in ~1990 when Windows NT was being talked about taking over the world. My girlfriend at the time (now my wife) saw it on a magazine rack and said "I saw a Byte magazine cover which said, 'Is Unix Dead?'". "Uh oh, are you going to have a job next year?" Ironically, Byte magazine is dead (although byte.com is still around). Is this the enforcement of accountability for journalists and analysts? [John Clingan]

That 1990 BYTE story, by the way, makes for an interesting re-read. Some backstory: my pals Tom Yager and Ben Smith wrote it, and all three of us objected to the sensationalistic headline and its hand-wringing subhead ("As Unix faces the stiffest competition of its long life--Windows NT--can it survive?"). These came from the editorial packagers, not from the writers. And naturally, they're all anyone remembers now. I found a copy of the article on the BYTE CD-ROM, and at this late date I don't think anyone will begrudge my posting it. So, back from the dead, here is BYTE's 1990 Is Unix Dead?. It contains some gems:

Reports greatly exaggerated:

Despite its problems, Unix is not dead; in fact, it's surprisingly healthy.

Imagining OS X:

Improving Unix is much on the minds of Unix vendors. "If you have an X-based desktop with Mac-like features, the end user won't care that Unix is underneath," says Ken Arnold, an engineer at HP's Distributed Object Computing Program. As base-level machines get more powerful, they can better run the larger Unix operating systems. Then, to the end user, it is simply a matter of what off-the-shelf applications are available.

Avadis Tevanian, director of System Software at Next, agrees. He envisions a GUI that can run productivity applications side-by-side with user-made custom applications. "To get up to millions of units, you have to get rid of [the Unix shell]," he says.

The Sun factor:

Solaris 2.0, a derivative of SVR4, is going to be the acid test for Sun spinoff SunSoft. It remains to be seen whether the software arm of a hardware vendor is truly willing to create a level playing field. Sun is trying to set itself up with a virtual monopoly on SPARC operating systems and, through SunSoft and Solaris 2.0, is planning to extend its reach into the realm of high-end PCs.

Novell's first Linux:

While NextStep will be one of the contenders for the high-end multitasking desktop, it appears that the fiercest salvo fired at NT will come from an unlikely alliance: Univel. USL, looking to get serious about marketing and distribution, and Novell, hoping to shed some of its proprietary image in the newly competitive climate, have joined forces to offer a new shrink-wrapped Unix operating system that may be available as early as this fall. Sold as SVR4.2 by USL and as UnixWare by Univel, it has a list of promises at least as long as NT's.

Pretty good story, on the whole. We'll never know how many more magazines that ill-fated headline sold, but clearly, it wasn't a winning strategy.

The computing landscape back then sounds oddly familiar. In many ways things have progressed more slowly than I'd have imagined. But the analyst/practitioner ecosystem is refreshingly new. "Who analyzes the analysts?" You do.

Back in April, we ran a wildly ambitious story by Sean McCown. Entitled Databases Flex their XML, it compared the XML features of DB2, SQL Server, Oracle, and Sybase -- and also made an excursion into Yukon territory. (My contribution was the speculative sidebar on the future of native XML database technology.) Yesterday Microsoft's Michael Rys, a database architect and a co-author of XQuery from the Experts, blogged a lengthy and thoughtful response to Sean's analysis.

To frame his response, Michael develops a taxonomy of XML structures and storage models and says:

It should be clear, that by making this distinction, the terms "shredding," "unstructured," and "structured" are confusing. XML's structure can be highly structured, semi-structured or markup-structured, but it is always structured. And either of these formats can be stored in a way to provide relational, InfoSet or textual fidelity using either relational or blob storage. [Michael Rys]

That's the kind of useful clarification that Michael has been consistently delivering on his blog. I hope this thread will continue. Sean's article was -- as Michael acknowledges -- as good a comparative piece as has ever appeared in the press. But the topic is huge, and will fuel ongoing discussion. We're living through an epochal moment in the history of the industry. The hybridization of SQL and XML will deeply transform the philosophy and practice of data management in ways that I think none of us fully understands. The story will emerge from conversations between practitioner/analysts like Sean, and architects like Michael. Happily, the online realm has become a pretty good place to have those conversations.

My decade-old LaserJet 4 recently developed a bad case of the dreaded "accordian paper jam" syndrome. It's been a workhorse. Maybe, I thought, I should just put it out to pasture. But I had a hunch that the process of getting it fixed would be interestingly different from the last time I had to do something like this. And sure enough, it was. I found several repair kits online, but zeroed in on FixYourOwnPrinter.com because their kit includes a video that illustrates the process.

Here's 45 seconds from my favorite scene¹, which demonstrates the right way to remove the clip from the end of a roller. I, of course, did it the wrong way. "Be careful not to lose these e-clips, they're easy to pop off," the guy said, just as my e-clip took the leap of faith. That was the only mishap, though. The printer's fixed, and I've joined the ranks of FixYourOwnPrinter.com's satisfied customers.

The video isn't going to win any production awards. It's handheld, and not always in focus. But it was plenty good enough to walk me through a complicated procedure that couldn't have been communicated as effectively in any other way. And because it didn't need to be better than that, it was doable for some folks whose business is printer repair, not video production.

¹ Courtesy of Blue Pacific's Turbine Video Encoder. I've been wanting to standardize on Flash as a universal no-hassle video playback format. Turbine, an encoder for Flash video, is a $39 product. And it offers a free version (which I've used here) that's unrestricted except for a subtle watermark. Looks like a nice solution.

The slide shown at the right comes from a presentation entitled Windows client roadmap, given last month to the International .NET Association (INETA). When I see slides like this, I always want to change the word "How" to "Why" -- so, in this case, the question would become "Why do I have to pick between Windows Forms and Avalon?" Similarly, MSDN's Channel 9 ran a video clip of Joe Beda, from the Avalon team, entitled How should developers prepare for Longhorn/Avalon? that, at least for me, begs the question "Why should developers prepare for Longhorn/Avalon?"

I've been looking at decision trees like the one shown in this slide for more than a decade. It's always the same yellow-on-blue PowerPoint template, and always the same message: here's how to manage your investment in current Windows technologies while preparing to assimilate the new stuff. For platform junkies, the internal logic can be compelling. The INETA presentation shows, for example, how it'll be possible to use XAML to write WinForms apps that host combinations of WinForms and Avalon components, or to write Avalon apps that host either or both style of component. Cool! But...huh? Listen to how Joe Beda frames the "rich vs. reach" debate:

Avalon will be supplanting WinForms, but WinForms is more reach than it is rich. It's the reach versus rich thing, and in some ways there's a spectrum. If you write an ASP.NET thing and deploy via the browser, that's really reach. If you write a WinForms app, you can go down to Win98, I believe. Avalon's going to be Longhorn only.

So developers are invited to classify degrees of reach -- not only with respect to the Web, but even within Windows -- and to code accordingly. What's more, they're invited to consider WinForms, the post-MFC (Microsoft Foundation Classes) GUI framework in the .NET Framework, as "reachier" than Avalon. That's true by definition since Avalon's not here yet, but bizarre given that mainstream Windows developers can't yet regard .NET as a ubiquitous foundation, even though many would like to.

Beda recommends that developers isolate business logic and data-intensive stuff from the visual stuff -- which is always smart, of course -- and goes on to sketch an incremental plan for retrofitting Avalon goodness into existing apps. He concludes:

Avalon, and Longhorn in general, is Microsoft's stake in the ground, saying that we believe power on your desktop, locally sitting there doing cool stuff, is here to stay. We're investing on the desktop, we think it's a good place to be, and we hope we're going to start a wave of excitement leveraging all these new technologies that we're building.

It's not every decade that the Windows presentation subsystem gets a complete overhaul. As a matter of fact, it's never happened before. Avalon will retire the hodge-podge of DLLs that began with 16-bit Windows, and were carried forward (with accretion) to XP and Server 2003. It will replace this whole edifice with a new one that aims to unify three formerly distinct modes: the document, the user interface, and audio-visual media. This is a great idea, and it's a big deal. If you're a developer writing a Windows application that needs to deliver maximum consumer appeal three or four years from now, this is a wave you won't want to miss. But if you're an enterprise that will have to buy or build such applications, deploy them, and manage them, you'll want to know things like:

• How much fragmentation can my developers and users tolerate within the Windows platform, never mind across platforms?

• Will I be able to remote the Avalon GUI using Terminal Services and Citrix?

• Is there any way to invest in Avalon without stealing resources from the Web and mobile stuff that I still have to support?

Then again, why even bother to ask these questions? It's not enough to believe that the return of rich-client technology will deliver compelling business benefits. (Which, by the way, I think it will.) You'd also have to be shown that Microsoft's brand of rich-client technology will trump all the platform-neutral variations. Perhaps such a case can be made, but the concept demos shown so far don't do so convincingly. The Amazon demo at the Longhorn PDC (Professional Developers Conference) was indeed cool, but you can see similar stuff happening in Laszlo, Flex, and other RIA (rich Internet application) environments today. Not, admittedly, with the same 3D effects. But if enterprises are going to head down a path that entails more Windows lock-in, Microsoft will have to combat the perception that the 3D stuff is gratuitous eye candy, and show order-of-magnitude improvements in users' ability to absorb and interact with information-rich services.

Zope Corp.'s layered strategy of engagement with open source and visible-source communities is a compelling blend of the strengths of free and commercial software development. In two previous columns, Open source citizenship and Giving back to open source, I explored the tendency of enterprises to fork open source projects rather than join them. Pedhazur suggests that a commercial entity supporting both an open source base and a visible-source layered product can reduce the need to fork. By outsourcing code enhancements, the argument goes, an enterprise can enjoy single-throat-to-choke control without seceding from a project's community. It remains to be seen how broadly this model can apply, but in cases where it does, what's not to like? [Full story at InfoWorld.com]

My hunch is we'll see more of this kind of thing as open source continues to climb up the stack and encroach on the business layer. The visible-source gated community is a particularly interesting construct in light of the Nicholas Carr argument that a lot of IT is shifting from competitive advantage to cost of doing business. In an environment of growing "co-opetition," the visible-source model can pool dollars and intellectual capital in a way that drives down cost for everyone without favoring anyone. Meanwhile it's a great opportunity for the business that manages the relationship between two worlds: the open-source product with its user/developer community, and the visible-source product with its user/developer/customer community.

In the first installment of this series of questions about Longhorn, I concluded that the compelling benefit of WinFS must lie in the realm of "organizing stuff" rather than just "finding stuff" -- else why not just leverage existing and well-understood relational, free-text, and XML search methods? And I posited that the signature feature of WinFS -- "relationships" -- must be powerful enough to justify the creation of a proprietary new storage model that will enable (but also require) new applications and developer skills. Admittedly my "finding versus organizing" distinction was a bit of a cheat, since finding depends sensitively on prior organization. Except when it doesn't: brute-force free-text search routinely trumps navigation and structured search. But OK, we've all got to hope that better organization, someday, will level the playing field.

Today's personal information systems are organized hierarchically. WinFS proposes that they be organized semantically. A number of observers have noted a family resemblance between RDF (Resource Description Framework) "triples" and WinFS relationships. An RDF triple, in geek-speak, is a subject-predicate-object relation. Sets of RDF triples can be (and Semantic Web people say must be) used to represent and organize knowledge. Microsoft blogger Joshua Allen explicitly connects the dots between RDF/SemWeb and WinFS:

WinFS is going to enable numerous application scenarios that simply are not practical to implement with today's technology. WinFS is not based on RDF, of course, but they both share similar data models. And, while the scope of WinFS is local and "Semantic Web" is global, the scenarios are not that different. When you start to imagine what it would be like to extend WinFS stores to publish and synchronize data with one another, or alternately imagine a "personal semantic web," you can begin to see that the visions have some serious overlap. [Joshua Allen]

Although this stuff can get dangerously abstract, it's easy to state the practical benefit. If my personal information store contains items of types Person, Organization, Project, and Document, and if it knows about relationship types like Employment and Authorship, then I can easily answer questions like "Which Project X documents were written by Doug?" or "Which Project Y documents were written by employees of organization Z?"

Not everybody buys into the triples-oriented data model. Among them is another Microsoft blogger, Dare Obasanjo, who writes:

It seems that the point being argued is that with RDF you can get more understanding of the information in the document than with just XML. Being that one could consider RDF as just a logical model layered on top of an XML document (e.g. RDF/XML) I find it hard to understand how viewing some XML document through RDF colored glasses buys one so much more understanding of the data. [Dare Obasanjo]

The lesson, of course, is that real-world information is chaotic. In any but the smallest "proof of concept" systems, the best that one can hope for is to be able to recognize small pockets of structure within a sea of otherwise unstructured information. [Joshua Allen]

Maybe it depends how you construe "small pockets of structure." I've been getting decent mileage using nothing fancier than unschematized XML fragments. Microsoft, meanwhile, has taken a great leap forward in Office 2003 with support for schematized XML documents. The first glimmer of this stuff came almost two years ago. It shipped last fall. If asked to paraphrase the Office XML strategy then, I'd have put it this way:

Let's get schematized information out into the open, where any XML-aware tool can see it and touch it and work with it -- locally and globally, on Windows or any platform -- and then let's see what happens. If we play our cards right we'll broadly legitimize schematization, and we'll be able to use Windows to layer semantic value on top of it.

Let's put schematized information into Windows, where any CLR-aware Windows application can see it and touch it and work with it.

The first strategy envisions a plurality of schemas arising from the grassroots. You won't often hear support for this strategy from Microsoft, but I heard it last fall at the Enterprise Architect Summit from Jean Paoli, who appeared (with Sun's Jon Bosak) on my panel Schemas in the wild.

The second strategy envisions a canonical set of schemas woven tightly into Longhorn. Years from now it'll ship. Years later, it'll reach critical mass, developers will have mastered its APIs, and schema-aware Windows apps could start to make a "semantic" way of organizing and finding information real for lots of people.

Why wait? Microsoft is telling us to disregard the grassroots Office XML strategy, which is here now and doesn't lock us in, in favor of the ivory-platform WinFS strategy, which is years away and does lock us in. If a compelling argument can be made for the second approach, I haven't seen it yet.

Developers who possess deep but tacit knowledge of complex hardware and software environments are notoriously unable to project themselves into the beginner's mind. Observation is the only way to bridge the gap. [Full story at InfoWorld.com]

The story includes an interview with Chris Rockwell of Lextant.com, a company that specializes in user research and interaction design. I really enjoyed my interview with Chris. In this 9 minute clip from our conversation, we discuss the value of raw user-experience instrumentation versus post-production highlights, the possibility of observing users throughout the lifecycle of deployed software, and the gap between users' and programmers' mental models.

In 1832, the Swiss crystallographer Louis Albert Necker discovered his famously ambiguous cube, which seems to jump back and forth between two orientations. Given the same raw data -- a particular arrangement of a dozen line segments -- our brains find different ways to interpret it. ... The real integration challenge resides inside our heads. There is no single frame of reference for data. [Full story at Infoworld.com]

At the heart of LibraryLookup there's a regular expression that matches a 10-digit ISBN. Wouldn't you know it, come January 1, 2005, that string of 10 digits grows to 13. Thanks to Tim Meadowcroft for the heads-up (via email, with permission). He adds:

All 10 digit ISBN's can be converted to 13 digits by adding a 3 digit standard code before them ("978" - it effectively puts all the existing codes into a single namespace), but as the last ISBN character is a base 11 checksum digit (that's why it can be "X" but all other chars must be digit 0-9), the last character will then change, see http://www.isbn.org/standards/home/isbn/transition.asp for details.

I gather that the new 13-digit ISBN will be compatible with the EAN / UPC [European Article Numbering / Universal Product Code] system. How will the variably-partitioned EAN / UPC mesh with the variably-partitioned ISBN? Beats me.

None of the publishers I know are freaking out about this impending change, so maybe it's not a huge deal for them. Regular folks probably won't even notice, except when required to speak ISBNs or type them into search pages. Like IP addresses -- and increasingly, like phone numbers -- ISBNs are just opaque identifiers. We rely on the Domain Name System, Google, Amazon, and other services to map those identifiers to names we can deal with.

In the digital realm this works out just fine. It's a bit shocking, though, when we reach for these mappings in the analog world and can't lay our hands on them. The classic dilemma: you call directory assistance from a cellphone, while driving, and try to remember the spoken digits long enough to dial them. My current solution: record a voicenote of the spoken number, and play it back a couple of times until it sinks into short-term memory. (I could pay for them to dial, but that would just gall me, and wouldn't plant the number in my phone.) The next step: do it as data, not voice. (Outside the valley of cellphone despair known as New Hampshire this is pretty common, I'm told.) After that: I dunno, but Ray Kurzweil figures we'll have ported consciousness to new hardware by then, which may solve naming and addressing once and for all. Or not.

A Webjay user named Brett Singer has been conducting an interesting experiment: a playlist of daily news clips. (Like all Webjay playlists, it can be subscribed in RSS.) I heard recently that TV remains the primary news source for three-fourths of Americans. Can that possibly still be true? I never watch TV news. But this new clip feed might change that, at least a little. TV has the resources to do things like take you to the North Pole to see and hear a scientist evaluate the melting ice pack, and a military analyst discuss the implications of an ice-free northwest passage. I won't watch something like that on CBS's schedule, and I won't even watch it on TiVo's schedule (since TiVo doesn't have the granularity for named two-minute segments), but I might find two minutes to watch it on RSS's schedule.

There's not a huge diversity of sources here -- the clips I've seen are mostly CBS, with some BBC and PBS. But that's already enough to give you a taste of what the RSS-ification of TV news will be like. It'll be a smorgasbord from which you sample, without regard for media brand, in response to the recommendations of your trusted group -- who are in turn influenced by your recommendations.

Webjay's creator Lucas Gonze uses the term broadcatching, which seems to have arisen at the intersection of RSS and BitTorrent. Given the relatively slow start for personal video recorders, it could take quite a while for this second-order phenomenon to catch on. If the PVR numbers that In-Stat/MDR has made up are even in the ballpark -- 40 million PVRs worldwide in 2008, extrapolated from 4.6 million this year and 1.5 million last year -- the RSS-ification of TV news can fly under the radar for at least a few years while CBS et al. absorb the impact of TiVo. And that's probably a good thing. Because if pages like this become pages like this too soon, the collaborative thing won't get a chance to happen.

Over the next few days I want to explore a series of questions about the "pillars" of Longhorn -- WinFS, Avalon, and Indigo. Last fall, when this stuff was first announced, I reacted with an entry entitled Replace and Defend. I argued then that Longhorn reinvents quite a few wheels. Nobody can blame Microsoft for seeking new ways to keep customers locked into its Windows franchise. That's a business strategy that every rational player must pursue, in one way or another. In chapter 6 of Information Rules, entitled Managing Lock-In, Carl Shapiro and Hal Varian write:

The great fortunes of the information age lie in the hands of companies that have successfully established proprietary architectures that are used by a large installed base of locked-in customers. And many of the biggest headaches of the information age are visited upon companies that are locked into information systems that are inferior, orphaned, or monopolistically supplied.

With respect to WinFS, Longhorn's new storage system -- an object/relational engine that also doubles as a conventional file system -- the claimed benefits are:

• Finding stuff. Those of us who sometimes blog things just so we'll be assured of finding them later have a special appreciation of the absurdity of the current situation. Unless we use an add-on to Windows such as X1, we can often find things on the Internet more easily and more reliably than we can find things on our own hard disks.

• Organizing stuff. We know that hierarchical foldering systems adapt poorly to the chaos of real life. Unix has always supported the concept of symbolic links, which give you the flexibility to construct alternate paths to the same thing. And indeed, modern versions of Windows do too. A little-known fact is that Junction, yet another wonderful utility from the indefatigable Mark Russinovich, enables you to create and delete symbolic links on Win2K or WinXP. But symlinking isn't something any normal user would be able to do routinely, and in any case it doesn't really solve the essence of the organizational problem, which is that we want to be able to group items dynamically based on the contents of individual items, and also -- crucially -- on relationships that tie sets of items together.

Nobody wouldn't want these benefits. The way in which Microsoft proposes to deliver them, though, contains some assumptions that I'd like to start unpacking. Let's start with the first benefit: finding stuff. Here's an example of a Longhorn search scenario:

For example, a user may want to use some pictures taken on a family vacation on her business Web site to promote a sale. She can tag these pictures already stored in a "\Family\Vacation\Photos" folder with a "Promote Sale" keyword when the sale begins. The application managing her Web site can then load all the pictures of this category and have them displayed as a slide show. When the sale ends, she can remove the tag from the pictures in a "WinFS" store. The website will stop showing them to the site visitors afterwards. [Longhorn SDK Documentation]

There's no need to wait until 2007 to see what this would be like. Just now, for example, I opened up Word 2003, wrote a short document, assigned it the keyword "Promote Sale," and saved it as XML. Here's a script to insert the document into a Berkeley DB XML database:

from dbxml import *
db = 'winfs.dbxml'
container = XmlContainer(None, db)
container.open(None,DB_CREATE)
doc = XmlDocument()
item = open ('myDocument.xml').read()
doc.setContent(item)
container.putDocument(None, doc)
container.close()

And here's a script that finds that document in the database, based on the keyword:

from dbxml import *
db = 'winfs.dbxml'
container = XmlContainer(None, db)
container.open(None)
context = XmlQueryContext(0,0)
context.setNamespace ('o', 'urn:schemas-microsoft-com:office:office')
xmlResults = container.queryWithXPath(None, 
    "//o:Keywords[contains(.,'Promote Sale')]", context)         

The power of pervasive free-text search, by the way, is something that Microsoft seems consistently to underestimate. Outlook, even in its latest incarnation, is helpless to find anything quickly. Everybody has to rely on third-party add-ons for this essential function. There's a hole in the market that you could drive a truck through, and the name on the side of that truck is Gmail, but I digress.

Here's the point of this installment. To the extent that our personal information stores contain information represented in XML, we have standard ways to search them. What's more, two powerful trends point to a brighter future for this scenario: the growing use of open XML file formats, and the steady advance of databases that can index and search XML content. WinFS embraces neither trend, and that looks to me like a looming headache. Personal information management, in Longhorn, will be a walled garden with its own notion of schema, and its own query language. To give users the benefit of finding stuff, Longhorn-style, developers will have to implement the Longhorn model. And then they'll have to find ways to unify that approach with the XML-oriented model prevailing in the world at large -- and indeed, even on pre-Longhorn Windows systems.

The justification for this headache, if there is one, must lie not in the realm of "finding stuff" but in the realm of "organizing stuff." WinFS relationships, in other words, must be capable of delivering such compelling benefits that there was no choice but to invent a proprietary storage model from the ground up. I'll explore that proposition next time.

Tim Bray raises some good questions about last week's Gillmor Gang episode:

First of all, a transcript would be so much better; I don't have an hour to listen and if I did it would be in my car, and even if I tried, sitting here in my office (even though the audio is excellent) my attention is continually getting pulled away by email or instant messages or red letters in NetNewsWire or whatever. If I'm writing code or a tricky position paper or reading something material or even just thinking about a hard problem I can tune out the distractions no problem, but four guys talking? The mind wanders. [ongoing]

Here's my thinking. Even for big media organizations with big budgets, it's a struggle to get audio transcriptions done quickly and well. But maybe, with the right set of tools, it'd be feasible to create a layer of indexing plus annotation that would contextualize and give meaningful random access to the audio stream.

Working through the process gave me a clearer sense of what tools we already have, and what tools we'd need to make it practical. A huge enabler is the ability to rely on a standard Web server, rather than a specialized streaming server. Last week, I indexed and annotated a downloadable RealVideo file. The same principle applies to a downloadable MP3, and that's the core of today's experiment.

The challenge then becomes to isolate segments, form links to the beginning of each segment, and pair each audio segment with annotations displayed in another pane. I found the Winamp player really helpful for fine-tuning start/stop times. You can use CTRL-J to jump to a minutes:seconds location, and can use the arrow keys to jump forward or backward in 5-second increments. It gets tedious to subtract by minutes:seconds in order to arrive at durations, but there are calculators that can help with that.

These conveniences only scratch the surface, of course. We're left with plenty of roadblocks. There was nothing to help construct the index, organize the annotations into a set of panes, or orchestrate linking from the index to the annotations. More woes: The result is specific to the Real player. It won't even work in QuickTime, which has SMIL support, never mind in Windows Media Player, which doesn't. Another possible issue: the index and annotations, encapsulated in .smil and .rt (RealText) files respectively, are (I suspect) opaque to Google, which defeats the purpose of using the annotations to make the audio partly searchable. And the elephant in the room: Real's RealText isn't HTML, and the Real player isn't a browser. We can awkwardly include AV content into a text/graphics viewer (i.e., browser), or awkwardly include text and graphics into an AV player, but we've never satisfactorily united the two modes.

Suppose we magically healed this longstanding breach. Suppose further that, in some hypothetical browser/player, we could even author for the combined medium -- for example, by capturing timecoded annotations in realtime, SubEthaEdit-style, or by collecting and presenting the URLs that participants visit during the event. Would the kind of hybrid presentation I'm envisioning still be a poor substitute for a complete transcript? If you had such a transcript, would the audio still be valuable, and if so, in what ways?

I can't answer these questions yet, but it's a fascinating area to explore -- and not only from the perspective of four (actually, five) guys talking on an IT radio show. Think about the meetings you attend. Think about the note-taking that does (or doesn't) occur in those meetings. Imagine being able to efficiently review what was actually said, not just what was summarized, when making decisions. In that situation, a complete transcript -- even if one could be produced cheaply and accurately -- won't tell the whole story. Recorded speech, linked to searchable annotations, would be an amazing enhancement to routine business communication.

I confess to a deep fascination with the seemingly mundane topic of logging. Software crashes, shopping cart abandonment, and security breaches are among the many situations in which you'll find yourself poring over logs trying to figure out what went wrong.

...

Logs can flood us with information, or they can tell us compelling stories. We can influence the outcome by artful and iterative refinement of the data we collect. [Full story at InfoWorld.com]

The event log subsystem was left on the cutting room floor, an executive told me, because hard choices had to be made in order to bring Win95 in under its 4MB memory budget. This was not so absurd as it now sounds. Win95's competition was Windows 3.1, which could run in 4MB. (As it turned out, of course, nobody ran Win95 in less than 8MB.) But while granting the case for prudent conservation of scarce resources, I argued that it was vital to get developers of mainstream Windows apps into the habit of logging not just outright failures and errors, but also routine status information that could be used to analyze patterns of software use and guide incremental improvement of software.

Developers of server applications were then already making liberal use of the event log. If the hordes of developers coming to Win95 from Windows 3.1 weren't immediately enabled (and expected) to do the same, I argued, an opportunity to improve software quality would be lost for a generation.

So here we are in 2004, I'm running Windows XP on my desktop, and there's essentially no interesting data in the Event Viewer's Application log. What are some examples of things I'd like to see there? Off the top of my head:

• Warnings. If the same warning appears repeatedly (or perhaps a set of related warnings spanning several apps), it's a sign that there's a problem with the software, or with the user's understanding of the software, or both. If we don't log these warnings, though, we can't detect patterns and respond to them.

• Settings changes. As a user, how many times have you tried to remember what settings were in place when something that's broken used to work? As a developer, how many times have you tried to get users to remember what they changed? Aren't such changes important events in the life of an application, worthy of logging?

• Launch and exit events. These are the most basic and obvious things to record, but we don't find them in the log. If we going to move toward "software as a service," shouldn't we keep track of what's used and how often?

Ironically there are much more detailed logs of our routine software activities on other people's machines (i.e., on Web servers) than on our own. There's no reason why this has to be so, and plenty of reasons why it shouldn't be. It's an accident of history, really. A questionable decision made during an era of resource scarcity now serves us badly in this era of abundance.

Yesterday I had a phone conference with Hadar Pedhazur of Opticality Ventures, during which Hadar mentioned that he's been using Asterisk, a Linux-based software PBX, with great success. Although Asterisk is VoIP-capable, Hadar's using cheap ($100) Digium cards to manage and route calls among his various business-related POTS lines. That really got my attention; I've long wanted such a capability. So I did some reading, and I also watched this presentation given by Digium's founder and Asterisk's developer, Mark Spencer.

I can't say more about Asterisk until I've had a chance to try it, but I do want to note that Mark's presentation -- a RealVideo stream synchronized to a slide show -- was extremely effective. The talk was given during Linux-Kongress 2003, at Universität des Saarlandes, home of the Virtual Courseroom Environment (VCORE) project. The project page notes:

Although many aspects of streaming multimedia are well understood there are many open questions concerning a real world implementation.

• The URL of the video, an HTTP-accessible RealVideo (.rm) file.

• A series of pointers to JPG renderings of the slides.

• Determining the duration of the video and encoding that in SMIL.

• Acquiring the JPG renderings.

• Determining the transition points and encoding the duration of each slide accordingly.

This is pretty nice! I've seen this done occasionally, but it's hardly routine. Since I'm on a random access kick lately, I decided to see what it would take to add an index to the presentation. A lovely example of how to do that can be found here, courtesy of UT Austin's David Gracy. Note that this example dates all the way back to 1999, yet demonstrates something quite compelling that we rarely see even today.

While I was at it, I explored how to incorporate the slides as text, rather than as images. Here's the result: a three-minute clip that includes the first four slides of Spencer's talk, and an index that links to each of the four transitions. The SMIL wrapper, playable in RealOne, looks like this:

<smil>
<head>
<layout>
<root-layout width="640" height="480" />
  <region id="text_region" width="320" height="480" left="0" top="0" />
  <region id="video_region" width="320" height="240" left="320" top="0" />
  <region id="text_region2" width="320" height="240" left="320" top="240" />
</layout>
</head>
<body>
<par>
  <textstream src="http://weblog.infoworld.com/udell/gems/asterisk_index.rt" 
    region="text_region" dur="3:08"/>
  <video src="http://graphics.cs.uni-sb.de/VCORE/Publications/\
    mark_spencer/Data/mark.rm?start=0:0&end=3:08" 
    region="video_region"/>
  <seq dur="3:08">
  <textstream src="http://weblog.infoworld.com/udell/gems/asterisk1.rt" 
    region="text_region2" dur="38"/>
  <textstream src="http://weblog.infoworld.com/udell/gems/asterisk2.rt" 
    region="text_region2" dur="32"/>
  <textstream src="http://weblog.infoworld.com/udell/gems/asterisk3.rt" 
    region="text_region2" dur="22"/>
  <textstream src="http://weblog.infoworld.com/udell/gems/asterisk4.rt" 
    region="text_region2" dur="96"/>
  </seq>
</par>
</body>
</smil>

Here are some things I discovered:

• Random access over HTTP. The other day I mentioned how HTTP 1.1 enables some players (RealOne, Winamp) to randomly access audio on a vanilla Web server. The same holds true for RealOne going against RealVideo content. However, although you can jump to a random location, it takes noticeably longer than when you do the same thing with a streaming server.

• Minutes-and-seconds notation. In the index file you can write URLs in terms of minutes-and-seconds, not the byte-range lingo that the client and server speak. These URLs aren't available outside that context, though. And while it's possible to package up the start/stop syntax into a .ram file that you can point a browser at, you can't (so far as I know) form an URL that indexes into the SMIL assembly.

• Text formatting is lame, but workable. Here's an example of a slide written using Real's markup.

This still isn't a great solution, but it's instructive to see what can be done with late-nineties technology. Given XML slide markup and a means of capturing transition timecodes, a VCORE-like system should be able to generate this kind of indexed presentation automatically. The slide content might need to be streamlined and simplified, but you could also link out from slides to richer Web pages if needed.

Conferences would be so much more useful if this were the norm. As an attendee, I should expect that when I return home, I'll have slide-by-slide random access to every talk. What's more, I should expect to be able to search for slide text and jump into presentations at the found locations. Remote attendees, meanwhile, would be able to purchase this level of access, thus defraying the cost of providing it.

The deluxe solution, of course, would make all these entry points bloggable by surfacing external URLs. But just having basic indexing done comprehensively and reliably would be a huge step forward. It sucks not to be able to take that for granted.

So we schlep to conferences, make painful choices between conflicting sessions, and feel vaguely guilty when we miss lots of them anyway because we're busy schmoozing. It's great to be bathed in WiFi signal at conferences nowadays, and able to blog them in realtime. Now that we've solved outbound access from the venue, let's solve inbound access to the content.

Michael Howard points to a free threat modeling tool written by Frank Swiderski, author of the forthcoming book Threat Modeling. The evolving formal discipline of threat modeling first came to my attention in 2000, when I read Bruce Schneier's Secrets and Lies. This picture, from chapter 21 of that book, is worth a thousand words:

One way to gauge the growing interest in threat modeling -- at Microsoft and elsewhere -- is to compare its coverage in the two editions of Michael Howard's Writing Secure Code (1, 2). In the first edition, threat modeling is mentioned in a section of Chapter 2, Designing Secure Systems. In the second edition, it becomes a chapter in its own right.

Swiderski's tool is a GUI-based .NET app that collects tree-structured information about entry points, protected resources, and threats. If you have the Visio drawing control, you can use that to add data flow diagrams, otherwise the tool includes a simple diagram editor. The classifications defined by the STRIDE methodology -- Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege -- are available as checkboxes. Likewise the classifications defined by the DREAD methodology -- Damage Potential, Reproducibility, Exploitability, Affected Users, Discoverability -- are available as numeric choices (1-10). ("The concepts of STRIDE and DREAD were conceived, built upon, and evangelized at Microsoft by Loren Kohnfelder, Praerit Garg, Jason Garms, and Michael Howard." -- Writing Secure Code, 2nd Edition)

I've always been suspicious of the kinds of software tools that just provide bookkeeping support for some methodology. Of course, a methodology that people can actually understand and use is really just a formalization of common sense, and I think the STRIDE/DREAD stuff falls into that category.

Here's a report on a very simple threat model, generated from the XML data captured by the tool:

This kind of report, and the process that leads to it, is no more than a framework for thinking through the issues involved in securing an application. But it is also no less than that. If the analytic framework is easy to pick up and use, you'll do more and better analysis. To that end I have a couple of suggestions. One's easy to implement, one's really hard.

Here's the easy one. The built-in report viewer failed when I tried to produce my report. I took a look at the included XSLT stylesheets and found a couple of C# methods defined using the <msxsl:script> mechanism. They weren't doing anything particularly vital, just removing whitespace and truncating strings, so I removed references to them. That enabled me to produce the excerpt shown above using an external XSLT processor, though still not from within the tool itself, for reasons I haven't figured out. I'm sure there's an easy fix for this. But relying by default on a non-standard extension like <msxsl:script> isn't a great public relations move. It encourages people to think the tool is more Microsoft-centric than it in fact is. True, it requires the .NET Framework (v1.1) to run, but the generated XML is entirely neutral. For example, it writes out a data flow diagram two ways: as a Base64 encoding of a bitmapped image, and also as a chunk of SVG that could be useful in all sorts of ways on any platform. My recommendation: make the default XSLT transformations similarly neutral.

Now here's the tough one. The real impediment to doing this kind of analysis is the classic problem of documentation that's not connected to code. It's tedious and boring to enumerate entry points (e.g. network ports, file systems) and their relationships to well-known threats. What are the odds that I'll update my threat model if I change the port on which my service is listening? Slim to none. Of course the code knows what port the service is listening on. What's more, the code can tell us a lot about potential attacks. For example, my XPath query service, written in Python, uses the BaseHTTPServer class. (I do this partly because it's so simple. There's no massive IIS or Apache edifice to worry about, just a small amount of code -- which I've read and which I understand -- that implements an HTTP responder.) Given a database of viable threats to BaseHTTPServer, an automated analysis of the code could fill in parts of the threat model for me. More broadly, automated analysis of the configuration data used by app servers, routers, firewalls, and other infrastructure software could help us automatically populate threat models. That'd be a great way to mine value from the XML that's now routinely used to describe these things. I predict that source-code analysis and configuration-file analysis will help us do more frequent and more reliable threat modeling. It'll be a challenge. But if we know that's where we're headed, we can design source-code metadata mechanisms and configuration-file formats accordingly.

Over the weekend I upgraded a kid's PC from Win98 to XP. I'd been dragging my heels because Win98 was "good enough" for games, IM, and writing school reports, but this installation had long since reached its half-life. Also, I was curious to see what a 98-to-XP upgrade would be like, never having done one. So I fired up the installer and posted the kid on guard to alert me when intervention was required.

He summoned me repeatedly, but in each case the reason was a popup ad, not anything technical. Call me naive, but the frequency and intrusiveness of these ads surprised me. Otherwise, though, the in-place upgrade went smoothly. It really is remarkable that the Win9x kernel can be uprooted, and the NT kernel inserted in its place, with so little disruption.

In order to ensure the maximum half-life for the new system, I made myself administrator and gave a limited account to the kid. Then I set him to work verifying that his games still worked. All of them did except for Age of Empires. The installation report suggested I should reinstall it. I switched to my account, did that, and fired up the game. It worked. Then I switched back to the kid's account and fired up the game. It still failed.

So for now, the kid is the proud owner of administrative privilege. I could milk this for irony by pointing out that Age of Empires is a Microsoft product. But I'd rather take this in a different direction. Partial trust is a hard problem, period, in all operating systems and environments. So hard that we either spend inordinate amounts of time figuring out how to make partial trust work, or we punt and allow more trust than we should. Or both.

In this particular example, had I the time and inclination to solve the problem, I'd probably fire up Sysinternals' Filemon and try to find out which file or directory Age of Empires is failing to read or write. Of course the problem could lie elsewhere -- with API permissions rather than file permissions, for example.

This isn't only a Windows issue. Across the board we need better ways to visualize trust boundaries and diagnose problems arising at these boundaries.

It's great to see Ward Cunningham's friendly face popping up on MSDN's Channel 9. In these segments, he connects the dots between the patterns that we increasingly use to guide software architecture, and the environments in which we formulate, discuss, and apply those patterns.

In the first clip, Ward recalls how the aboriginal Wiki was a place for programmers to work out, in collaboration, a set of ideas about how to do object-oriented programming. In the second clip, he notes that what gates programming productivity isn't syntax, but rather API/library/framework surface area. "Keeping up with what's available in the libraries," he says, "is the number one information overload challenge."

It's hard, maybe impossible, to master all the existing and emerging disciplines that flow together in modern programming work, but then, we shouldn't have to:

I wouldn't think to start a program from first principles. If I want to make a program, I want to find the people who know kind of how to do it, and say, come sit with me, come help me get started. Let's talk to each other about what we're doing, let me get the feel for how it's supposed to go. Once you have a program that's working, then it's just...improving it. [Channel 9: Ward Cunningham]

How and where do we have the conversation in which we merge our individual understandings? Patterns are ways to frame that conversation; Wikis and other online venues are places to have it.

Seymour Hersh's The Gray Zone was this week's blockbuster New Yorker article. Hardly anybody commented on Paul Goldberger's High-Tech Bibliophilia, a review of Seattle's new library. I wouldn't either, were it not for the contrast with an earlier review entitled Brutal Architecture and written by Keith Pleas, a software architect, author, and trainer who now works in the area of patterns and practices at Microsoft.

The contrast between the two reviews could not be more striking. The Goldberger piece, which opens onto a two-page spread dominated by a huge photo of the library's angular exterior shell, is full of the kinds of airy proclamations that art and architecture critics love to make:

...the most important new library to be built in a generation, and the most exhilarating...

...not so much a rejection of traditional monumentality as a reinterpretation of it...

...a reinvention of the idea of the public library...

Keith Pleas decided to find out for himself. Armed with his digital camera and a deep understanding of how architecture (in the software realm) can fail the test of use, he investigated the library from the inside out. Pleas conducted his tour last month, before the Goldberger review appeared, but he deconstructs an earlier Seattle Times magazine cover story with devastating effect:

On page 25 of the magazine we see a picture of Koolhaas, Ramus, and good old Paul Maritz(!) standing on a tiny balcony:

(Seattle Times photo)

What you can't see in the electronic version of the image is that the architect and designer are holding onto the thin metal railing at waist height. Ramus, who knows more about how the building is actually put together, is holding on with both hands. Maritz, who's probably the smartest of the three, is standing further back. And why are they holding on? Well, here's the view down from where that picture was taken:

You can see the same waist-high railing on this, a main passageway. You can also just see an edge of a substantial industrial table placed against the railing. And if you have any imagination at all, you can see how things placed on this table - which seems to just be begging to be climbed on - will have full opportunity to demonstrate their glide characteristics as they descend 6 stories (I forget, but I think that's the number) to the busy floor below.

...

Of course, the architects / designers didn't completely ignore "life safety" issues in designing the new library. In fact, here's an...innovative...solution to the double-issue of both tripping and konking your head on an angled support while you're exploring the "unity of knowledge":

[Keith Pleas: Brutal Architecture]

Bravo, Keith! The New Yorker and its readers don't know it yet, but architecture criticism is (or should be) forever changed by what you've done here. And those of us who care about the architecture of software should be heartened to see the instinctive concern for user experience that motivates your analysis.

Jeremy Zawodny notes that Yahoo's DomainKeys proposal is now public. Here's the Internet-Draft; here's the blog chatter as seen by Technorati.

In the blog introduction to my story on sender authentication schemes, I included some clips from an interview with Sendmail Inc.'s Eric Allman. Here's another excerpt, in which Eric discusses the issue of roving users. Although DomainKeys can potentially deal with this case -- by mapping its DNS selectors to individuals -- he notes that you're better off making an authenticated connection to your home MTA, if not through a VPN then by means of port 587 message submission. Here's the Internet-Draft on that topic, which Eric co-wrote and hopes will become a BCP (Best Current Practices) document.

Eric concludes this segment by saying that, for the first time in a long time, he's "cautiously optimistic" about doing something effective against spam. Likewise, I'm cautiously optimistic about the long-term value of publishing keys in the DNS. The DomainKeys scheme initially maps keys to organizations, but has the flexibility to map them to individuals as well.

Doug Kaye's ITConversations has the first installment of a new online talk show called The Gillmor Gang. My ongoing interest in the ability to form URLs that link into large media objects has now infected Doug, and we've been talking about how to enable that capability on his site.

And then it all came rushing back to me, like the hot kiss at the end of a wet fist. I recalled an exchange, some months ago, with Kevin Marks, a former Apple QuickTime engineer who is now Technorati's director of engineering. Our conversation was prompted by my mobile webcasting column. Kevin wrote to point out that streaming is really only useful for live events, and that downloadable files are otherwise superior. But what about random access, I asked? HTTP 1.1 supports that, Kevin pointed out.

I've known for a long time that certain applications -- notably Adobe Reader -- make use of the HTTP Range header to request partial content. I'd never seen the protocol in action, though. It took me a while to find a PDF on the Web that exhibits random-access behavior -- perhaps because it's not really necessary for the vast majority of sub-1MB PDFs out there -- but eventually I found this 4MB document and was able to watch Adobe Reader requesting a sequence of chunks in the background, and skipping ahead when I scrolled to the end of the document to view the last page.

What about downloadable MP3s? I tried QuickTime, no joy. Windows Media Player, no joy. RealOne Player: bingo! And likewise Winamp. How did I never notice this before? Here's some of the chatter between Winamp and Doug's server:

GET /mp3/2004/The%20Gillmor%20Gang%20-%20May%2014,%202004.mp3 HTTP/1.0
Connection: keep-alive
Host: rdscon.vo.llnwd.net
User-Agent: WinampMPEG/5.0
Accept: */*
Icy-MetaData: 1
 
HTTP/1.0 200 OK
Date: Sat, 15 May 2004 03:05:23 GMT
Server: Apache/1.3.29 (Unix)
Last-Modified: Sat, 15 May 2004 02:43:02 GMT
ETag: "216a53-1173385-40a583b6"
Accept-Ranges: bytes
Content-Length: 18297733
Content-Type: audio/mpeg
Connection: close
 
GET /mp3/2004/The%20Gillmor%20Gang%20-%20May%2014,%202004.mp3 HTTP/1.1
Connection: keep-alive
Host: rdscon.vo.llnwd.net
User-Agent: WinampMPEG/5.0
Accept: */*
Range: bytes=9902232-
 
HTTP/1.0 206 Partial Content
Date: Sat, 15 May 2004 03:05:23 GMT
Server: Apache/1.3.29 (Unix)
Last-Modified: Sat, 15 May 2004 02:43:02 GMT
Accept-Ranges: bytes
Content-Type: audio/mpeg
Content-Range: bytes 9902232-18297732/18297733
Content-Length: 8395501
Connection: close
 
GET /mp3/2004/The%20Gillmor%20Gang%20-%20May%2014,%202004.mp3 HTTP/1.1
Connection: keep-alive
Host: rdscon.vo.llnwd.net
User-Agent: WinampMPEG/5.0
Accept: */*
Range: bytes=15105006-
 
HTTP/1.0 206 Partial Content
Date: Sat, 15 May 2004 03:05:23 GMT
Server: Apache/1.3.29 (Unix)
Last-Modified: Sat, 15 May 2004 02:43:02 GMT
Accept-Ranges: bytes
Content-Type: audio/mpeg
Content-Range: bytes 15105006-18297732/18297733
Content-Length: 3192727
Age: 214
Connection: close

There remains the problem of link addressability. Doug would have to invent, and hack into his server, some kind of URL parameterization -- which, in fact, he's considering doing. Of course somebody must already have thought of that, and sure enough, Ari Luotonen did in his original 1995 proposal for byte ranges:

EXAMPLES OF THE BYTERANGE URL PARAMETER
 
The first 500 bytes:
   host/dir/foo;byterange=1-500
 
The second 500 bytes:
   host/dir/foo;byterange=501-1000
 
Bytes from 501 until the end of file:
   host/dir/foo;byterange=501-

According to this comparison of HTTP 1.0 and 1.1, the URL parameter idea ran afoul of HTTP 1.1's conditional GET feature, and so byte ranges migrated into the realm of HTTP headers.

To sum up, an ordinary downloadable MP3 sitting on a conventional Web server (as opposed to a streaming MP3 hosted on an Icecast or Shoutcast server) is perfectly able to be randomly accessed -- but only by means of HTTP Range headers, not by means of parameterized URLs. And some (but evidently not all) MP3 players are prepared to exploit that random-access feature.

What's missing?

• A Web server convention for accepting parameterized URLs like the ones Ari Luotonen proposed way back when. By "convention" I mean something like Real's ramgen, a virtual directory that invokes special processing. The handler for that directory would be a server extension, implemented in various ways on various servers, that would convert from parameterized-URL lingo to HTTP-Range-header lingo.

• An audio player convention for exposing such URLs to users. I envision it as a Link button that goes active when the player is paused, and that produces the parameterized URL when clicked.

I can see at least one major objection. The byte range syntax isn't human-friendly. The hours/minutes/seconds format that streaming servers support would be nicer. Knowing nothing about MP3 formats, I can't say whether it would be feasible for a sufficiently smart server extension to translate from hours/minutes/seconds to byte ranges.

Recently I spoke with Dave Lewis, vice president of deliverability management and ISP relations at Digital Impact. His company's motto: "Making e-mail marketing more effective is our single-minded passion." In one of his online essays, entitled "How to Keep B-to-B E-mail From Getting Caught in Filters," his first rule is "Get permission."

I argued that RSS does away with the need for marketers to ask our permission, for us to grant it, for marketers to play by the rules when we revoke it, and for us to trust that marketers will play by the rules. With e-mail marketing, control resides with the sender and permission is a "best practice." With RSS, control resides with the recipient and permission is an inherent property of the medium.

I feel Dave's pain. E-mail direct marketers are stuck between a rock and a hard place. They believe e-mail is necessary because it's an "intrusive" medium, yet they are forced to neuter e-mail's intrusiveness by complying with the opt-in gold standard. Unfortunately, there's no middle ground. With RSS recipients can have, and increasingly will demand, control of the channel.

Dave and I agreed on one point. "You'd be crazy not to communicate with your customers in their medium of choice," he said. My preference is RSS. Trust me with control of the channel, and I'll be more likely to trust you with my business. [Full story at InfoWorld.com]

I think the rhetoric of email direct marketing -- that it's an opt-in, customer-controlled medium -- should correspond to the reality. It makes email direct marketers understandably nervous when I point out that RSS has all the right characteristics -- including, nowadays, lower cost, given the expense incurred on both ends of the email pipe in order to keep the channel clear.

Obviously direct marketers will be among the last to relinquish channel control to the customer. Meanwhile, there's another species of email that's ripe for migration to RSS: institutional alerts. My bank, for example, sends me email alerts when my checking balance falls below $500. To separate those alerts from my spam filters on the one hand, and from my interpersonal email on the other hand, I had to write a filter to catch them and route them to a folder. Many (probably most) people won't go that extra mile. They'll have to pluck the bank's messages from a chaotic email stream, and will wind up missing some alerts.

The obvious alternative is a personalized RSS feed. Does anyone have this already? I'm hoping that, before the end of this year, at least one of the institutions that currently sends me email alerts will offer an RSS option.

A couple of years ago, after I heard Alan Cooper speak about his company's ethnographic approach to interaction design, the word persona first appeared in this blog. Last Friday, "persona" popped up in back-to-back phone interviews, and made me realize that Cooper's formulation of IT stakeholders as characters in a story has become deeply rooted and widespread. The first interview was with Microsoft's Bob Muglia who, in the course of laying out the Windows server roadmap, said this:

Over the last 18 months we've focused on trying to understand the different audiences, or roles, within IT, and how they consume technology. We do this by associating personas with the individual roles.

What we've been focusing on lately is behavioral segmentation and modeling, typically as represented by a persona -- a one-page front end with a face and a name, and a narrative description of the person's behaviors. We do that to encourage companies to design for a small number of segments about which they are very well informed.

The literary theme continued today, when Roland Piquepaille blogged a Michael Shrage article in CIO.com that coins the term 'plog' for 'project log' -- a powerful technique that I've used myself and written about.

Persona is an ancient and beautiful word. Plog is a brand-new word that's even uglier (if possible) than blog. But the words don't matter. What's striking is how the art of storytelling -- our instinctive human way of making sense of the world -- has woven itself into the science of information technology.

Peter van Dijck wrote to tell me about his tool for converting the URL of a Real stream, plus start/stop times, into a link to the specified segment. A while ago, I mentioned Rich Persaud's version of the same idea, which works with Windows Media and QuickTime as well as Real. Using either of these, you can do what I did the other day -- namely, link to a segment within a video stream -- without hacking URLs and wrapper files.

As helpful as these tools are, I've come to see that the hassles they alleviate are only part of the reason why we're as yet unable to weave video effectively into blog conversations. In the case of yesterday's clip, for example, there's probably a 50-50 chance that my carefully-prepared link actually worked for you. C-SPAN's streaming setup is amazingly robust, but invariably the content that's most likely to attract links occurs at times of peak load. If I really wanted to make sure you could see that 30-second clip, I might have done better to capture it and post a downloadable version.

That, of course, would raise all sorts of questions. First of all, how? It's doable, but not easily and not (to my knowledge) with free tools. Second, in which format? Third, does fair use cover these kinds of quotations? (I think it should, and will be testing that hypothesis.)

Despite these issues, the overriding consideration may be that streams require specialized servers, whereas downloadable clips (which nowadays play progressively) do not. Downloadable clips are, of course, inherently link-addressable, and since they're short, it's not imperative to be able to point to locations within them.

What we're left with, though, is an asymmetry. Big media organizations, for now, still have the advantage over small independents, because the big organizations are more able to deploy and manage streaming infrastructure. Bloggers can link into those streams, and/or capture and post quotes from them, but can't yet easily produce streams. What we can do easily is produce short downloadable clips.

All this could change, of course, if a hypothetical video-oriented version of Audioblog.com were to emerge. For $X per month, I'd be able to send streams from my iSight camera to this hypothetical service, which would support X concurrent viewers of the stream. Hmm.

I mostly avoided the hearing today, but tuned in to C-SPAN's video stream just long enough to catch this brief segment in which Senator James Inhope argues that "if pictures are authorized to be disseminated among the public, then for veery picture of abuse or alleged abuse of prisoners, we [should] have pictures of mass graves, pictures of children being executed, pictures of the four Americans in Baghdad who were burned and mutilated." He concludes: "Let's get the whole picture."

Absolutely. The notion of authorized dissemination is problematic, though. In the transparent society that we are becoming, the whole picture most certainly is developing. The Net is a force of nature. It superconducts information and superdistributes awareness.

Of course the military, like every enterprise, is entitled to try to control the terms on which its employees can engage with the Net. So the Seattle Times reports that Tami Silicio, who gave us another piece of the picture, was fired for violating the Pentagon ban on pictures of flag-draped coffins. Likewise, Seattle's other paper, the Post-Intelligencer, reported last fall that Michael Hanscom was fired for his pictures of G5 Macs on a loading dock at Microsoft. Fair enough. In a similar position of responsibility, I'd have to make similar choices. But let's be clear: the whole picture, by definition, cannot be authorized.

In a recent column on how we use and abuse email, I mentioned the idea of passing attachments "by reference" rather than "by value." Unfortunately I overlooked a product recently reviewed by InfoWorld that does exactly that. The Xythos WebFile Server has a companion WebFile Client that hooks File Attach (in Notes and Outlook) and replaces attachments with secure links to an access-controlled and versioned instance of the document. Cool!

The $50K price tag, as our reviewer noted, "may keep smaller companies away." But other implementations of the idea are clearly possible. I've received a bunch of responses to the column saying: "We attach files because IT gives us no alternative." Xythos offers an alternative. I'd like to see the "Intellittach" concept turn into a broadly-adopted convention.

The other day I wondered why some well-known technology acronyms -- notably UPnP -- aren't expanded on the home pages of the organizations promoting those technologies. In the case of UPnP, at least, the reason is that it isn't (any longer) an acronym:

The UPnP mark is not an acronym and should not be represented as such. The mark is a single entity that happens to consist of four symbols (i.e., letters), which individually do not have any particular meaning. [Tips for using the UPnP Certification Mark]

I found this puzzling in light of this Q and A from the trademark FAQ of a Boston technology law firm:

13. Can I register an acronym of my company name as a trademark?

Companies with lengthy trade names will sometimes use the acronym of their trade name as their primary service mark: e.g. Columbia Broadcasting System, National Broadcasting System, and American Broadcasting System, use the acronyms CBS, NBC and ABC, respectively, as marks for the service of providing news and entertainment services over electronic media.

Following last week's critique of XBRL, I had an interesting email exchange with David vun Kannon, a manager in KPMG's financial services practice and one of the editors of the XBRL spec. The dialogue went far beyond what InfoWorld's letters column could ever accommodate, so with David's permission, I'm reproducing it here.

David vun Kannon:

I feel your analogy was inadequate and the "too complex" criticism misses the point. XBRL isn't designed to be hand-written, and that level of simplicity is not a virtue in the design space it targets.

As one of the designers of the XBRL specification, I sympathize with your desire for a simple XML format for the exchange of financial and business reporting data. But as your article's lead paragraphs point out, the world of accounting standards is wickedly complex. The design scope of XBRL had to address that complexity, as well as the use of financial data in all kinds of tax and regulatory filings worldwide. Did you really expect something simple from that target?

Here's a recipe for a "simple" financial reporting format:
- assume a single accounting framework
- assume the framework never changes
- assume one currency
- assume one language
- mix content and presentation
- assume businesses will change how they report to fit your design

The above recipe actually works for single application languages where there pis one dominant consumer, such as the IRS' XML format for tax filings. But the world doesn't need a thousand different financial reporting languages. That is the "stovepipe application" thinking that misses the forest for the trees. That is why XBRL is trying to provide a unifying framework.

It is nice to know that your blog can get by using RSS. However, Reuters and Dow Jones can't, and I doubt InfoWorld runs on RSS. For them, there is NewsML. Ever read the NewsML spec? Looked at the latest version of FpML, for describing financial derivatives? An "apples-to-apples" comparison of XML languages would compare XBRL to these languages, because of the breadth of the business problem they are each trying to solve.

There are thousands of companies that report financial results according to US, international and local rules, as well as separate tax reporting. If you wrote every blog entry in four separate languages, with an eye to satisfying a different set of picky editorial rules for each, your blog analogy would be more appropriate.

The companies with financial reporting needs that are similar to your blog example will be served by software, such as Microsoft's Excel add-in now in beta, that manage the complexity for them.

The number of developers that will have to face head-on the complexity of the XBRL spec is low. You can write an XML Schema without delving into the depths of the XML Schema spec. Only the writer of an XML Schema validator has to do that. Similarly, developers at businesses can write XBRL instance and taxonomy documents using tools. Only the developer of XBRL support software has to go the limit with understanding the spec.

Jon Udell:

> It is nice to know that your blog can get by using RSS.
> However, Reuters and Dow Jones can't, and I doubt InfoWorld
> runs on RSS.

As a matter of fact InfoWorld does, in a variety of ways. I'm not convinced Reuters and Dow Jones couldn't either, as RSS is now modular and extensible.

> For them, there is NewsML. Ever read the NewsML spec?

Yep. And it's not small, I agree. I'll also agree that modular extensions to RSS that would bring it to parity with NewsML would yield complexity equal to that of NewsML. However the key difference, in my view, would be a lower activation threshold and smoother growth curve -- i.e., the ability to start with something simple and concrete, and evolve to the more complex and abstract.

> Only the developer of XBRL support software has to go
> the limit with understanding the spec.

The "tools will manage the complexity" argument is always compelling, but also always worrisome to me. Over and over again I've seen stupidly simple formats and protocols triumph over highly-engineered counterparts, especially when -- as I believe is true in the case of XBRL -- the goal is widespread, if not universal, adoption by a broad constituency. We'll probably just end up agreeing to disagree, but I've noted with great interest the evolution of XML specs, in the Web services realm, away from the monolithic and towards the granular and "composable." This seems to me a fundamentally correct way to attack complexity. And XBRL seems monolithic, not composable, hence my reaction.

David vun Kannon:

As background, a paper I gave at XML Europe a few years ago on the design of XBRL 1.0 is still available on the web at http://www.gca.org/papers/xmleurope2000/papers/s26-01.html. While the particulars of XBRL 1.0 have become dated, the motivating sections are still relevant. You might also want to note how much smaller/simpler the 1.0 spec is, compared to 2.1!

I've had to think about the issues you raise since 1999 and the design of XBRL 1.0. There are many different aspects to the complexity problem, including scope of the business problem and choice of base technologies. For instance, I'm asked relatively frequently "Why don't you use RDF?" as if RDF was pixie dust that could be sprinkled on a problem to make its complexity go away. Complexity is conserved.

BTW, a link to your column has been posted to the xbrl-public Yahoo Group. As the only public (non-member) Yahoo Group for XBRL, it attracts most of the newbies and naysayers, and the latter are happily adding to the thread agreeing with you. For the sake of the former, I've posted my response to you over there as well.

I agree with your points on modularity. XBRL is designed so that the simple is simple and the complex is possible. Believe it or not! The "Hello, World" test for a single financial fact is

<xbrl namespaces for XBRL, XML Schema Instance, 
and US GAAP taxonomy go here>
<us:assets contextRef="c1" unitRef="u1" 
  precision="18">7</us:assets>
<unit id="u1">ISO4217:USD</unit>
<context id="c1">
  <period><instant>20041231</instant></period>
  <entity>
    <identifier scheme="http://www.duns.com/D-U-N-S">
      1234567890
    </identifier>
  </entity>
</context>
</xbrl>

It is hard to point to any part of the above as unnecessary, though most votes go to the precision attribute.

So the use of XBRL by a company is modular, and can expand in a gradual, modular way. I'm not sure if the 2.1 spec is organized quite the way a primer should be. Also, while XBRL is committed to modular expansion into the future, the current spec and conformance suite are monolithic. During the last version design phase, I argued for profiles that would let tools or validators claim conformance to XBRL while not implementing the whole spec. This didn't make the cut. But a properly written intro to XBRL would show the natural breakdown of the parts of the spec and how different parts (different linkbases for example) can be used independently.

I think a big influence on why the spec isn't "officially" modular is that XBRL has succeeded most with global financial regulators, who have typically wanted all the bells and whistles. Indeed, they want modules that aren't finished yet, such as the Formula Linkbase I am designing now. This adoption process has damped the "small is beautiful" psychology and grass roots momentum dynamics that drive some specs to wildfire rates and levels of adoption. Web pages and blogs were pioneered by individuals, not businesses or government departments. True bandwagon dynamics for XBRL will wait until the SEC (the 800 pound gorilla of regulation) requires using XBRL (for external financial reporting) and until the Excel add-in becomes widely available (for internal management reporting and financial consolidation).

So I think we agree far more than we disagree. XBRL is an undoubted challenge to developers. Its linkbases are the first use of out-of-band hyperlinking. For developers used to working with numbers, it is surprising that so much of accounting is navigating a hypertext!

XBRL was advised by Tim Bray in a recent conference keynote to take six months off (or more) and stop inventing/using bleeding edge technology. It hasn't happened, of course, but the market is starting to catch up the spec.

David is right to point out that a government-mandated reporting format is an unlikely source of grassroots innovation. But this evocative statement -- "it is surprising that so much of accounting is navigating a hypertext" -- does make me wonder. Years ago I worked on the first incarnation of a business information product (which still exists) that blended financial reports with news, biographies, and other contextualizing information. It was a read-only product delivered on a write-only medium, CD-ROM. Back then there was no other choice. Now we produce some kinds of hypertext almost as naturally as we consume it. Will we be able to paint financial information on the universal canvas, mixing it with text, charts, math, and other XML brushstrokes? For the sake of our ability to step back and see the big picture, I hope so.

In my interview last week with John Shewchuk, one of the Indigo architects at Microsoft, I asked whether XML disciplines can or should model data, as well as exchange it. I like the answer John gives in this audio clip. There really isn't a primary data model, he suggests. (Note to self: Get over it!) Relational, object, and XML disciplines are just aspects of a relativistic universe of data. Very postmodern!

I edited this clip with Audacity, by the way. I've been using it on the Mac for a while, but only just recently noticed that it's a wxWindows-based app that runs on Windows and Linux too. Like other sound editors, it offers a bunch of effects filters. I rarely use them. I just want to capture, crop, and post. Audacity makes it pretty straightforward to find a segment in an audio track, zoom in to precisely mark its boundaries, and save the result to MP3.

A few of us InfoWorlders spoke yesterday with one of Microsoft's Indigo architects, John Shewchuk. In the course of our conversation, Shewchuk mentioned the recent WinHEC announcement about device support for Web services protocols, reported in InfoWorld on May 5:

Microsoft Corp., Intel Corp., Lexmark International Inc. and Ricoh Co. Ltd. on Tuesday detailed new Web services technology designed to make it easier for users to connect devices such as printers, digital cameras and digital music players over a network. The companies at Microsoft's Windows Hardware Engineering Conference (WinHEC) officially announced a Devices Profile for Web services, which describes how devices should use Web services protocols. The announcement builds on WS-Discovery, a Web services specification that Microsoft, Intel, Canon Inc. and BEA Systems Inc. introduced in February. WS-Discovery describes a way for devices to find and connect to Web services. [InfoWorld.com: Web services find way to devices]

Shewchuk sees this as a "singularity":

There is nothing different about the Web services on a printer, than the Web services at Amazon. That's mind-blowing. And it means the same Visual Studio tool that I pick up to do my cross-enterprise application, I can now point at my printer. And the same reliable messaging protocol that makes sure my information gets to Amazon also makes sure that I don't drop packets when I'm moving from room to room on WiFi sending a print job.

Update: I got to wondering about cross-platform Rendezvous, and that led me to Porchdog Software's Spike, a dynamically-discoverable network clipboard for both Windows and OS X. Spike, which Just Works, is built on Porchdog's Howl, an open-source SDK that brings Zeroconf/Rendezvous capabilities to Windows, Linux, and FreeBSD. Very cool.

¹ When your organization and domain name are both the same acronym, e.g. UPnP, you'd think it would make sense to expand the acronym on your home page. But I can't find the phrase "universal plug and play" -- or even any of the constituent words "universal," "plug," "play" -- on this page. And this isn't uncommon. www.svg.org doesn't bother to expand SVG to Scalable Vector Graphics. The phrase "Portable Document Format" appears nowhere on www.planetpdf.com.

I've seen other examples of this, and I know why it happens. If you're so far inside a technology that you run an organization and website dedicated to it, you've long since lost touch with the world that might not know what that technology's acronym stands for. But while you can't cater to every newbie question, a site that aims to be an educational resource should probably answer the first and most obvious question: "What the heck does ___ stand for?"

We don't yet know what the steady state of the blogosphere is going to look like. As has been snarkily reported, lots of blogs die on the vine. Of course plenty don't, and there's also a steady influx of new voices. Here are three that have enriched my daily trawl for ideas and perspectives.

Brendan Eich, creator of JavaScript and architect of Mozilla:

The challenge for Mozilla and other open source projects is not to "react to Microsoft", any more than it is to "react to Macromedia". MS and MM are reacting to the same fields of force that govern everybody. The prize we seek is a better way to develop common kinds of networked, graphical applications. [Brendan Eich: roadmap]

Martin Roberts, enterprise architect:

When a Process fails where do you need to route the fault to? Normally a human - so why do most tools make this a cumbersome task? Why do these so called next generation tools find dealing with people such an alien idea? I believe the answer lies in the fact that most of these emerging tools have been built by people used to handling classes that rarely touch humans directly. They tend to be focused on the J2EE/.Net like frameworks which are low level in the inspirations and have failed to take into account the gains of the 4GL world of the early 1990's. [Martin Roberts: Architecture Musings in IT]

Evelyn Rodriguez, engineer turned freelance marketer:

Have you ever watched a start-up make that corporate transition from the inside? It's not just that the dogs and beer bashes go, but something subtle, intangible seems to shift. The palpable energy evaporates. It's not the transparency that's at issue. Maybe not even the quarterly view of the world (most start-ups have to watch their cash closely anyway and thus balance the short-term and long-term). It's more the unspoken effect and influence of "best practices" and the pressure to conform to a more respectable and familiar culture that are the hallmark of measurable metrics of Wall Street. Who knows what SAS's life-friendly practices are worth? Just looks like a cost to me on a balance sheet. Giving Googlers 20% of time to goof off on pet projects? That's productive time being wasted! [Evelyn Rodriguez: Crossroads Dispatches]

A more descriptive name for Adobe Designer 6.0 might be "InfoPath for PDF." The concept is brilliant: exploit Microsoft's failure to make InfoPath ubiquitous by putting interactivity and XML smarts into Adobe's free PDF viewer, and by offering a forms builder that targets both Adobe Reader and Acrobat. Announced last summer, in beta now, and scheduled for release this summer, Adobe Designer is that forms builder.

Adobe says that Designer targets version 6 of the PDF players. I had to upgrade both to the (still unreleased) version 6.02, though, in order to use Designer-built forms. You can start a form from scratch, or by importing a layout from sources including PDF, Word, and even InfoPath files. Either way, you can associate the form with an XML Schema. But while the schema defines the shape of the data collected by the form, there's limited runtime enforcement of schema constraints in Acrobat or Reader.

Some constraints, such as field lengths, are handled automatically. But when I wrote a regular-expression restriction into the schema, Designer's preview didn't complain when I entered text that didn't match the pattern. In Acrobat, I was able to save an invalid XML instance. Bottom line: if you want real schema validation, you'll have to do it yourself in the back-end process that receives the data.

Designer enables you to specify repeating elements, but they only work in concert with a server that regenerates the form with space for new data. You can't grow a region interactively, a la InfoPath. That's a limitation of the Acrobat/Reader forms player, of course, not of Designer.

Despite evident weaknesses, the Designer/Reader duo offers two key strengths: digital-paper fidelity, and a ubiquitous runtime. Using the free Reader, I was able to fill out a Designer-built form, print a high-fidelity copy for my records, and post its XML data to a Web server. No matter how the future of e-forms unfolds, that's going to be a popular scenario.

Note: This item appears on page 18 of InfoWorld, May 3, 2004, in the Product Previews section. Normally I point to InfoWorld articles on InfoWorld.com, but since we haven't yet found a home online for Product Previews, I'm publishing (the original version of) the item here.

The XBRL [eXtensible Business Reporting Language] spec describes how the parts of an XBRL instance interrelate, using state-of-the-art XML technologies such as XLink and XPointer. And it talks at length about the syntax and semantics of "taxonomies" that abstractly define chunks of financial reports. No sign of any actual financial data, though. And the link to a sample page at xbrl.org, returned a "404 Not Found." I'm not surprised. The poor bloke whose job it was to produce that sample must have suffered a polymorphic recursive brain meltdown. [Full story at InfoWorld.com]

Seriously, let's think about where the middle ground lies here. What the hammer is to the carpenter, the spreadsheet is to the accountant. In 2003, the dominant spreadsheet -- Microsoft Excel -- gained the ability not only to read and write XML, but also to guarantee fidelity to arbitrary schemas. We've yet to see the impact of that key development, but in the short run I expect we'll see a thousand flowers bloom as organizations, for their own purposes, begin to schematize their business information. In parallel, we're seeing the evolution of a global interconnected business network, implemented as a fabric of web services. The Platonic solution that XBRL envisions will, I'm guessing, more likely result from Darwinian forces now in play.

You'll schematize your own information because you can, and because it's intrinsically valuable to do so. What self-respecting accountant wouldn't want an automatic check on the validity of data? Meanwhile, your schematized information will be drawn inexorably into the interconnected business fabric. To survive in that ecosystem, you'll wind up transforming your stuff. The purpose of the transformation won't be to conform to a reporting specification, but rather to interoperate with the fabric. Proxies within the fabric will crank out the reports we need to see.

PS: Sorry about the title, but when the phrase "Attack of the killer accountants" came up blank on Google, I just had to claim it.

I recently met with Groove's Jack Ozzie and Michael Helfrich. Jack is a co-founder and VP, development; Michael is VP, applied technology. The subject, of course, was the forthcoming V3 of Groove, a product I first saw in beta four years ago this spring. We had a wide-ranging discussion; here are some of the key takeaway points.

Sayonara, top-to-bottom XML I don't believe that I pay a performance penalty for using XML, and depending on how you use XML, you may not believe that you do either. But don't tell that to Jack Ozzie. The original architectural pillars of Groove were COM, for software extensibility, and XML, for data extensibility. In V3 the internal XML datastore switches over to a binary record-oriented database.

You can't argue with results: after beating his brains out for a couple of years, Jack can finally point to a noticeable speedup in an app that has historically struggled even on modern hardware. The downside? Debugging. It was great to be able to look at an internal Groove transaction and simply be able to read it, Jack says, and now he can't. Hey, you've got to break some eggs to make an omelette.

I'm sure Groove has made the right choice here. Still, it's troubling if you believe -- as I do -- that a high-performance XML database ought to be a core piece of client infrastructure. Groove's original XML database vision was probably too forward-looking. Version 1.0 was effectively done, for example, by the time the ink was dry on the XPath specification. XML storage technology didn't then support what Groove wanted to do. Arguably it still may not. We don't yet know what Chandler will be able to achieve with Berkeley DB XML. Meanwhile WinFS is turning out to be less like the XML database I imagined, and more like a record-oriented (or rather, CLR-object-oriented) database.

So how do we resolve the impedance mismatch between our desktop storage engines -- the file system, conventional databases -- and the XML content model that is increasingly the choice of desktop applications? I'm still looking for the solution to this puzzle.

Groove and .NET The managed-code interface to Groove gets an overhaul in V3 but the core product itself does not rely on .NET. Not because Groove's developers wouldn't like to use .NET. They very much would, Jack says. But rather because the already steep ante -- Groove's a 10MB download -- looks even steeper when you pile on a 20MB .NET download. This isn't news, just another datapoint, but every time I hear this it tells me two things. The CLR and .NET Framework aren't yet infrastructure that a mainstream Windows developer can take for granted. But when that finally becomes true, a whole lot of pent-up developer demand for .NET services will be released.

Python? During a demo of the V3 forms builder, which gains some nice yardage on the previous version, I noted that the scripting languages supported are VBScript and JavaScript. Hmm, thought I. Is this thing an ActiveX Scripting Host? And if so, can I plug in another scripting engine that works in that environment, say Python? The answers were "Yes" and "Don't see why not, we'll get back to you."

Groove Web services The Web services stuff that I explored a while ago has matured, and is baked into the product. Groove V3 comes up listening for SOAP calls from localhost, and can be configured to listen for SOAP calls from remote nodes.

The good news: in addition to using forms, you can write scripts that reach through the Web services layer to find things in Groove spaces. The bad news: you'll have to, there's still no built-in search capability.

Challenges Early reviews of the 3.0 beta have showered praise on the product's revamped UI, and I'll add mine: it's cleaner and better optimized for common tasks. There remain challenges. Groove is a holistic solution that shares idioms with both Windows and the web in ways that seem familiar, but sometimes aren't. In the Groove "Files Tool," for example, you're shown what looks like a file in a folder, but is actually an encrypted and synchronized Groove object. Double-clicking the file opens it into its default editor, which may (or may not) reveal the fact that the file has been decrypted to your local temporary directory for viewing and editing. Quitting can result in a two-step tango. First the editor asks if you want to save. Then Groove, detecting changes, asks again: "Do you want to save?" It's the classic dilemma of every document manager that hooks File Open and File Save gets in between apps and storage in order to add value. In Groove's case, that value is considerable: automatic secure synchronization, and change notification, across all instances of a shared space. But until and unless a more intimate relationship can be forged between Groove's secure/transacted/synchronized storage and the OS-level storage APIs that applications expect to see, there's just no way to make this seamless.

Groove's use of web metaphors raises other challenges. For example, navigation from tool to tool within a Groove space uses browser-like back/forward controls. But in a space that includes an embedded browser, you end up with two separate sets of back/forward controls. Groove's hyperlinking is also similar-yet-different. Depending on the Groove tool you're in, you may be able to form a link to a record, a view, or the tool itself. For which audience is the link relevant? It depends. Members of the space that contains the tool can jump to it from a link pasted into, say, a chat window or discussion. The same link pasted into another space may or may not be accessible to everybody, depending on who's also joined to the target space. Can the link point into Groove from the outside, say from an intranet web page? In theory yes, though I'd be surprised if anyone has ever done it.

Groove's transacted/synchronized storage model envisions a species of applications that don't yet exist outside of Groove. Likewise its hyperlinking model envisions collaborative scenarios that don't yet flourish outside of Groove. Such applications and scenarios would present thorny usability challenges even if there were no legacy to consider, because the total experience is so different from what we're conditioned to expect. Of course there is a legacy. Reconciling it with Groove is incredibly hard, but there's been steady progress all along, and V3 is another big push forward.

It's true that you can use native XML databases to manage the growing number of business documents created by the new generation of XML-savvy end-user applications. It's handy, for example, to search an insurance database for incident reports that match some structured pattern of in-line metadata. But hybrid SQL/XML databases can do that too, and they can also join the structured XML content with relational columns -- a powerful combination. So XML databases are migrating into a niche that SQL/XML can't and won't occupy. They're becoming the high-performance pumps that push XML traffic around on the emerging services web. [InfoWorld.com]

I've followed the odyssey of Sonic XML Server, née eXcelon, née ObjectStore, for quite a long time. I wouldn't have predicted that XML databases would become the context engines of the services web, but I guess it's not too surprising. More surprising, I have to admit, is the extent to which the SQL discipline is merging with the XML discipline in the conventional database engines. "It's possible that developers will want to stay within an XML abstraction for all their data sources," said Oracle's Sandeepan Banerjee when I interviewed him for last summer's story on SQL/XML hybridization. Wow. I still can't believe that an Oracle guy said that!

E-mail is the jack of all trades, but the master of none. There are better ways to transfer files, hold discussions, deliver notifications, broadcast newsletters, schedule meetings, work collaboratively, and manage personal information. But even though e-mail isn't the best tool for any of these tasks, it provides a single interface to all of them. Here's a challenge: Let's improve the various functions performed by e-mail without multiplying the interfaces people must learn in order to use those functions. [Full story at InfoWorld.com]

I also mentioned the old idea of passing attachments "by reference" rather than "by value" -- that is, emailing links to uploaded attachments, rather than including the attachments themselves. Several people responded to that, including two whose emails I'm quoting here with permission. For John Heery, the issue is IT control:

I thought your solution for having an e-mail client that could pass a file "by reference" was a great one, and one that several of us at work use with a two step process. We drop the file on a shared drive, and then just send a link.

However, your assessment of why people use e-mail to transfer files may be accurate for Infoworld, but I seriously doubt it is on the mark for most companies. As IT locks down systems in an ever increasing game of black ops, e-mail is just about all we poor users have left. My laptop doesn't have a floppy or CD-RW, so I can't write onto removeable media. The USB port is my current option, until IT discovers I bought a jump drive to move files around. FTP isn't an option, in fact, the FTP ability of IE 6 has been disabled on my machine. My co-workers and I couldn't stop laughing as you rattled off WebDAV, scp, and Radio UserLand. These may be great little secrets for IT people, but at least at our company they aren't made available. We can't even determine what we set as our default browser webpage.

Lotus Notes is our mail client, and it's forced to do the file transfer. For a while, several of us received training in Lotus Application Development and developed some great database tools for our groups. IT has removed that ability. They only support work they developed, and even if you agree to forego support, development by users is not an option. In the ever expanding cold war with IT, my fellow Engineers and Technicians have now retreated to the MS-Office applications. Converting our former Lotus Notes apps to Access with VBA has given us power to develop flexible tools...for the time being. Last week we discovered our ODBC connection between Access and Notes had been disabled. Another battle in the war.

I've used a desktop computer since 1984 when I was required to own one for Engineering school. It's insulting to be told by the new MSCE qualified IT kid that if I'm given the ability to change screen resolution on my laptop, I'll just get into trouble down the road. Please. I run the same OS on my home machine as an Administrator and never have any problems. The problem isn't the variety of tools, nor is it the users. It's the availability. [John Heery]

Several other correspondents said the same thing: they'd love to implement the idea, but lack the means to do so. It's ironic but inevitable that the PC, which was originally the information worker's secret weapon in the "ever expanding cold war with IT," has become the raised-floor sanctuary guarded by the priesthood. I can definitely see both sides of that argument. But in this day and age, when anybody can sign up for a free blog site that requires only a vanilla browser to use, I guess I'd ask this of IT: Do you want users to route around you by sharing files insecurely in free services, or would you rather admit that link-addressable filespace on the public web is as essential a tool of modern work as an email address is?

Jon Hoover also liked the idea, is in a position to do something about it, and wonders how to bend Exchange to this purpose:

Just a comment on your "E-mail's many hats" article, which I enjoyed reading. Recently, an "administrative assistant turned graphics and marketing person" in our organization was found to be sending SEVERAL 100-350 MB attachments to users out on the Internet -- via email, of course. This became apparent very quickly as our Pentium 2 333 MHz Exchange 5.5 server choked down that much data, our T1 flooded, and our mail store approached the Exchange file size limit it has been flirting with for quite some time. I instituted a limit policy that very day, which had always been in the back of my mind (for example, what happens when a virus is created expressly for the purpose of filling Exchange mail stores by sending huge attachments to an entire Global Address List when it detects it is on a LAN connected to such an Exchange server -- sending small attachments to other users not directly connected to the server).

The problem, of course, was that everyone was sending large files. The limit I instituted for outgoing is now 3.5 MB, incoming at 10 MB. These are, in my opinion, very large limits, but complaints quickly grew. I created a samba share on our network which users could drop files into, making them immediately available through a symlink to our public web server. The URL could then be emailed instead of the actual file.

Now, how big of a next step is it to create a form in Exchange which can automatically copy a file into the share, and insert the URL (or URLs) into the email message. Taking it a step further, can the form accept directories to send, zipping them first and copying them to the share? Can it add a password to the zip archive and place it into the body of the message?

Thoughts? I may just have to find a guy in house to put this to task, the more I think about it. [Jon Hoover]

I've no experience with Exchange development, but I told Jon I'd float his query here in case somebody has a solution they'd like to share. For a first level of security, the URL contained in the email message could look like this:

https://user:password@domain.com/~user/proposal.pdf

By the way, I notice that Chad Dickerson is hiring a developer for InfoWorld. Cool! I'm sure there are lots of other priorities, but maybe we can also task the person to make our own email infrastructure smarter.

Sam Ruby pinpoints the glitch:

Let's take a closer look into Jon's RSS feed:

<title>Active r&amp;#233;sum&amp;#233;s</title>

Arguably, the InfoWorld process did parse the RSS feed, once. [Sam Ruby]

Thanks Sam, and apologies to the InfoWorld crew -- it was my fault after all. Clearly Sam's right: we could, indeed, learn a lot from those 13th century artisans. And I guess Micah Dubinko would agree.

C-SPAN captured David Weinberger's excellent rant yesterday at the Technology and Politics Summitin DC. The stream is overloaded at the moment, but I captured a clip (WinMedia, QuickTime).

I wish I could say it was easy to do this kind of videoblogging, but it's just not true. What I meant to be a quick, spontaneous thing turned into a chore. It's frustrating, really -- we're so close, yet so far, in terms of being able to sling video clips as easily as we sling text, still images, and even audio.

The always-interesting Sean McGrath has a great column this week about software customization. He says, in part:

In order to stay sane, most programmers concentrate on the part of the problem they are working on today. As a consequence, their view of what pieces of the functions under development need to be parameterized and which do not, tends to be a quite low level. Indeed, most of the items programmers will chose to parameterize will amount to double dutch to the business analysts. [Sean Mcgrath: The mysteries of flexible software]

Closely related to this theme are the tools and frameworks for capturing and manipulating business rules. A while back I wrote a column on the subject, and James Owen -- a seasoned user of the various rules engines -- wrote to me about it. After a bit of back and forth I recruited him to review this class of product for InfoWorld, and he's produced a series of articles: JRules, Blaze Advisor, Jess and OPSJ.

I'm also quite curious to see what Microsoft will make of Ward Cunningham's ideas and techniques. I interviewed Ward in Refactoring the business and, in my blog companion to our feature on test-driven development, he talks about the FIT framework that he's used to push testable business logic into spreadsheets that business analysts can make and use.

We can all agree that software must be customizable. But when programmers alone decide how users can do things, you often end up with a scenario like Aunt Tillie's OS X adventure: a dashboard packed with incomprehensible dials and knobs. If the dashboard was built with a dynamic language, the programmer can at least rearrange the controls more quickly and more easily. But the rules engines that James Owen has been writing about, and the FIT framework that Ward Cunningham has created, point toward a radically altered relationship between software makers and software users. It can't happen too soon.

Traveling in France in 2001, I visited Chartres Cathedral and was lucky enough to show up in time for Malcolm Miller's lecture. Seemingly unchanged from the last time I'd seen him, in 1978, Miller again made the architecture and stained glass come alive in his inimitable way. This time, though, I heard something I hadn't the first time -- about standards. When the construction project drew in artisans from the 13th-century French countryside, the first order of business was to agree on standard weights and measures. I wonder what those negotiations were like!

It all seemed kind of quaint until, a couple of days later, I found myself in an Internet cafe struggling with a French keyboard. The @ symbol was the showstopper. I finally abandoned typing and, feeling ridiculous, copied the symbol from a web page and pasted it into the email message I was composing.

What reminded me of all this was the title of last Thursday's entry: "Active résumés." To be honest, I took the lazy route at first and wrote it as "Active resumes" because I knew that using a LATIN SMALL LETTER E WITH ACUTE would likely cause some problems. But then, mindful of Sam Ruby's recent admonition to test international characters "in every nook and cranny you can find," I went with the correct spelling.

Since I write in XML, my input strategy was to use numeric references, which meant typing this string of characters: "r&#233;sum&#233;s" -- and that's exactly what showed up on the InfoWorld home page when the item was excerpted there. Evidently the process that creates those excerpts is reading, but not parsing, RSS feeds.

The item itself displayed correctly, but other subtleties emerged. For example, Technorati and Feedster produce hits when searching for the wrong spelling (T, F) but not when searching for the right one (T, F). (Update: Hmm. Technorati does find active résumé, though. So does Google, but it finds a lot more instances of active resume.)

I discovered that my own XPath search does find the entry, though entering the search term presents a bit of a challenge. Copying an instance of 'résumé' into the search form works, as does the extra-geeky method of writing the URL-encoded version ('r%C3%A9sum%C3%A9s') directly into the URL. But the resulting display was wrong, until I switched the browser's text encoding to UTF-8. I guess I should have my search server emit the appropriate UTF-8 header.

Sam's essay points to a Joel Spolsky article that is the single most lucid treatise I've seen on the subject of internationalization. We've come a long way with Unicode, but there's still some distance to go. Chartres Cathedral still stands, so apparently those 13th-century carpenters and stonemasons got things sorted out reasonably well. I trust we will too.

Today's New York Times includes a brief article on music blogging. The story links to Webjay and quotes Lucas Gonze and Alf Eaton. I've written three recent entries about this phenomenon: The media-player fireswamp, Blogs + playlists = collaborative listening, and Networks of shared experience. My fascination with the topic may seem like diversion from my usual themes, and in a way it is, but I think the issues transcend music, copyright, and the RIAA.

Alf Eaton writes today:

I think the MP3 blogs (which are essentially annotated playlists) might well be taking the middle ground in the P2P vs music industry wars - I hope that the record industry will begin to see the value in what these grassroots enthusiasts are doing to promote their music. On the other hand, a large part of making these playlists under current laws involves turning your back on the major labels and concentrating on the music libre, the 'free music', the stuff that wants to be shared. Those artists that make their tracks freely available online are the ones that will benefit most from the collaborative filtering and recommendation networks that are being set up. [Hublog]

What an active résumé should include will vary by profession and according to personal inclination. For a musician, a couple of complete tracks from each CD. For a home renovator, photos and write-ups of some completed projects -- and for extra credit, video walkthroughs. For a programmer, links to those of your applications, tools, or specifications that touch the public domain.

Here's the bottom line. What Alf calls "collaborative filtering and recommendation networks" will rival -- and my guess is, largely supplant -- conventional marketing and promotion. But if those networks can't find you, they won't be able to help you.

In our July 18 feature, Canning Spam we mentioned an Internet draft proposal from Hadmut Danisch, called RMX (Reverse Mail eXchange). It was an elaboration of an earlier proposal by Paul Vixie, architect of BIND (Berkeley Internet Name Domain), who in turn attributes the idea to Jim Miller of JCM Consulting. The idea is elegantly simple. In addition to publishing the MX (Mail Exchange) DNS records that identify inbound mail hosts, an organization also publishes reverse MX records that identify outbound hosts. A receiving server queries the DNS to find out if the sending host is so authorized. The name yahoo.com is easy to forge, but the IP addresses of Yahoo's outbound servers are not.

The devil's always in the details, of course. It's remarkably difficult to define exactly what "sender" means in today's complex e-mail environment. Three current proposals -- pobox.com's SPF (originally Sender Permitted From, now Sender Policy Framework), Microsoft's Caller ID for E-Mail, and Yahoo's DomainKeys (unpublished) -- take differing approaches. [Full story at InfoWorld.com]

Here are some clips from my interview with Eric Allman. First, Eric explains why Sendmail Inc. is implementing DomainKeys in preference to the other schemes. Then, Eric and I discuss crypto and the end-to-end principle, relative to DomainKeys.

Steve Vinoski, middleware architect at IONA and a prolific columnist, has been blogging for a couple of months at Middleware Matters. Back in 2002, his IEEE Internet Computing column used the title that I stole for this blog entry: Middleware Dark Matter. The reference is to Clay Shirky's excellent meme "PCs are the dark matter of the Internet," which helped the peer-to-peer movement define itself circa 2000. Vinoski wrote:

We can apply a similar analogy to middleware because the mass of the middleware universe is much greater than the systems -- such as message-oriented middleware (MOM), enterprise application integration (EAI), and application servers based on Corba or J2EE -- that we usually think of when we speak of middleware. We tend to forget or ignore the vast numbers of systems based on other approaches. We can't see them, and we don't talk about them, but they're out there solving real-world integration problems -- and profoundly influencing the middleware space. These systems are the dark matter of the middleware universe. [Steve Vinoski]

Absolutely true. When I read this, though, I couldn't help but imagine the same column having been written, for another audience, like so:

The mass of the middleware universe is much greater than the systems -- based on Perl, Python, CGI, FTP file transfer, Unix shell, Visual Basic -- that we usually think of when we speak of middleware. We tend to forget or ignore the vast numbers of systems based on other approaches such as message-oriented middleware (MOM), enterprise application integration (EAI), and application servers based on Corba or J2EE. We can't see them, and we don't talk about them, but they're out there solving real-world integration problems -- and profoundly influencing the middleware space. These systems are the dark matter of the middleware universe.

Both of these passages make perfect sense to me. Though driven apart by a deep cultural schism, the two integration styles are utterly co-dependent.

A brief special announcement for folks living near the intersection of New Hampshire, Vermont, and Massachusetts. The Nashville duo called Betty Dylan, whose signature tune American Trash has been percolating through the Webjay playlists, will be returning to Keene, NH, on Thursday April 22. Where: E.F. Lane hotel on main street. When: Happy hour, 5PM. I'll be there!

And now back to your regularly scheduled program...

One of these years, my bank will upgrade to a new system that's built around Web services. They'll probably offer a basic "rich Internet application" -- for Windows, Java, or Flash -- that connects to those services. When the bank announces the upgrade, it will stress the richer user experience and choice of interchangeable clients.

Those will be crucial benefits indeed. What won't be said, because it's harder to explain, is that the system will also have become radically extensible. Suppose I want to trigger an alert when a transfer exceeds some limit or when a duplicate amount appears. Today, if the system doesn't implement these rules, I'm stuck. In a services-oriented environment, though, I needn't depend on either the bank or my client software. If neither delivers the features I want, I'll inject an intermediary that does. Local proxies are geeky curiosities today, but someday we'll wonder how we lived without them. [Full story at InfoWorld.com]

In this screenshot, Firefox is pulling this week's InfoWorld column through a proxy based on the one included in the Twisted framework for Python. Inside the proxy, I'm using mxTidy to convert the text of the page to XHTML. Then I'm using libxml2's XPath search to find just the paragraph elements with the class attribute ArticleBody, and rewriting the page to include only those elements.

It's kind of a parlor trick, I'll admit. But realtime XML transformation of Web pages could have applications that go way beyond ad blocking. Suppose I store all my XML-convertible Web content in an XML database. (Some stuff can't be XHTML-ized, but it turns out a lot can.) It's just text, after all, I bet a year's worth of content is a drop in the bucket compared to a typical MP3 collection.

Given such a database, the on-the-fly filter could do some clever correlation. Suppose that for the pages I read -- and maybe also for each link in those pages -- the filter extracts URLs, queries the database for elements that mention those URLs, and rewrites the current page with links to the query output. Voila! Instant context.

I don't yet know if this will be practical, and in fact my XML.com column is late this month because I haven't figured that out yet. But it's an exciting idea. We have a surplus of storage and processing power on the desktop, but never enough useful context. When more of our data flows are XML, local proxies will really shine. Even now, though, they can do more than you might think.

David Weinberger's recent essay, There's No "I" in "Identity", advances a notion of real-world identity that's so different from mine I had to sort out why. David writes:

In the real world, we don't identify everyone. We only identify those about whom we have doubts that we have to resolve for some purpose. Identifying is not the default in the real world. Nor, IMO, should it be online. [JOHO]

Authentication is about the continuity of relationships, knowing who to trust and who not to trust, making sense of a complex world. Even nonhumans need authentication: smells, sound, touch. Arguably, life itself is an authenticating molecular pit of enzymes, antibodies, and so on.

Distinguishing between self and other is what every living organism does, all the time. So is identifying others. Humans are hardwired to recognize faces, voices, gaits. We do it always and automatically. Perhaps so automatically that we don't notice, for the most part, that we are doing it. When my teenage daughter comes downstairs there's rarely any ambiguity about who she is. (Though there can be, sometimes it's one of her friends.) But at 100 yards, watching someone that might be her walking up the street, identification becomes a foreground task. Is that her gait? Her hairstyle? Her clothing? Once these questions are asked, it becomes imperative to answer them.

Suppose she has just returned from shopping downtown, where she made a cash purchase in a store. We might be inclined to call this an anonymous transaction. There was no need for identification, so none presumably occurred. Except that's not really true. If she paid with a twenty-dollar bill and forgot to pick up her change, odds are she can return to the store and collect it. The store clerk's lizard-brain will authenticate her face, her voice, her gait. Or, what's becoming increasingly likely, the store's surveillance camera will.

Of sci-fi's three "killer B's" (Gregory Benson, David Brin, Greg Bear), the one most often cited in discussions of identity and privacy is David Brin, whose book The Transparent Society I can't recommend too often. But I think it was Gregory Benson who, in his 2000 keynote talk at the O'Reilly Open Source Convention, said that we "shed data trails" as we move through the real world, just as surely as we do when we move through cyberspace.

With cameras proliferating in meatspace and blogs pervading cyberspace, it's getting harder and harder to distinguish between "real" and "virtual" data trails. Does it matter? David and I agree that it doesn't. We're 180 degrees apart on the default case, though. I think identification defaults to always-on.

The XML-Deviant column at O'Reilly's XML.com (index, rss), which began in January 2000, would have been called a blog had the term been more current then. Written first by Leigh Dodds and now by Kendall Grant Clark, the concept was a brilliant one. Recruit literate developers who participate in key mailing lists (Dodds: xml-dev, Grant Clark: W3C Technical Architecture Group), and have them publish reports that summarize and comment on weekly activity.

This is a potent form of communication. For people who lack the time to closely monitor activity in some area, these bulletins are a way to keep a finger on the pulse. For the participant/narrator, they're a way to build personal brand and -- perhaps -- influence the agenda.

It's been clear to me for a long time that the participant/narrator, armed with easy-to-use Web publishing technology (aka blog tools), will be a key player on every professional and civic team. A couple of years ago I sketched out how blog narrative can work as a professional project management tool. Just today, I learned of a great example from the realm of civics. Not co-incidentally, it involves another XML.com regular, Simon St. Laurent.

Simon lives in Varna, NY, which is between Ithaca and the town of Dryden, whose Democratic Committee he now chairs. Today's Ithaca Journal fills in the backstory:

St. Laurent can be seen, notebook and digital camera in tow, at Planning Board and Conservation Advisory Council gatherings, as well as at special meetings on fire departments, speeding and comprehensive plans. So I admit, my curiosity was piqued. What could motivate this seemingly normal man to submit himself to hours of political talk and legalese? Talk that even elicits occasional groans from those delivering it. Turns out, it's all in the name of a blog -- http://simonstl.com/dryden/.

"I volunteered with the local Democratic party in the last elections and made some calls for them. People would ask me questions and I'd have partial answers and they'd have partial answers. It seemed like an opportunity to learn more about what was going on and to help the person on the other end of the phone."

So on Nov. 6, St. Laurent launched his Dryden site. Six months later, he hasn't missed a posting. [Ithaca Journal]

Now that the hype about political blogs has died down, it's clear that this is the real deal: a grassroots effort to connect a political process to itself, to its constituency, and to the outside world. No fanfare, just steady and reliable information flow.

Every team can benefit from this approach. By narrating the work, as Dave Winer once put it, we clarify the work. There can be more than narrator, but it makes sense to have one team member own the primary role just as other members own other roles.

The new device in the right-hand column of my template is a SafariBox -- it's like the GoogleBox, but for Safari Books Online. Disclosure: I helped design Safari and sometimes still advise the project, though rarely nowadays. I'm using the SafariBox here because I enjoy being reminded about books, and -- as with the GoogleBox -- because I enjoy making serendipitous search-driven connections.

To receive HTML from the SafariBox, use an URL like this:

http:\//safari.oreilly.com/safaribox.asp?
  v=s&t=0&q=javascript

Try it.

If your blog software can make HTTP calls at page-construction time, you can use this version to dynamically generate SafariBox content into your statically served pages. I'm doing that in a Radio UserLand macro, for example.

Alternatively, you can let client-side JavaScript handle things at page-load time. It's the same strategy that, as I mentioned the other day, could enable a Technorati trackback counter.

To receive JavaScript that writes the SafariBox HTML, tack on j=1 and wrap the URL in a SCRIPT tag, like this:

<script 
  src="http:\//safari.oreilly.com/safaribox.asp?
  v=s&t=0&q=javascript&j=1">
</script>

That's the method I'm using in this post. There's a character encoding/decoding glitch, by the way. If you see "Top?5" instead of "Top 5", the question mark was was originally, I think, a Unicode non-breaking space -- U+00A0. My Radio UserLand macro sees it as \xA0, and converts it to a space. Firefox renders it as a question-mark when using the UTF-8 encoding, but as a space when you switch to ISO-8859-1. MSIE, though, seems to render it as a space using either encoding. Go figure.

If there isn't a place on the web that collects ad-targeting misfires, there should be. And here's an entry for it:

This is the article: GNU/Linux is changing the face of the New Europe. Could Google have been thinking about donkey, a password calculator? Was there an ungulate-based connection: gnu <-> donkey, via Linux adoption?

Anyway, it's even funnier than Amazon's Customers who shopped for this book also wear clean underwear. Keep 'em coming!

Jefferson Provost has written a thoughtful essay on music sharing as viral marketing. He writes, in part:

The big issue here is how serious music fans decide what music to buy. I'm talking about the people who maintain large CD collections and spend a lot of money on music -- the customers that the music industry should be holding close to their hearts. These people not only spend a lot of money themselves, but they influence their less musically-inclined friends. These people tend to have idiosynchratic tastes, and are picky to the point of snobbishness. They don't buy music based on music industry mass-marketing. They buy it based on hearing it and liking it, and the way they hear new music is by sharing it with friends. Radio used to play a part, too, but consolidation has turned music radio into a steaming pile of crap, so what's left? Networks of like-minded friends sharing music are what's left. [Jefferson Provost]

How much do you want to pay?
$5$6$7$8 (recommended)$9$10$11$12$13$14$15$16$17$18

But smoothing out the payment process matters only when there are people who want to pay. Let's look at some of the evolving ways to arrive at that state of mind. Sebastién Paquet recently posted a blueprint for a blog-based music recommendation network. Alf Eaton responded with an implementation that connects the dots between reading weblogs that talk about and link to freely-available MP3s, aggregating those weblogs, converting an aggregated page to a playlist, and -- directly from the player -- inserting a track into a Webjay playlist.

Here's what the process looks like. From this page of African griot tunes by Dembo Jobarteh, I used Alf's SMIL bookmarklet (drag to your toolbar) to synthesize a playlist of the MP3s linked on the page, and launch the player:

While the tune Allah la ke is playing, I click Recommend this tune using Webjay, and here's the result:

One more click adds the tune to my African griot playlist. Slick, huh?

I'm not much of an audiophile, to be honest, and there are lots of other people who will get more deeply into music-blogging and playlist-sharing than I'm likely to. But the process at work here is deeply fascinating to me, and generalizes to other realms. Every kind of digital experience can thrive in the virtuous cycle of the blogosphere: use it, capture part of it, link to it, write about it, search for it, read about it, aggregate it, rinse, lather, repeat.

Consider another kind of digital experience: software. The ability to try before I buy is great, but it's so much more powerful to tap into the shared experience of a knowledgeable user of the software I might want to buy. That's why Paul Everitt's spontaneous demo seemed like such a revelation.

I once read an interview with Michael Kinsley, right after he stepped down from the editorship of Slate. What had he expected Web publishing to be, the interviewer asked, and where had the medium fallen short? His answer was immediate and precise. He'd thought of the Web as a medium for shared experience. So, for example, music reviews and film reviews would quote from, rather than merely describe, songs and movies. That mostly hasn't happened yet, for both legal and technical reasons, but I see signs of a breakthrough. In the long run it has to happen. We crave access not only to intellectual products, but also to other people's experience and understanding of those products. When we focus on sharing experience -- which is sometimes, but not necessarily, the same as sharing product -- we'll unleash powerful economic forces.

BoingBoing's other blogs commenting on this post feature, added yesterday, has provoked a flurry of responses. Co-incidentally, I had just made myself a Technorati Trackback bookmarklet: drag this link -- TT -- to your toolbar, then click while visiting a blog article to see Technorati's roundup of posts commenting on the article.

As Cory Doctorow mentioned in an email I was cc'd on, the "other blogs commenting" feature ideally should display the count of inbound links, or, in case there are none, vanish. Here's a picture of a trial implementation:

There are two moving parts. First, a service that asks Technorati for the count, and returns some JavaScript. Here's the guts of my trial implementation:

tsearch = 'http://www.technorati.com/cosmos/search.html'
tpage = urllib.urlopen(tsearch + '?url=' + url).read()
m = re.search ('from <span class="greentext">(\d+)</span>',tpage)
count = int (m.group(1))
return """
document.write('Technorati comments: 
<a href="http://www.technorati.com/cosmos/search.html?url=%s">%d</a>')
""" % (url, count)

Second, a template modification to call this service for each published item, passing the permalink of the item, and including the results in a <script> tag.

This is the classic pattern for pages that are dynamically generated but statically served, as is true for most blogs. You make client-side JavaScript add up-to-the-minute information at page-load time.

Although I can host the service that queries Technorati, I'd rather not, and in any case most bloggers can't. So I've disabled the feature for now, but I'd love to see Technorati offer a callable counter.

The fuzzy intersection of official and unofficial data has never been a comfort zone for information technologists. In chapter 4 of Klaus Kaasgaard's Software Design and Usability, Xerox's Palo Alto Research Center (PARC) alumnus Austin Henderson says that "one of the most brilliant inventions of the paper bureaucracy was the idea of the margin." There was always space for unofficial data, which traveled with the official data, and everybody knew about the relationship between the two. [Full story at InfoWorld.com]

It strikes me that all of my recent experimentation -- with XHTML microcontent, semantically-oriented CSS, and structured search -- has a similar flavor. I've been looking for ways to scribble in the margins of the Web. Not because it's the right thing to do, but because it's perhaps the only feasible way forward.

Last Friday I visted CoreStreet, a company whose ingenious approaches to large-scale credential validation and physical security I mentioned in my Permissions on the edge column last fall. While I was there, CoreStreet's president, Phil Libin, who blogs at vastlyimportant.com, showed me a neat gizmo intended to help Aunt Tillie understand where she's really going on the web. Consider this screenshot:

In the lower right browser window, I'm on CSPAN's Booknotes.org site, where -- David Sklar reminded me -- you can watch Brian Lamb's interviews with authors. In the upper left window, I'm watching the George Soros program. Note the extra toolbar in that window, which says: You're on virage.com. That's CoreStreet's Spoofstick in action. In this case, CSPAN's relationship with media partner Virage is made plain in the pop-up window, even though the URL-line is hidden. But when bad guys are running the show, it's all to easy for Aunt Tillie to wind up in the wrong neighborhood without realizing it.

Spoofstick is a beta extension for Firefox, with IE support "right around the corner." (Didn't things used to be the other way around?) It fits right in with one of the the themes I've been developing lately: we need to standardize on the UI conventions that contextualize secure interaction on the web.

I don't think Spoofstick is a final solution, and neither do the CoreStreet folks. In this particular case, for example, what's Aunt Tillie to make of the fact that she's been transported by CSPAN to Virage? Is that OK or not? How's she supposed to evaluate all this?

In the case of a benign third-party relationship like this one, you could argue Spoofstick raises more questions than it answers. Nor would it surprise me if somebody discovers a way to spoof Spoofstick. But the principle at work here is sound. The information superhighway needs a standard system of roadsigns that Aunt Tillie can trust. The SSL lock was and is helpful, but we need to do more. Spoofstick suggests an important next step.

A growing number of vendors now use Flash videos to augment the obligatory lists of customers, features, and benefits that they publish on their marketing pages. It's a strategy I highly recommend. What hadn't occurred to me, until it happened this week, was that users might do this for you! [Full story at InfoWorld.com]

It's funny how these things happen. I put very little consideration into making that narrated demo. I had posted something about XSLT and said it was easier than advertised. In a weblog comment, someone asked for evidence to back up my assertion. I offered to make a recording, he took me up on the offer, and I spent 15 minutes with no post-production to respond to him. [Zope Dispatches]

The fact that Flash has become the de facto standard for such videos is interesting in light of Microsoft's "quietly announced" ¹ Channel 9. I'm hardly the first to point out that Channel 9's Windows-Media-only format excludes crucial audiences. Joe Wilcox says so here, and Robert Scoble responds here.

Joe's point is spot on. Although Larry O'Brien says he won't watch the videos, they're actually the only part of Channel 9 that I have tuned into. Of the first batch of videos, the one that I found most important (albeit not as entertaining as Bill Hill's Homo Sapiens 1.0) was Michael Howard's observation ² about how the computer science curriculum gives short shrift to security. His book, Writing Secure Code, is a remarkably candid, Cluetrain-like piece of work. In this passage, for example, he draws attention to past ActiveX-related screwups in Microsoft products:

That honesty, coupled with the book's exhaustive analysis and recommendations, makes Howard the best and most credible voice inside Microsoft on an issue that desperately cries out for credibility. But because of the format lock-in, he winds up preaching to the choir. A further irony was that Channel 9 asked me to accept a signed ActiveX control! The people who really ought to see and hear Michael Howard never will.

As for Robert Scoble's response, I dunno. "When we came up with the idea of Channel9," he writes, "we didn't just get unlimited resources to do everything perfect." Well OK, but QuickTime Pro is $30, and Flix Pro is $149. Using these, I was able to produce QuickTime and Flash versions of the Michael Howard clip. The quality's not great, partly because I couldn't figure out how to download the .WMV files behind the .ASX wrapper, so I resorted to a Camtasia Studio screen capture. And I'm not sure Microsoft would appreciate my posting alternate versions in any case, so I won't. But, though I'm far from an expert on video formats, it doesn't look like a budgetary or logistical issue to me.

No other company comes close to the transparency that Microsoft is achieving with its blog activity and now Channel 9. I've applauded such efforts and will continue to do so. But I'll applaud Channel 9 more loudly when its message can reach the unconverted.

¹ What's up with that "quiet" meme?

news.com: "Microsoft quietly launched a new site on Tuesday that combines blogs, discussion forums and other technology to improve communications with developers."

infoworld.com: "Microsoft has quietly expanded its Microsoft Developer Network with a Web site that combines a host of social networking technologies in a move to improve communications with outside software developers."

² Note that I had to dig these direct links to the videos out of the RSS feed. They're not directly available on the surface of this page. This is typical of MSDN Web designs that use video snippets, and I think it's un-Weblike and blogger-unfriendly.

The last time I asked Microsoft why there's no plan to make the InfoPath runtime ubiquitous, the answer I got was: "We don't hear customers asking for it." Well, I do. Here's a typical rant from one customer who, because his company has a relationship with Microsoft that he doesn't want to jeopardize, asked me to anonymize his comments:

I believe a primary requirement of a forms application is to make it possible for the form to be completed by a wide audience of people from whom I wish to gather data. A key driver, at least in the world of my customers, is to be able to distribute the form widely to people who aren't necessarily connected to the network and get them to fill it in and return it. I don't want to authenticate these people in my network. They won't install software on their computers just to fill out my form. They don't want to learn a new application.

It seems InfoPath has completely ignored the question of how the form will actually be filled in by the responder. There is no free viewer as there is with Adobe Acrobat. There is no ability to save the form template as an ASP.NET web form. It appears that Microsoft expects everyone to purchase a full copy of InfoPath--the complete form design application--just so they can fill out a form. They can't possibly believe the product will gain any traction with this licensing and deployment model, can they? [1] What are they thinking? [2]

So my main question is, is there any way to deploy InfoPath forms without putting full InfoPath on every desktop? [3] Do you know whether Microsoft understands this issue and are planning anything to address it? [4] The two applications that are widely available on everyone's desktop are a web browser and Adobe Acrobat, and it seems like it would be a good idea for InfoPath to support forms deployment via one of those means. Am I missing something here? [5]

My answers were "I don't know" [1], "I don't know" [2], "No" [3], "Apparently they don't see a problem and aren't planning to do anything" [4], and "We're in the same boat: I don't get it either." [5]

Yesterday's item provoked a flurry of responses. Steven J. Vaughan-Nichols, who wrote the Washington Post story I dissected, points out that the nature of his assignment precluded broader coverage, and that he'd otherwise gladly have included bloglines. There's been lots of chatter about bloglines lately -- Chad Dickerson mentions it today -- so I was interested to hear from Martin Thornell about another web-based product, Rocket RSS reader. Doubtless there are others too. An implementation of one of these licensed for behind-the-firewall use, as Chad suggests, would be handy. As a matter of fact, that's how I use Radio UserLand's reader. It's nominally a desktop product, but I run it as a server and authenticate to it over SSL.

Vaughan-Nichols' critique of .NET's performance raised hackles with several readers, include Mark Levison:

I'm doing smart client (no touch deployment) .NET development at the moment. I find that we've no trouble getting excellent performance out of our app. When we do have problems it is usually algorithmic. Jon, what .NET client side apps have you tried? SharpReader? RSS Bandit? NewsGator? Are any of these slow? Let's test claims like this before repeating them. [dotnetjunkies]

When I said .NET performance is "a real issue that will dog client-side .NET in the same way, and for the same reasons, that it has dogged client-side Java," I did not mean that I believe, as Vaughan-Nichols does, that use of .NET automatically means sluggish performance. In fact I don't think that. But the perception does exist, as it has existed for Java, despite evidence to the contrary (e.g., Eclipse), because there is also evidence to support it. Modern managed runtimes are a huge and necessary step forward, but the desktop is an unforgiving environment in which to deploy apps that depend on them. That's been a challenge for Java, and it's a challenge for .NET too.

Meanwhile, Russ Lipton brings me back to my original point:

Jon Udell reminds me yet again how pathetically inept we are at explaining technology so that normal human beings can make sense of it. As a result, normal human beings intelligently dislike the technologies that fascinate some of us. [Coffehouse at the End-of-Days]

This morning a story on RSS newsreaders appeared in the Personal Tech section of my local paper. The title was A simple program to 'refresh' the news; the byline was The Washington Post. I'm keenly interested in how the story of RSS is being told to Aunt Tillie, so I deconstructed this one with some care.

The first order of business was to find the article online so I could quote from it, and cite the URL in this posting. I went to washingtonpost.com, registered, and searched for the phrase "inefficient bundle of code"; we'll get to why I used that search in a moment.

The Washington Post is evidently even more restrictive than the New York Times. This two-week-old story is already parked behind the costwall, where you're asked to buy it for $2.95. No thanks. I did, however, learn that the original title was Refining Paperless News, and that the author was Steven J. Vaughan-Nichols.

When I'm looking for costwalled New York Times stories, I've noticed that you can often find them for free elsewhere. Sure enough, a Google search for "inefficient bundle of code" landed me here.

A couple of points in the article caught my eye. Exhibit A:

RSSReader (Win 98 or newer, free at www.rssreader.com) leaves out FeedDemon's price tag, but also its performance. It was easily the slowest newsreader we tried -- partially because it runs on Microsoft's .Net Framework, an inefficient bundle of code [emphasis mine] that lets developers add Web functions to their software. [Refining Paperless News (TechNews.com)]

I don't, by the way, entirely disagree with Vaughan-Nichols. Although I think he overplays the ".NET is slow" card here -- using it three times -- this is a real issue that will dog client-side .NET in the same way, and for the same reasons, that it has dogged client-side Java. But that's way more software-industry inside baseball than Aunt Tillie needs here, if the point of the article is to introduce her to the fundamental concepts and benefits of RSS, and acquaint her with the kinds of tools available for reading feeds.

Exhibit B:

Unfortunately, you can't just click that button to subscribe. You must right-click it -- on a Mac, hold down the Ctrl key as you click -- to copy the link's address, then paste it into your newsreader. [Refining Paperless News (TechNews.com)]

Exhibit C:

ADC Software's NewzCrawler (Win 95 or newer, $25 at www.newzcrawler.com) is perhaps the most flexible newsreader around. Beyond RSS, this fast, easily customizable program also collects and presents newsfeeds delivered with a newer protocol called Atom and postings from Usenet newsgroups.

One final observation. The article focused entirely on a single species of RSS newsreader: the standalone GUI program. If Aunt Tillie happens to be reading email in Outlook, she ought to have been made aware of the Newsgator option. An even more glaring omission was bloglines.com. Nowadays when RSS newbies ask me which reader to use, I point them to bloglines; it's the perfect quickstart. I tell folks they can deal with selecting, installing, and learning to use a "real" newsreader after they've gotten a taste of what RSS newsreading is all about. I don't tell them the reasons why, for certain advanced users of RSS, bloglines winds up being the "real" solution. That's too much information for an elevator pitch. However in an article of this length, which mentions Atom and harps on the performance of the .NET runtime, I think Aunt Tillie should have been told that Web-based readers exist, require no installation, can be used from anywhere, and are always synchronized.

My point here isn't to pillory Steven J. Vaughan-Nichols, whose work I've known and respected for a long time. All of us who belong to the geek tribe -- myself included -- tend to focus on our issues, not the issues that will matter most to Aunt Tillie. But we're the gatekeepers of this story. As syndication goes mainstream, we're the ones who'll be asked to explain it to Aunt Tillie. Here's hoping we can all put the geek stuff in its place and tell her what she really needs to know.

There is a place in Los Angeles I've never visited, but would love to: The Museum of Jurassic Technology. It is the subject of Lawrence Wechsler's delightful 1995 book, Mr. Wilson's Cabinet Of Wonder: Pronged Ants, Horned Humans, Mice on Toast, and Other Marvels of Jurassic Technology. One Amazon reviewer called the museum "a straight-faced, Andy Kaufman-esque joke, blending exhibits that look too nutty to be true, but are true, with outright hoaxes."

Sometimes the jokes are pretty broad:

The very first display you encounter is an exhibit entitled "Protective Auditory Mimicry." Together, encased under glass, are displayed a luminous iridescent beetle and next to it a similarly tiny iridescent pebble. The wall placard to the side asserts that over the eons this beetle has adapted to make precisely the same sound when threatened that this pebble makes at rest. [transcript of 1996 NPR radio documentary by Lawrence Wechsler]

I was reminded of all this on Thursday when, for hours, nobody seemed to know whether Google's GMail announcement was real, or was an April Fools day prank. It wasn't only the date of the announcement, but also its tongue-in-cheek tone -- "Search is Number Two Online Activity -- Email is Number One; "Heck, Yeah," Say Google Founders" -- that led many to conclude it must be a prank. Even one of the savviest observers on the scene, Doc Searls, was momentarily taken in. And when I posted my response to Doc's initial posting, suggesting that Google had executed a brilliant double head fake, I wasn't yet 100% certain that this was no hoax -- even though I had read John Markoff's story, datelined March 31, in the dead-trees version of the Times on the morning of April 1. Indeed, it was Doc's near-instantaneous correction, after receiving a call from a Google insider, that finally settled the matter for me -- and, I'm sure, for many others. It's interesting to consider why. I trust Doc Searls as much as I trust John Markoff, and it was Doc's site, not the Times' site, that first reported a Google source both acknowledging and dispelling the possibility of a hoax.

Here are the remaining questions. Did Google intentionally leverage the reality-bending April 1 tradition -- to which it has famously contributed -- in order to crank up the buzz surrounding the announcement? (Update: Doc says yes, based on this posting from Google employee Jason Shellen.) If so, was the strategy a brilliant PR coup, as I suggested on Thursday, or a colossal blunder, as Doc concluded on Thursday? In retrospect, I'm inclined to think Doc's right. But either way, the period of confusion on Thursday was a very weird time. The sensory apparatus that tells me what's going on in the world is a complex machine whose gears -- weblogs, newspapers, Google -- were grinding.

The Museum of Jurassic Technology wraps the frame of conceptual art around the experiences it delivers. But how do we frame what happened on Thursday? I'm reminded of a story a graduate school professor once told me. He was stationed in London, covering the art scene for Time Magazine, and went to Hyde Park to report on a work of performance art that was scheduled to happen there at a certain time. A bunch of people were milling around, waiting for the event to begin. Much later the artist finally arrived, surveyed the crowd, and asked: "Where do I sign?"

Update: Doc just wrote his post-mortem, in which he says: "I've long since lost my PR edge." No, Doc, I don't think so. I've changed my mind since Thursday, and I think your gut reaction was the right one.

Update: Bryan Field-Elliot thinks the whole thing was a feint to distract attention from Gmail's privacy policy.

A reader took me to task for suggesting, in this week's column, that we need to do a better job of spelling out the user-interface implications of Internet standards. Robb Beal agreed with me, though, and today I found another example of the kind of guidance that his functional annotations provide.

Last July, I mentioned RMX (Reverse Mail eXchange) in an article on anti-spam technologies. Since then there's been a lot of activity on this front. Now I'm looking into SPF (proposed by pobox.com), Caller ID for Email, (proposed by Microsoft) and Domain Keys (proposed by Yahoo, not yet published).

The various strategies for weaving authorization and email policy into the Domain Name System are quite fascinating. But I was also struck by this passage I found in the Caller ID spec:

Common historical practice in mail reading software regarding the mail originator and resent headers has been to present only the contents of the From: header to the users; the other related headers (Sender:, Resent-From:, Resent-Sender:) have not been shown. This behavior SHOULD change. Messages with combinations of identities in the originator headers SHOULD be rendered differently than messages in which the identities are the same. Specifically, it is RECOMMENDED that if the purported responsible addresses of a message is not the same as the address that would be rendered as the From: address that both these addresses be exhibited to the user. For example, the message in the example -3.2.3 might be presented by e-mail client software as being

From bob@forwarderexample.com on behalf of adam@example.com

or

From adam@example.com via bob@forwarderexample.com

instead of the historical

From adam@example.com

Exactly! Nobody should care what's jammed into the DNS TXT records used to authorize an SMTP sender, but everybody should care about dodgy identity trails. While acknowledging that such matters are "properly a role of mail filtering and e-mail client software," the spec nonetheless ventures "some suggestions regarding how that might work." Applause.

Or, maybe, look at the brilliant marketing strategist who was out-Cluetrained by a brilliant marketing strategy :-)

Update: Doc recants:

Just when I think I've given all the PR advice a former PR guy who's still a journalist can give, here's one more: If you're gonna shake the Earth with an unexpected announcement, don't pick the one day out of 365 when everybody's yanking everybody else's chain, okay?

The Flex strategy first began to crystallize two years ago when Macromedia rolled out the Flash 6 player, Flash MX development tools, and ColdFusion MX server. The possibilities were exciting, and the back-end environment was comfortably based on Java and Web services. But the client-side discipline was alien to the corporate programmer.

One obstacle was the ActionScript 1.0 language, which lacked the strong typing and formal class model that a Java programmer would expect. The solution to this problem arrived last fall when Flash MX 2004 introduced Flash Player 7 and support for ActionScript 2.0. Yet the Flash IDE was still built around the concept of making a movie, not coding an application. Flex presents a development model that will make immediate sense to an enterprise developer. [Full story at InfoWorld.com]

It's interesting to consider these two admirably compact implementations side-by-side. Some points of comparison:

This mixed pattern of green (good) and red (bad) pretty much sums up my conclusion. I want all the green stuff in one column. Actually, I want all the green stuff in multiple columns: Flash, Mozilla, .NET. Heck, if I want to write a tool for Groove 3.0, I should be able to use the same XML-based UI definitions, objects, and events as I can use everywhere else. At this level of abstraction, all this stuff is too similar to justify the differences.

We had a great thing going for about 10 years: the universal HTML/JavaScript client. And while it's still a great thing, there are good reasons to advance the state of the art. But can we please, please not lose the standardization that's served us so well?

Something wonderful died with Napster: the collaborative discovery and sharing of a wide diversity of music. Lucas Gonze is on a crusade to bring that experience back, legally. On his site, webjay.org, users share playlists -- i.e., lists of URLs that point to MP3s that are posted on artists' websites, or that are otherwise authorized for distribution on the Web.

My first (and so far only) Webjay playlist began as a couple of tunes by Betty Dylan, a Nashville-based duo who played my hometown recently and won me over with their energy and charm. Hunting around for more Betty Dylan tunes, I ran into some other Bettys -- Betty Roche, Betty Sue -- so I included them too.

Yesterday I noticed that the Betty Roche tune had migrated into one of Lucas' playlists, Streak of lean, streak of fat, and the Betty Dylan tunes had found their way into another of Lucas' lists, The Betty Destroyer.

In a recent blog essay, Lucas talks about the collaborative filtering dynamic he hopes to encourage:

There's one song in Treebot from Tofuhut, Yusef Lateef's Strange Lullaby. There's also one song from LargeHeartedBoy, Julie Doiron's mind-blowingly beautiful Pour Toujours, and that song had gone through three generations of filtering. In fact, every song in Treebot made it through multiple cullings, and that's why it's a good playlist.

It took Tofuhut to introduce "Strange Lullaby" into the ecosystem, and if he didn't have both taste and writing ability his recommendation wouldn't have made it through. But it always takes more than one person to do collaborative filtering. I want to make the path from obsessive record collectors to the average iPod as short as possible, and that's what Webjay does for him. [Lucas Gonze, 3/18/04]

Here's the business problem: I want to help music businesses sell products, then make my money on affiliate revenues. That way everybody's incentives are lined up in the same direction. The listeners are looking for the best music, I'm trying to find the music they'll like the most. Music businesses are looking for listeners charged up to buy, I'm trying to get the listeners charged up.

So how do I do it? An Amazon search for a song title? Amazon's product database isn't big enough (hard to believe, I know) and the lookup algorithms aren't smart enough -- I need a relevance match, not a keyword search. ISRC identifiers? Good luck getting them for online music, much less matching them to vendors. So help me out here, Music Industry: given a product and a buyer, how do I find a seller? [Lucas Gonze, 3/23/04]

There are a bunch of things that frustrate me about playlists. Competing formats: m3u, smil. Inconsistent behavior: if you want your tunes (and associated images) to render as you expect, you're looking at an insane test matrix. Crappy metadata: missing or incomplete, and often hard to find. Despite all these irritations I find myself returning to Webjay for the same reasons I write this blog and read others. What I know, I want to share with others. What others know, I want to know too.

If it's easy to buy music online, I sometimes will. But first it has to be easy to find, listen to, talk about, and share tunes. The intersection of blogs and playlists isn't yet nearly as smooth an experience it should be, but the ideas that motivate webjay.org are exactly right.

Apple, of course, wrote the book on human interface guidelines by visualizing and documenting a range of interaction scenarios in meticulous detail. Today we have a variety of platform-specific guidelines -- for Windows, for GNOME, for Flash MX. But we lack general guidelines for how Internet applications should behave on all platforms. E-mail programs don't agree on how threading, foldering, and filtering should work. Web browsers don't agree on how drop-down search boxes should work. RSS readers don't agree on how the orange XML icon should work. Media players don't agree on how playlists should work.

We need HCI (human/computer interface) guidelines more than ever. And we need them not only for Windows, OS X, GNOME, and Flash, but for the uber-platform that subsumes them all. We need human interface guidelines for the Internet. [Full story at InfoWorld.com]

Yes! Every technical spec that has user-facing implications should have a corresponding functional spec.

See my functional annotation of Mark Pilgrim's HTTP tests for an example.

Robb pointed me to some other examples as well. Why isn't this done more? Robb thinks it's because developers tend to want platform vendors to do this work for them. But even on a given platform, essential guidance about user interaction is often lacking.

Scanning the responses to my posting on S/MIME signatures, I realize some people took it as a condemnation of S/MIME. Not so. I was trying to illustrate how interactive context affects the implementation of a protocol, and how the nature of that context can be (but rarely is) specified.

I had suggested, for example, that a mail client displaying a signed message should always display the address in the From: header (not just a friendly name), should display a standard signature icon, and should link the icon to a certificate viewer. Outlook 2000 breaks the first guideline. Darrell Dykstra wrote to point out that Outlook 2002 and 2003 comply with all three guidelines, which is great. Except, of course, they aren't guidelines written down anywhere, and that's my point.

The other day, NPR's Day to Day ran a segment on phishing. In this clip, John Dimsdale interviews David Jevans, chairman of the anti-phishing working group, who says:

Typically it's the average consumer, who's quite Internet-savvy, and they get an email in that looks exactly like it came from their bank, with very compelling information -- it will have the logos, it will really try to fake the website.

Update: John Patrick on phishing:

The moral of the story is to be increasingly careful. Anti-virus and anti-spam are not enough. Anti-spyware is not enough. Hardware and software firewalls are not enough. All of these are essential but the other ingredient is common sense. Look at your email carefully. Even if the "from" address is one you recognize, look also at the context.

...

Digital ID's are essential to add authentication to email and software downloads. We need to be able to establish that we are who we say we are and to be sure that others (people, links, software) are who they say they are. You can read more about this in the patrickWeb Privacy and Trust series. [John Patrick: Phishing Update]

We are social animals for whom networked software is creating a new kind of habitat. Social software can be defined as whatever supports or amplifies our social behavior as we colonize the virtual realm. The category includes familiar things such as groupware and knowledge management, and extends to the new breed of relationship power tools that have brought the venture capitalists out of hibernation. [Full story at InfoWorld.com]

There's also an interview with Valdis Krebs and Gerry Falkowski. Valdis wrote me over the weekend with a correction to this bit:

IW: Does it cut the other way, too?

VK: We wouldn't take a job that we knew would lead to a resource action.

IW: Resource action?

VK: Layoff.

The pro-outsourcing arguments advanced by economist Daniel Drezner, writing in Foreign Affairs, break no new ground. I was struck, though, by this comment about anecdotal evidence:

When forced to choose between statistical evidence showing that trade is good for the economy and anecdotal evidence of job losses due to import competition, Americans go with the anecdotes. [ForeignAffairs.org, via Dan Gillmor]

He says he can't prove this attack was motivated by anti-outsourcing sentiment, but thinks so based on the fact that his website was also recently defaced with messages like "*&*&&** you have taken our jobs!"

Sigh.

Most of the reactions to my recent column on outsourcing, in which I interviewed MAPICS CEO Dick Cook, were favorable. To the minority of critics, I wrote back and asked: "How do you propose to deal with the situation?" No answers yet.

Meanwhile, an issue that was never abstract to me has become even more concrete. CNET's Builder.com will be outsourcing some of its "content production" to an editorial firm in India. The CNET spokesperson cited in news coverage of this story is senior editor Rex Baldazo, who worked for me at BYTE years ago.

I keep coming back to the exchange between Daniel Pink and Shirley Turner. "We've done it before," says Pink, "going from farm to factory, from factory to knowledge work, and from knowledge work to whatever's next." To which Turner responds: "I'd like to know where you go from knowledge."

Not, let's hope, to rock-throwing.

Håkon Styri figured out the answer to yesterday's puzzle. The page in question -- The analog hole -- does mention magnets. The text is hidden in the Strategic Developer widget on that page. As Hekon points out, that's very confusing. Indeed, it calls into question the common practice of decorating web pages with all sorts of auxiliary info-widgets.

The problem isn't just with Marc Barrot's nifty expanding activeRenderer widget, which I use in a couple of places in my standard template. Even visible text that's unrelated to the primary item on a page will cause problems. For example, you can't do an effective fulltext search of my blog for anyone whose name appears in one of my blogrolls.

In theory a CSS attribute could say: "Don't index this element." Parsing it would likely be more work than search engines are currently willing or able to do. But it's a competitive market again, and there's going to be a struggle to differentiate premium search from commodity search. "Don't index this element," in and of itself, isn't a feature to write home about. But if an Internet-scale engine could deliver the kinds of structured search I've implemented locally on this site, that would be a serious advantage. I wonder who'll get there first?

My referral log shows four visits today as a result of this query: research with circle hole magnets. It presents so many interesting questions to divert me from writing the column that's due today!

1. What was really intended? Maybe magnets "circular holes"?
2. Does anybody out there get the difference between magnets circular holes and magnets "circular holes"?
3. There were four clickthroughs to my page -- The analog hole -- from this query? Wasn't it evident, after the first, that it doesn't even mention magnets?
4. Why is the cached version of the found page empty?
5. Why is the found content:

   ... My refrigerator magnets still don't receive weather reports, but when they do, we'll ... CEO and Research Chair of the Burton Group, Jamie is a longtime industry ...

   actually from another item, Playing the Internet scales?
6. Why does the "refrigerator magnets" blurb appear in the summary of other pages too?

   Jon Udell: My conversation with Mr. Safe
   ... My refrigerator magnets still don't receive weather reports, but when
   they do, we'll need something like PreCache to make them work. ...
   weblog.infoworld.com/udell/2003/06/27.html - 101k - Cached - Similar pages

I can't answer questions 1 through 4, but I've got a hunch about what's happening with 5 and 6. Try this query: refrigerator magnets. The Atomz search engine, which I formerly used to search my blog locally (but never discontinued when I switched to InfoWorld's UltraSeek engine) appears to have suffered some kind of aphasia. When you search for "refrigerator magnets" it finds hundreds of articles, and uses the same summary for each. Doesn't happen with any other query I try, only "refrigerator magnets" (or "refrigerator" or "magnets"). Cool, huh?

Now, did Google find this refrigerator magnetized page? If it did, how exactly did the "refrigerator magnet" summarization glitch infect Google? By linking to that wacky Atomz query, from this posting, will I make all my Google summaries be about refrigerator magnets?

Alas, my deadline looms, so the answers to these pressing questions will have to wait.

The future of "great Windows applications," we're told, lies with Longhorn's next-generation presentation subsystem, Avalon, which will reboot software development sometime in the latter half of this decade. Of course, even Microsoft can't wait until then. Consider InfoPath. It's a great Windows application and a rich Internet client that had to ship in 2003. Its foundation is none other than Internet Explorer -- or rather, the suite of components and Internet standards on which Internet Explorer depends. Could InfoPath have been built on a Mozilla foundation instead? You bet. And the result wouldn't just be a great Windows application. It would be a great application, period. [Full story at InfoWorld.com]

It's easy to imagine a Mozilla-based organizer that does the kinds of things Onfolio does, but on all platforms. The UI would be handled by XUL, JavaScript, and components; storage would be handled (as in Chandler) by Berkeley DB XML. What's missing from this equation is the enormous value that I'm told (and that I believe) the .NET Framework is delivering to Onfolio. Of course Mozilla can use the .NET Framework on Windows too. The wild card in the deck is Mono, which could in theory deliver similar value on other platforms. One of these days, I guess somebody's going to dive in and test those technical (and legal!) waters.

In response to the Firefox column, Peter Traeg asked:

You mentioned in your article that you have been using Firefox to build apps that fetch, transform, and search XML documents. Do you have some links you could share with information on how to do this?

For those who don't want to switch (or aren't allowed to), another reader recommends a way to have, in IE, the kinds of enhancements that have been showing up in Mozilla:

There is a product called My IE 2 that adds not only a very flexible Tabbed Browsing interface to IE 6, but ad blocking, pop-up blocking and a whole lot more...It is an amazing piece of freeware and it does all of this and much more, including mouse gestures, in a very small, 700+ Kilobyte file that basically sits atop IE 6 and makes your browsing experience a much, much better one.

Finally, I've noticed growing interest in Firefox search plugins. Oracle guru Steve Muench is rolling his own, and Flash guru Mike Chambers has made a bunch of installable ones for searching Macromedia resources.

The other day I received an email message from jon_udell@infoworld.com, accompanied by a valid S/MIME digital signature. But the message wasn't from me, it was from David Wall (see earlier post), and here's what it said:

As mentioned here is a spoofed email that appears to come from you and is digitally signed. Note that I signed up using another person's email address, another person's SSN, another person's phone number, chose your name as the password for the key, etc. In other words, these "precautions" Thawte demands don't provide any real security any more than checking IDs will stop terrorism. Only the honest will comply.

And what's worse, the person who really has the SSN that I provided won't be able to get her own certificate now because I've locked it up, yet Thawte doesn't know who I am to resolve matters.

Ouch! This withering critique of S/MIME deserves a closer look. I was at first perplexed because I've tested S/MIME forgery myself, and have verified that when the From: header doesn't match the certified address, S/MIME-aware mailers tell you that the signature is invalid. So let's look at how David's trick works.

I began by retracing David's steps, because it's been a very long time since I originally signed up with Thawte -- a process which, as reader Matt Dirks notes, begins here. (Another reader, Dennis Wurster, pointed me to this overview of the signup process.)

Like David, I was able to use a random 10-digit number to satisfy Thawte's requirement for a "national ID." He's right: that's lame. The freemail cert does one thing, and one thing only: it binds a public key to an email address with minimal assurance. Thawte, like other certification authorities, will sell you certificates that offer more robust assurance. Only then, arguably, should official credentials -- SSN, driver's license, passport -- play a role in the process. I'd love to hear from Thawte (or another CA offering free S/MIME certs) on this point.

Here's the information I gave Thawte when I created my new account:

        Surname: Gates
      Forenames: Bill
    Nationality: American
USA National ID: xxx-xxx-xxxxx
      Thawte ID: JUDELL@MYREALBOX.COM
  Date of Birth: 1955/05/12

Cool, huh? It probably wouldn't occur to Aunt Tillie to click on the signature icon. If she did, here's what she would see:

In another context, that bit of misdirection doesn't work so well. Here's the same message in OS X Mail:

David Wall and I draw different conclusions from all this. Mine follows from last week's posting, standards versus conventions: we can't neglect the subtle user-interface details. For example, RFC2312 says:

Receiving agents MUST check that the address in the From header of a mail message matches an Internet mail address in the signer's certificate.

• A receiving agent that displays a signed message MUST display the address in the From header along with the friendly name.

• A receiving agent that displays a signed message MUST one of the standard signature icons: {URL}

• The signature icon MUST link to a certificate viewer.

Historically, of course, we don't spell these things out. When I suggested that perhaps we should, Marcus Ramberg suggested that I've been "taking a deep hit of the crack pipe":

For one, standards like this would conflict with UI standards on the respective operating systems the apps run on, and anyways, the point of making a standard is so entities can interact with each other. How applications should interact with users is a topic for UI Design 101. [Marcus Ramberg]

Paul Everitt's Zope Dispatches blog today features a narrated screen video that demonstrates oXygen, Paul's weapon of choice for wrangling XML and XSLT. I invite everyone -- and in particular the marketing folks at SyncRO Soft, Ltd (oXygen's maker) -- to compare what's happening on the oXygen site with what's happening on Paul's blog.

The oXygen site has all the familiar paraphernalia: a features and benefits list, a customers list, a bunch of articles and documentation. Yawn. OK, I should look into that, someday...

Meanwhile Paul, who's "merely" a user of oXygen, shows me and tells me what the tool does, and why he values it. The customers that the oXygen site lists are just names and websites that otherwise mean nothing to me. Paul, on the other hand, is someone I know. And even if I didn't know him personally, I could get a sense of the guy by absorbing the identity he's projected into his blog over time. So his recommendation feels personal.

Reading his commentary on the screen video he made, I hear the voice of experience and the ring of truth:

FWIW, Komodo is a nice XML environment as well. It has the one feature I miss the most in oxygen, which is an XSLT debugger. This is just wildly useful in Komodo: set a breakpoint in an XSLT file, and watch as the result document is rendered, stepwise. Still, oxygen makes a nicer XML environment, as it is really geared towards XML semantics (such as enforcing the XSLT schema and learning structure).

The fact that Paul's assessment of oXygen includes a comparison with Komodo (and an implicit criticism oXygen) makes his final recommendation all the more credible. As does the fact that an oXygen user liked the product enough to spend time and effort demonstrating it to all interested parties on his blog.

Very, very cool. It reinforces my hunch that the combination of easy-to-create blogs and easy-to-create narrated screen videos could put users in charge of software marketing, education, and training.

For almost a decade I've used the Web -- and most recently my blog -- to research, develop, and enhance the articles I write for magazines. When I ran into Dan Gillmor at SXSW we discussed some of my strategies, and Dan asked me to write them up. Seems worth doing, so here goes. Much of this concerns the IT trade pub ecosystem specifically, but I think the principles will generalize. The basic pattern is simple: a story gestates in blogspace, appears in print and online, and then matures in blogspace.

Pre-publication phase: Announce story on blog, publish draft outline, solicit feedback. The preview of my .NET cover story was a good example of the role the blog can play in the pre-publication phase of a story. Among the purposes served by that posting:

• Validate the idea. There's a lot of complaining, lately, about the "echo chamber" effect in the blogosphere. But in this case, the blog is a way of breaking out of another kind of echo chamber: the editorial ivory tower. Every magazine has some version of the editorial meeting, a session in which ideas are pitched and vetted. The external feedback loop that governs this process is highly attenuated, though. If an idea was incomplete, or poorly focused, you'll hear about it from readers -- but only after the article hits print. Since readers are stakeholders in this process, I figure I should involve them up front. This makes particularly good sense in the realm of IT trade journalism, where we writers serve as proxies for the readers. I enjoy privileged access to vendors, but with that privilege comes a responsibility to ask and answer the questions that matter to readers. By operating transparently, in blogspace, I invite my reader-stakeholders to keep me on track.

• Gather expertise. I start with topics to which I bring a certain amount of expertise. Then I leverage what I know (and who I know) to find what I don't know (and who I don't know). Of course in the trade magazine business, there is a whole profession dedicated to helping me do that. When a story appears on the editorial calendar, I'm swamped with phone calls and emails from PR folk who want to supply me with analysts, executives, domain experts, and customers. This isn't necessarily a bad thing. I sometimes accept these opportunities, and in some cases, I learn from them. It's dangerous, though, to be led down the path of least resistance. So I rely on the blog to find other people who have important things to tell me. As you can imagine, this makes PR folk really nervous. It's their job to try to control my story. It's my job to route around that control, and the blog is a tremendously powerful tool for doing that.

• Focus the PR energy. The journalism/PR game is made more antagonistic that it needs to be when there's insufficient data in play. For example, I neglected to blog proactively about the e-forms story we ran in January. As a result, the PR people were forced to rely on our editorial calendar, which described the story as something like "Life of a document." They concluded, not irrationally, that it was going to be a story about document management. And then many took a further leap of faith and figured that, given the impending Sarbanes-Oxley deadline, I should write a story about document management systems that help companies comply with that legislation. I can't tell you how many calls and emails I got inquiring about my "Sarbanes-Oxley story." But this was really my fault. Had I spelled out my intention -- which was to compare the Acrobat, InfoPath, and XForms approaches to e-forms -- I'd have spared a bunch of people from making phone calls and writing emails that were as fruitless for them as they were annoying to me. And I'd have encouraged the folks who really should have been contacting me to do so.

• Dialogue with vendors. In the IT trades, readers aren't the only stakeholders. Vendors are stakeholders too. They're creating products and services that, over the years, have grown steadily more complex, and more difficult to understand fully and explain well. They rely on trade pubs to help get the story out, but the pubs have less and less space for detailed explanation and analysis. There's much more to say than InfoWorld (or any other trade pub) has room to print. By narrating my evolving views in my blog, I invite everyone -- including vendors, who are of course the best experts on their own stuff -- to help me refine those views. That give and take yields valuable insight and -- when it can take the form of cross-blog conversations (i.e., isn't secret, as many things aren't) -- valuable content. A tip of the hat here to Microsoft, by the way, whose developers are miles ahead of their counterparts at Sun, IBM, Apple, and elsewhere when it comes to engaging with the blog medium.

• Promote the story. I hadn't thought about this until recently, but blogging the run-up to a print story can help create buzz. That may matter less to a controlled-circulation magazine like InfoWorld than it would to a newsstand pub, but it's still an interesting notion. When you've got a major story on an evergreen topic -- one that isn't going to break news or reach shocking conclusions -- opening up the process a bit may be a useful marketing strategy. That's what movie-makers do, after all, and the magazine game is a species of show business.

Post-publication phase: analysis, feedback, enhancement. Since the advent of the Web, magazine sites have used the "TalkBack" device to enable readers (and authors) to comment on stories. This was a great way to work around the severely-bottlenecked "letter-to-the-editor" medium. In the blog era, there's another way to skin this cat: aggregate what readers (and authors) say on their blogs about the published article.

I think we'll see more of this TrackBack-like approach as time goes on. In fact, InfoWorld.com takes a step in that direction, following a suggestion of mine. Blog entries that reference InfoWorld.com stories, found by way of Feedster and Technorati, are collected into a database. Then a selected few are shown on every page, in a box labeled "Top Site Referrals." I find this label confusing, and would rather see something like "Bloggers talk back." But that wouldn't work well either because, currently, the items appear sitewide, not per-article.

InfoWorld.com doesn't have the resources to collect all the substantive blog postings (and letters to the editor) that relate to each published article, and use them to advance the story in a coherent way. But as the author of a few of those articles, I have the bandwidth -- and the motivation -- to do exactly that. Here are some of the ways the blog can add depth to a printed story.

• Respond to readers. I used to mention my published stories on the blog immediately. Lately, though, I've decided to let InfoWorld's RSS feeds announce the stories, and hold my posting until I've had a chance to collect and process email and blog feedback. Last week's column on email identity is a case in point. It posted to the Web on Friday the 12th, but it wasn't until a week later -- last Friday -- that I'd gathered enough feedback to support a substantive follow-up.

• Publish interview out-takes. I've used the blog to expand on published interviews with various people including Ward Cunningham and Dick Cook. I was going to add Jean Paoli to this list, but when I went back and looked, that entire interview ran as a column. Interestingly, Phil Wainewright saw where this was going even before I did. He originally wrote:

  This is cutting-edge journalism, by the way -- neither a finished article nor a weblog entry but something in-between that would never have happened without the influence of weblogging or the convenience of online publishing -- an analytical journalist publishing his interview notes accompanied by his reflections on them. [Loosely Coupled]

  Then, when he realized that in this case the interview-plus-reflections appeared in the column, not the blog, he added:

  Jon subsequently noted that the article was his weekly column, so I shouldn't really have implied that it was less than a finished piece. But I almost wish that I had been right, because the idea of supplementing traditional published formats with new ones appeals to me.

  Me too. Phil was right, just not about that particular example. His comment helped crystallize the approach I've taken with subsequent interviews, and plan to continue.

• Publish demos and examples. My item on secure use of private keys, which featured screen videos of advanced private-key security configuration in OS X and Windows, was a companion to the column on email identity. It's time-consuming to do this kind of thing, but with more practice using the capture/edit tools, and some refinement of my presentation skills, I hope to be able to make it happen more routinely. Clearly you can't do this in print, but it makes a powerful complement to the printed article.

The rhetoric swirling around blogs and journalism often takes an adversarial tone. One of the reasons for that, I think, is the relationship of the two cultures to their primary sources. Bloggers feel obliged to cite them, journalists often don't. A startling example of this was the Dan Geer incident, which revolved around a PDF report on the Web. Every blogger who commented on the matter linked to that report. No conventional journalist did.

I won't always report everything that someone said to me, or cite every information source I've consulted, because I'm trying to tell stories here, and I want to keep the narrative lively. But using the blog to open a window onto my primary sources before, during, and after the publication of an article helps me -- and the various stakeholders -- in all sorts of ways.

I mentioned the other day that OS X Mail and Outlook handled a DoD email certificate differently: OS X Mail trusted the cert, and Outlook didn't. The obvious explanation -- that OS X has the DoD root certificates pre-installed, whereas Windows doesn't -- somehow never occurred to me. But according to Daniel Dulay, that is indeed the case:

I have worked in the computer security field in the past, and I have experience with deploying PKI in enterprises. I also have had a little exposure to the DoD smart card, the Common Access Card or CAC card. I'd like to comment on your story about receiving an email signed by DoD user and your description of Mail.app as "questionable" for having trusted this digital signature.

First, a kludgy little trick I learned in OS X. Do you know how to read the certificate authorities that Apple has shipped with Panther? The certs are stored in /System/Library/Keychains/X509Anchors and /System/Library/Keychains/X509Certificates, and you may use Keychain Access to read these files. In Keychain Access go to File -> Add Keychain... and point to one of these files. I should add the caveat that I have always made a copy of these files first because I don't know how robust Keychain Access is or if this functionality is supported by Apple. (Another way to access these files is the command line certtool utility. See "man certtool" for some surprisingly detailed documentation.)

So if you open up the cert authorities, then you will find that the DoD certs are already installed on your system! This is why Mail.app trusted the digital signature from the DoD. Your windows box probably did not have the DoD cert installed (I know win2k does not, but I am not sure about XP).

Why are these certificates already there? Because Panther is supposed to have CAC card support built in! I have not seen it for myself, but you can find some tools under /usr/libexec/SmartCardServices. Panther is supposed to support smart card logins, and I assume that a smart card's certificates can be used with Mail.app or Safari. There is a detail-free article on Apple's web site, http://docs.info.apple.com/article.html?artnum=152235, and I would love to find out more.

Interesting! I checked and sure enough, OS X trusts a bunch of DoD root certification authorities. Who would have thunk it? Thanks, Daniel.

I've watched with bemusement as Bill Gates has been making the rounds lately -- the World Economic Forum, the RSA Conference -- to announce that Microsoft is "innovating on many different fronts" to eradicate spam. Really? The hashcash scheme, which requires the sender to spend CPU cycles, dates back to about 1992 or so. And "caller ID for e-mail" derives from RMX (Reverse MX), a more recent proposal to bind senders to authorized relays via DNS records.

The truth is we've had plenty of innovation over the years. What we've lacked is follow-through. Consider S/MIME digital signatures. It's very likely that your e-mail client supports them. But it's overwhelmingly unlikely that you've ever digitally signed an e-mail message. [Full story at InfoWorld.com]

There's no excuse for not having checked that myself. Typically I do. I've probably installed more digital certificates, in more browsers, on more operating systems, than anybody. But sadly I was willing to believe that the painful procedure outlined in that O'ReillyNet article was necessary because, well, that's the universal experience of S/MIME. Everything's ten times harder than it should be.

Whenever I write something about digital signatures, a handful of folks are inspired to send me signed messages, and since this happens so rarely, I always learn something new. One such message came from a DoD employee, who wishes to remain anonymous. His was the first cert I've ever received from a DoD certification authority. Outlook and OS X Mail, as it turns out, have inverse policies for dealing with this case. Outlook refused to trust the cert until I explicitly approved the issuing DoD CA. OS X Mail, questionably in my view, trusted it implicitly.

Anyway, the DoD guy had written to me to find out how to require per-message passwords, an advanced feature I describe in the column. In his office they use smartcards. When he hits Send in Outlook, he's challenged once for the smartcard PIN. Subsequent access to the signing key requires no further interaction. He's concerned about walking away from the machine and leaving signing enabled. For that, at least, there's a solution: yank the card when you walk away. But I'd add another concern: that a piece of rogue software could show up even while he's sitting there, and silently impersonate him. Clearly you're not going to yank the card during a session. So per-message confirmation of access to the private key -- which I've now also learned how to do in OS X Mail -- seems like a good idea to me.

But according to this fellow, what I have been considering a feature of Outlook is actually thought, by Microsoft, to be a bug!

If I am understanding the document at the following URL properly, Microsoft considers it a BUG if you get asked for your password before sending each digitally signed message (using Windows XP) and they have a BUG FIX so it will STOP asking you each time. This seems BACKWARDS to me from a security standpoint!

http://support.microsoft.com/?kbid=821574

Finally, I received this thoughtful response from David Wall, chief software architect of Yozons Inc., which I quote with permission:

You are to be commended for fighting through the free email certificate acquisition and installation process. And to think you just have to do it again next year. Or when you get another computer. Or you want to send email from your office, laptop and home computer using the same email address. Or when you change your email address, and you realize there's no way to invalidate the certificate for the old email address.

And if just you and the rest of the world would actually do this complicated process, S/MIME would finally become useful for email, provided all those desktops were secure enough to keep hackers and virus writers from stealing your keys. Also, if you encrypt on your desktop using a recipient's public key, you'll likely be violating corporate policies because the company will not be able to meaningfully audit or archive the encrypted message.

But do you suppose free email certificates wouldn't be free today if people actually wanted them? They are free because nobody will pay for them, and even at the cost of nada, few actually do. I think this points out that people as a whole just can't work with PKI's complexity, portability and constant renewal hassles. Have you ever tried to validate a digitally signed email from a few years ago? Do you really have the certificates that went with old message today? And even if you're one of the rare folks who actually keeps all of these thousands of certificates -- one per email address per year does add up quickly -- because they expire, you will get signature failures and have to note that the error was related to expiration and not because it was tampered with or the cert was revoked.

T-Mobile is our most recent large business customer. There are working alternatives today, like our Yozons business private network, and unlike S/MIME, they can also produce legally recognized electronic signatures as well as keep messages secure, provide full tracking and auditing, and there's no fuss about installing, revoking or otherwise keeping digital certificates current and secure.

The S/MIME problems that David Wall cites are quite real. And since we have so far failed to tame these problems on the public network, we are -- quite rationally -- retreating to various kinds of private networks. Yozons' solution is one example. Groove is another. Aggressively-whitelisted email services are yet another. It's far more practical to establish trust within private networks than on the public network, and there are very good reasons to do so.

But private networks are islands. We ultimately need a workable trust solution for the global public network. That's clearly a daunting challenge. PKI is only a first draft of the solution. It's possible that that we'll need to rip it up and start over. It's also possible, though, that we can refine and improve it. But not if current implementations don't evolve in response to use.

The word used again and again lately to describe distributed information systems is "composition". The Unix idea of piecing together solutions from reusable parts has morphed into XML-based, service-oriented architecture. This time around, though, it's all happening on the Web, in an environment where everybody can compose simple and popular tunes. When technologists forget that, I hope users will administer the dope slap we deserve. [Full story at XML.com]

I really enjoyed meeting and hearing from my fellow panelists. They're on the forefront of reinventing the institution of the library and the profession of librarianship. My message to them, and to people in every other profession, is: expect spontaneous integration. My message to IT propellerheads: don't disappoint that expectation. Larry Wall got it right: hard things should be possible, and easy things should be easy.

Alf Eaton looked inside a Magnatune MP3 file to see what metadata is really contained there, and concluded that the media players indeed can show all of it: artist, title, date, and in the comment field, the text "magnatune.com." I checked and Alf's right: additional info about licensing and purchasing doesn't seem to be present.

Let's assume that the ID3 spec spelled out, in precise detail, how a company like Magnatune would embed its licensing and purchasing hooks into an MP3 -- in some more specific way that just dumping extra text into the comment field. From the perspective of the spec writers, it's case closed. Black and white. Either you conform to the spec or you don't. Done deal.

Except here's what these specs never talk about. In QuickTime/Mac, to access this metadata, I use the Get Movie Properties function ("Movie Properties" for a music track?), and then look inside Annotations. In RealOne/Mac, it's Window->Clip Info. In iTunes, File->Get Info. (In MediaPlayer/Mac, it's...never mind, can't seem to get that one to work at all.) At least the platform convention, Apple+I-key, invokes these differently-presented "get info" functions in a standard way.

Meanwhile over on Windows, another set of behaviors. Quicktime: Get Movie Properties->Annotations (Control-I). Real: File->Clip Properties->View Clip Info (Control-I). WinMedia: File->Properties->Content. WinMedia seems to lack an accelerator key. Arguably it's not needed, since WinMedia runs the metadata as a CNN-style crawl. But then, arguably, it is needed, because a License or Buy option would require a context for interaction, like a dialog box.

So here's the point, and I see the same thing in other metadata standardization efforts such as the RSS/Atom fiasco. Technologists focus on formats and APIs, because that's what we know. How users will interact with the formats and APIs is left as an exercise for the implementer. But of course that's where the rubber meets the road. So syndication still lacks a well-known mechanism for one-click subscribe. Online music lacks a well-known mechanism for one-click licensing or purchasing.

This is a crucial kind of standardization that tends to fall through the cracks. The IETF, W3C, and OASIS don't deal with such matters. Who could, and who should?

By way of Christopher Allen, I got to meet John Buckman here at SXSW. John founded Lyris, a company whose hosted email list services I have used on behalf of clients. Although I prefer RSS to email as a direct marketing tool, the latter isn't going away anytime soon. So it's been a pleasure to rely on Lyris, a service that runs with impeccable integrity. John's new venture is Magnatune, an online record label I discovered a few months back whose endearing motto is "We are not evil." Equally endearing is this snippet from Magnatune's purchase page:

How much do you want to pay?
$5$6$7$8 (recommended)$9$10$11$12$13$14$15$16$17$18
(50% goes directly to the artist, so please be generous)

Interestingly, when given a choice (and the assurance that artists will be properly rewarded), users sometimes choose to pay more than the suggested amount.

I got to wondering how Magnatune and Webjay might work together. Webjay is Lucas Gonze's idea, a site whose tagline is "Listener-created radio." Nothing prevents me from extracting an MP3 URL from a Magnatune playlist and including it in a playlist that I publish on Webjay. But is this fair to Magnatune? The interstitial ads that Magnatune uses are included in their playlists, but not embedded in the individual MP3s.

I asked Lucas and John (via email) to consider what would be the fair and right way to contextualize a Magnatune MP3 in a Webjay playlist. And just as I sent that message, Chris Allen -- who works with Magnatune (along with a bunch of other interesting ventures) -- sat down with me, here in the hallway at SXSW, to clarify his take on the matter. The 128kbps MP3 streams served up by Magnatune are made available under a Creative Commons Attribution-NonCommercial-ShareAlike license. So according to Chris, it's perfectly kosher to include them in playlists that you publish. The idea, says Chris, is that the ID3 tags embedded in the MP3s are sufficient to let listeners find out about Magnatune and its purchasing and licensing options.

So I did the experiment, and it was a complete failure. None of the players on my PowerBook -- not iTunes, not RealPlayer, not QuickTime -- presented this metadata. That's hardly surprising. The various media players are, collectively, a train wreck. Publishing Web content that works in a standard and reliable way, in any browser, is a walk in the park compared to publishing AV content that works in a standard and reliable way in any media player.

We can't blame the problem on the record labels. It's the computer industry that gave us this fragmented and broken media platform. Now, suddenly, there's an explosion of content that can legally be ripped, mixed, burned, and blogged. The RIAA isn't the problem here. We need to find our way out of the QuickTime/Real/WinMedia/Flash fireswamp.

An article in today's New York Times features this Amazon-derived network map by social network analyst Valdis Krebs. It's another fascinating illustration of an idea that Krebs mentioned when I interviewed him for the O'Reilly Network in mid-2002:

Given good pictures of social networks, what will we use them for? Valdis Krebs has lots of practical ideas. For example, consider Amazon's related-book feature. If you follow these links a few steps out, says Krebs, clusters emerge, and sometimes those clusters represent disjoint interests connected only through one book. He offers Thomas Petzinger's The New Pioneers as an example. It connected two different groups -- one reading books on business and strategy, the other reading books on complexity science and chaos theory. Now there are a number of books that broker that connection, but Petzinger's was one of the first popular books to do so, according to Krebs.

The general principle at work here, Krebs says, was articulated in Ron Burt's Structural Holes: The Social Structure of Competition. It states that networks with "holes" -- that is, unbrokered connections -- present the most opportunity. A successful actor is one with ties to many points in the network who can uniquely fill one or more of those holes. To that end, Krebs -- who is writing a book on his experiences with social networks and business organizations -- plans to mine Amazon, map out the communities of interest relevant to his themes, and tune his presentation to optimally broker among them. [WebServices.XML.com: Seeing and Tuning Social Networks]

In today's Times story, Krebs identifies Bush at War and Sleeping with the Devil as the current political books that are being read by conservatives and liberals alike. Will publishers begin to apply this strategy consciously, as Krebs suggests might be possible? Filling the "structural holes" in networks, and creating large audiences from sets of smaller ones, is a fascinating idea -- though I'm sure it's easier said than done.

Valdis Krebs will also appear in InfoWorld's March 29 issue, by the way. For a feature on social software, I spoke with him and his business partner Gerry Falkowski about their use of social network analysis inside large enterprises such as IBM.

The other day Robin Good posted a link, via George Siemens, to a Register article by Scott Granneman. The article illustrates Google queries that find passwords, web-accessible databases, and financial data. Nobody should be surprised by what these queries reveal, but I'm sure a lot of folks will be.

A couple of websites have even sprung up dedicated to listing words and phrases that reveal sensitive information and vulnerabilities. My favorite of these, Googledorks, is a treasure trove of ideas for the budding attacker. As a protective countermeasure, all security pros should visit this site and try out some of the suggestions on the sites that they oversee or with whom they consult. With a little elbow grease, some Perl, and the Google Web API, you could write scripts that would automate the process and generate some nice reports that you could show to your clients. [The Register: The Perils of Googling]

Indeed. What does surprise me is that there isn't a well-known tool for doing this. It would be the 21st-century equivalent of SATAN, the first security scanner I pointed at my website back in the mid 1990s. Or more recently, Nessus.

Perhaps such a tool is well-known, but not yet to the good guys? It would be really useful. The mechanism, as Granneman points out, is trivial, but assembling the database of vulnerabilities isn't. If a credible project has formed around this idea, I'd like to know about it.

I've added a few more search engines to Firefox, and I'm parking them here so I can easily transfer them to my other machines.

• Safari Books Online

• InfoWorld

• Jon's Radio

Here's the procedure create these plugins, by the way:

1. Capture the image. To do this I fetch the favicon.ico file from the site's root, and use ImageMagick to convert it to a GIF.

2. Write the control file. For example:

   <search 
      name="Feedster"
      method="GET"
      action="www.feedster.com/search.php?btnG=Search&sort=date">
   <input name="q" user>
   </search>
   

   The action is the query URL minus the query parameter, in this case "q" -- it goes separately as part of the <input> tag. When a site uses POST instead of GET, you'll need to dig a bit deeper to come up with the query string. I used to use the LiveHTTPHeaders extension. Even better, though, is Chris Pederick's wonderful Web Developer Extension which does all kinds of handy things, including converting between GETs and POSTs.

To add a plugin, just drop a pair of these files -- the image and the control file -- into Firefox's searchplugins directory. The additional step I'm illustrating here -- one-click installation of the plugin -- depends on a snippet of JavaScript:

function addEngine(name,ext,cat)
{
  if ((typeof window.sidebar == "object") && (typeof
  window.sidebar.addSearchEngine == "function"))
  {
    window.sidebar.addSearchEngine(
      "http://weblog.infoworld.com/udell/gems/"+name+".src",
      "http://weblog.infoworld.com/udell/gems/"+name+"."+ext,
      name,
      cat );
  }
  else
  {
  alert("Netscape 6 or Mozilla is needed to install a search plugin");
  }
}

Note that there's a registry of these plugins. And I should probably register the Safari plugin there. But I'm sure this blog isn't searched often enough to warrant registering a "Jon's Radio" Firefox plugin. For such cases, it's nice to know that a more decentralized, ad-hoc solution is available.

Update: One reader wondered where the search plugin dropdown list is hiding. In plain sight. Though I'll agree it's easier to miss than Safari's equivalent, which remembers search history. Hmm. Would it make sense to offer both funtions? One handle to drop down the list of engines, and another to drop down the recent searches for that engine. Nah. Too cluttered, probably.

I finally got around to installing a digital certificate on OS X, so I can sign email messages in Panther's Mail app as I always do in Outlook on Windows. The recipe for acquiring and installing the cert is, unfortunately, guaranteed to scare away Aunt Tillie. But if you've gotten that far, you might want to consider an extra step to secure the use of your private key.

In Outlook, I've set things up so that messages are always signed. What's more, I have to type a password to unlock my private key each time I use it to sign a message. If the signature is going to be meaningful, I want to be sure -- and I want you to be sure -- that some piece of rogue software hasn't coerced Outlook into using cached credentials. I also find the extra confirmation step helpful, in the same way that a real signature can be. Even though it becomes an automatic reflex, it's not a completely unconscious act. And I don't send so many emails in a day that I can't afford a few seconds to consider the consequences of my words.

Achieving this effect in Outlook is wildly obscure. Once the cert is installed, I haven't found a way to up the security to require a per-use password. It's only when requesting the cert that you're given that option. Here's a movie that shows how it works when requesting an Outlook S/MIME cert from Thawte.

The analogous procedure in OS X is nicer. Here's a movie showing how to twiddle the settings on your private key, in Keychain Access, in order to require the keychain password (not, as in Outlook, a per-key password) when signing. And this movie shows the result: you have to type the keychain password in order to send a signed message.

I used the trial version of Qarbon to make these movies. Based on the comments I see here, it seems that Macromedia's RoboDemo should be the next screen video tool I try.

Aparna Jairam

Shirley Turner

The February issue of Wired features an article on offshoring by Daniel Pink, author of Free Agent Nation. Wired's story, entitled The New Face of the Silicon Age, might instead have been called Free Agent World. Here's a stunning exchange between Pink and New Jersey state senator Shirley Turner:

I toss a slur across her desk. I call her a protectionist.

"Oh, and I'm proud of it," she responds. "I wear that badge with honor. I am a protectionist. I want to protect America. I want to protect jobs for Americans."

"But isn't part of this country's vitality its ability to make these kinds of changes?" I counter. "We've done it before - going from farm to factory, from factory to knowledge work, and from knowledge work to whatever's next."

She looks at me. Then she says, "I'd like to know where you go from knowledge." [Wired: Kiss Your Cubicle Goodbye]

Where indeed? I think protectionism is the wrong approach. And I think Dick Cook's ideas are right. But let's not kid ourselves. What's at stake here isn't just call-center jobs, or mind-numbing code-writing jobs, or accounting jobs. Creativity, innovation and hard work are the levers that move the global economy, and anybody, anywhere, will be able to grasp those levers.

When I was in kindergarten, my family lived in New Delhi. It was a magical year in which I made permanent memories of the sights, sounds, and smells of India. A decade ago I returned to India for a tour of its software industrial parks. That visit changed me in another way. I met programmers and tech journalists who were my equal or better in every way, but whom you'll likely never hear of unless they're profiled in an article such as this week's cover story. Their faces and their voices became permanent memories, too. For me, the offshoring debate isn't abstract. I know that it turns on a mere accident of geography. [Full story at InfoWorld.com]

Dick said a lot more in our interview than I had room for in the column. Here are some outtakes:

On jobs data: Everybody in this political season is jumping on offshoring but although you can find anecdotal information, it's hard to find real data. I've looked hard, but neither the Bureau of Labor Statistics nor anybody else can give me concrete evidence that this world movement of jobs is netting down as much as everybody perceives.

On jobless recovery: It doesn't necessarily mean we're moving jobs offshore, it means we're workign more efficiently. Me and MAPICS (an ERP package) are probably one reason for that. People pay a lot of money for our software to be a cause of that. In this new world, the customer places an order online. The order department doesn't need to add people to handle more orders. Customers can check order status online. I have 175 customers in the furniture industry, there is a 6- to 8-week lead time. Three weeks prior to delivery, every customer calls the manufacturer or the store to ask when it's going to ship. By creating tools, we allow the manufacturer to automatically send a shipping notice 2 or 3 days before when the stats tell you would be the day the customer would call.

On China: In 2008, they're going to surprise everyone. The government and the people all realize they'll be on televisions in every home in the world for 17 days, during the Olympics. And they intend to present themselves, not as the largest developing country in the world, but as the largest developed country. We met with the Olympic planners. There are two goals. First, if you can believe it, is to be the green Olympics -- shutting down coal-fired power, building a big dam for hydro. Second, to be the digital Olympics. They're laying fiber everywhere, and they fully anticipate you'll use an ID card as your main security device and to charge meals.

The unanimous response to my question "Why no 'use strict' in Python?" was: PyChecker. Thanks to everyone who pointed me to this excellent tool.

The first person to respond to my query was David Ascher, architect of ActiveState's Komodo. Why, I asked David, isn't PyChecker included with the standard Python kit, and accessible by way of a command-line switch? David's response (via email, quoted with permission¹):

I suspect that it goes something like this:

1. the parsing infrastructure was developed with two goals in mind: correctness and speed, and maintaining extra data that you'd need for doing linting wasn't high enough priority early on.
2. the "right" way to do it is to use the new compiler system
3. since pychecker "works", the incentive to do it right is only one that appeals to those people in pursuit of beauty for its own sake.
4. Those guys are busy.

Another noteworthy comment on this subject comes from Ted Leung:

I don't know the history behind various Python features, so I can't comment on strict. What I can comment on is that strict is nice, but a type inferencer for Python would be better (as I've posted before). One of the reasons that I'm excited to be going to PyCon this year is Michael Salib, an undergraduate at MIT has written Starkiller, a type inference engine for Python. [Ted Leung on the air]

Fair enough. Based on the email I've been receiving, though, it's clear that I'm not the only Python programmer who's been unaware of PyChecker. Evidence suggests that it might deserve to be elevated to a command-line-accessible option.

¹ Emails from Ross Mayfield, CEO of Socialtext, include a .sig that ends with:

this email is: [ ] bloggable [ x ] ask first [ ] private

Yesterday I had the opportunity to speak with Anders Hejlsberg, father of both Turbo Pascal and C#. Of course I had to scratch my dynamic language itch, so we talked some about that. The upshot is that Anders believes compile-time type checking is valuable, but also thinks we can (and probably should) use type inferencing to make static type checking feel more dynamic.

During our conversation, he reminded me of an issue that I've been meaning to ask the Python folks to comment on. To illustrate it, consider exhibits A, B, and C.

Exhibit A. This Python program produces no compile-time error when the misspelled variable aNyme is referenced. It produces an error at runtime.

$ cat loose.py
aName = 'abc';
print '[' + aNyme + ']';
$
$ python -c "compile(open('loose.py').read(),'loose.py','exec')"
$
$ python loose.py
Traceback (most recent call last):
  File "loose.py", line 2, in ?
    print '[' + aNyme + ']';
NameError: name 'aNyme' is not defined

Exhibit B. This Perl program produces no compile-time or run-time error.

$ cat loose.pl
my $aName = 'abc';
print "[" . $aNyme . "]\n";
$
$ perl -c loose.pl
loose.pl syntax OK
$
$ perl loose.pl
$ []

Exhibit C. This Perl program produces a compile-time error.

$ cat strict.pl
use strict;
my $aName = 'abc';
print "[" . $aNyme . "]\n";
$
$ perl -c strict.pl
Global symbol "$aNyme" requires explicit package name at strict.pl line 3.
strict.pl had compilation errors.
$
$ perl strict.pl
Global symbol "$aNyme" requires explicit package name at strict.pl line 3.
Execution of strict.pl aborted due to compilation errors.

A few others out there have made this observation, for example:

I find python confusing on the other hand. e.g. sysmsg = sysmsg.replace('&', ' ')
what if you wrote "sysmgs = sysmsg.replace('&',' ')"
there is a small typo! In perl "use strict;" would find that for you, but python has no equivalent yet. [anonymous comment at LinuxJournal.com]

Are there reasons why Python can't, or shouldn't, support something like "use strict"?

Mark Levison, one of the developers I interviewed for the .NET story, thinks that Microsoft has undersold the benefits of No-Touch Deployment (NTD), the current solution for running rich .NET clients from the Web. Having done the gruntwork required to understand and use NTD, Mark's not so sure that developers ought to write off this technology and wait for Whidbey's ClickOnce.

I think David [Treadwell] misses much of the point. The caching features of No-Touch Deployment (NTD) work well enough. Click Once will be useful, but there are many other issues dealing with NTD apps. My impression is that MS didn't dog-food enough NTD. I think there are a few key areas that need work. [Mark Levison]

Mostly, Mark's asking for documentation and tutorials that will enable other developers to use NTD effectively, and spare them much of the painful R&D he had to go through. In his posting, he ticks off a very specific and well thought-out list of suggested items to cover in tutorials.

Consider two versions of a Word document saved as XML. There are "structured diff tools that can map the changes at an intermediate level, in terms of XML elements. For example, IBM's AlphaWorks site offers the XML Diff and Merge Tool for Java, while Microsoft's GotDotNet site offers XML Diff and Patch for .Net. Both of these free tools can track element-level change. To get a sense of what's possible, check out Monsell EDM's online demo of its Delta XML technology. The demo compares two subtly different versions of a complex graphic -- the standard SVG (Scalable Vector Graphics) "tiger" benchmark -- and animates the differences between the two. It's stunningly cool.

As XML becomes the standard way to represent prose, graphics, and other content, we should expect such change visualization to become routine. What about code? It has sections, subsections, and paragraphs, too. XML isn't -- and probably shouldn't be -- the primary way we read and write code. But the underlying abstract syntax tree has structure that can -- and arguably should -- help us see and comprehend the code's evolution. [Full story at InfoWorld.com]

Subversion's support for copying and renaming files and directories aims to reduce one of CVS's worst points of friction. Since I work with lots of XML data -- including just about everything I write -- I'm also eager to try plugging in some structured diff programs.

Several folks wrote with questions and comments about the OS X screen video I posted the other day. I mentioned that Media Encoder was the capture tool, but didn't specify how I got from Windows Media to Flash. For that, I used Camtasia Studio. I've heard good things about Qarbon but haven't had a chance to try it yet. Chris Ryland, from Em Software, wrote to recommend SnapzPro X 2 specifically for OS X (and QuickTime).

Also, Yoz Grahame wrote to alert me to a cool VNC hack, vnc2swf, a VNC viewer that records Flash movies. Getting it running, on either Fedora or OS X, failed my 5-minute rule. (I.e., if it takes more than 5 minutes, it's not a good use of my time.) But the example movies prove that it can work. And it's interesting to watch the author of vnc2swf, Yusuke Shinyama, driving various applications in a mixture of Japanese and English.

By the way, have you ever wondered what happens if you point a VNC viewer on one box (say, a Mac) at another box (say, Windows), then launch a VNC viewer on the second box and point it back at the first? Here's what:

Update: Karl Fast reports that he's seen a demo of a (still unreleased) new screen recording tool from Users First (great name!). The product is geared for usability analysis:

It is a client-server system. You have a CD for the client machine (Windows). It automatically runs off the CD. No installation required. This is a huge plus for capturing real work environments.

The recording program runs on MacOS X. It finds the client machine over the network. It can record an audio stream and multiple video streams. So one stream would be the screen video, but you can also capture users facial reactions and an audio stream, all synchronized.

You get pixel-perfect capture (it uses VNC), over the network, without having to install anything on the client.

There is more, but like I said, it's slick. Finally something really geared towards the usability-engineering/ information-architecture/interaction-design/user-experience crowd.

Despite lots of second-guessing, there is no consensus that the CLR is inherently unfriendly to dynamic languages. The JVM didn't bend over backwards for such languages either, and yet Jython is a great success thanks to the heroic efforts of its inventor, Jim Hugunin. Now Hugunin has turned his attention to .NET, and reports promising results with a prototype Python implementation for .NET called IronPython.

Such projects always seem to spring from an inspired individual or small team. In fact, Microsoft has such a team. It created JScript.NET, the most dynamic of Microsoft's .NET languages. But JScript.NET is the unloved stepsister of C# and VB.NET.

Dynamic languages are rooted in a culture that is simply not indigenous to Redmond. That may change, but for the time being, the future of dynamic languages in .NET lies with non-Microsoft innovators. [Full story at InfoWorld.com]

When we argue over whether or not a programming language should have types, we are not discussing a matter of fact. Instead, we are participating in what [linguistic philosopher John L.] Austin would call confessional language; what we are really doing is saying something about ourselves.

In particular, I think that those who advocate typed languages are (generally) participating in different kinds of programming exercises then those who advocate untyped languages. In particular, people who argue for strongly typed languages tend to be involved in projects that are

• large, with lots of interacting components;
• require multiple people to work together;
• will take a long time to develop (weeks or months, not hours); and
• will live for a long time, changing over that time.

On the other hand, people who like untyped languages tend to be involved in projects that

• require lots of prototyping;
• are done by one person, or a small group of people;
• tend to be small or short term; and
• often are used for a short period of time, or are not altered through their lifetime.

I've been thinking about this for a couple of days, because it's true that my own programming work is better characterized by the second list of attributes than by the first. Does this mean my passion for dynamic languages merely reflects my own orientation?

This led to another question: what is a large system? I tend to regard any application -- even a dozens-of-modules, millions-of-lines-of-code application -- as a good-sized component that participates in the large system we call the Web.

It's no accident that Perl was the original language of choice for programming that large system. Perl's dynamic nature was just what we needed in an environment that was itself dynamic, producing new services that could interact in unpredictable ways to yield emergent outcomes.

Nowadays Python is my first choice. Its approach to typing -- strong and dynamic -- is part of the reason why. But for programming the Web, I'll reach first for any of the dynamic languages before I'll reach for C# or Java. I've only recently been able to explain why. It's about the data. No one programming language's (or VM's) type system can (or should) span the Web. That's why XML has become one of the primary ways we invent, absorb, and interconnect data models. Dynamic languages offer complementary affordances.

Ten years ago I wrote my most widely cited BYTE cover story, called Componentware. I said then that software development was becoming a two-tiered system. There would be a relatively small number of component builders, working in compiled (today: JITed) languages such as C and C++ (today: C#, Java) to produce reusable components (today: services). And there would be a relatively large number of solution builders, working in scripting (today: dynamic) languages such as Visual Basic (today: Perl, Python, Ruby) to produce applications.

That components-and-glue metaphor still describes the software world today -- if anything, much more powerfully. The object orientation and static typing features of the JVM and the CLR are tools of the component builder's trade. And the dynamic features of what we still often call scripting languages are tools of the solution builder's trade. This isn't an either/or situation, though. Software development works best when the membrane that divides the component builder from the solution builder is flexible and porous, because the two activities are not as distinct as we suppose. This, I think, is why Sean McGrath calls Jython "Java's strategic weapon for the 21st century." And it's why I continue to want first-class dynamic language implementations for the CLR. The two tribes that Jim Waldo identifies are, roughly, the component builders and the solution builders. Dynamic languages are not only the solution builders' best tool. They're also the best way for the two tribes to collaborate on programming the planetary web of data.

In a pair of recent essays, Eric Raymond tears into the open source community -- rightly so -- for its failure to deliver software that Aunt Tillie can use. He's spot on. One of his comments got me wondering, though:

If the designers were half-smart about UI issues (like, say, Windows programmers) they'd probe the local network neighborhood and omit the impossible entries. If they were really smart (like, say, Mac programmers) they'd leave the impossible choices in but gray them out, signifying that if your system were configured a bit differently you really could print on a Windows machine, assuming you were unfortunate enough to own one. [Eric Raymond: An Open-Source Horror Story]

Actually this was a kill-two-birds-with-one-stone experiment. I've been wanting to be able to record screen videos on OS X, just like I do on Windows using Media Encoder 9, but I didn't have the software to do it. Or thought I didn't. Then I remembered VNC. I pointed a VNC viewer on Windows XP at a VNC server on OS X, and ran Media Encoder on the viewer. It works.

The upshot, for you fast-forward types, is that Aunt Tillie didn't have a picnic on OS X either. Raymond wrote:

Clicking on the menu, I am presented with the following alternatives:

Networked CUPS (IPP)
Networked Unix (LPD)
Networked Windows (SMB)
Networked Novell (NCP)
Networked JetDirect

Here is our first intimation of trouble. If I were Aunt Tillie the archetypal nontechnical user, I am at this point thinking "What in the holy fleeping frack does that mean?

Rather to my surprise, I found an oddly similar set of choices on the Mac:

  AppleTalk
  IP Printing
  Open Directory
  Rendezvous
  USB
x Windows Printing

I'm sure that on OS 9, talking to a PostScript printer, Aunt Tillie would never have needed to know about the dreaded CUPS (Common Unix Printing Systems) which provoked Eric Raymond's rant. Even so, I don't think her OS X misadventure blunts the force of that rant. Aunt Tillie has always been the problem. Her life may be a bit easier on Windows and on Mac OS, but it is far from comfortable. There's room for order-of-magnitude improvement. Will open source folk ever conclude that Aunt Tillie represents a hard engineering problem, and decide to wrap their collective heads around it? Stranger things have happened.

Every couple of years Microsoft wraps a marketing label around all the major initiatives in the company. In 2000, the label was .NET; in 2003, Longhorn. As developers and IT managers ponder what the "Longhorn wave" might mean to them, InfoWorld decided to assess the current .NET wave. Its goals were many and ambitious. At the core of .NET, the Common Language Runtime (CLR) and its associated Framework (class library) would usher Microsoft developers into the world of managed code, whose benefits were already well-known to their Java counterparts. In parallel, Web services would become the pivotal integration technology, and XML the lingua franca of data representation. These were, and still are, the central themes. Don Box, architect of Longhorn's Indigo communication subsystem, put it plainly on his weblog: "We're betting that the future is managed code and XML." [Full story at InfoWorld.com]

Back in '96 it was Dave Korsmeyer who popped up on the radar screen, to tell me about an interesting use of Java for distributed data visualization at NASA's Ames Research Center. I just heard from Dave recently. Now he's Chief of the Computational Sciences Division at the Ames Research Center, and his team has built several software tools to support the current Mars mission, including MERCIP, which Dave describes as "distributed web information application using XML as its messaging protocol."

In similar fashion, a number of folks popped up on the radar for this .NET story. I'd like to thank everyone who took the time to think about and discuss various issues. And I wonder what they'll be up to 8 years hence!

1060 NetKernel Suhail Ahmed alerted me, via email, to a really interesting project called NetKernel, from 1060 Research. The docs describe it as "a commercial open-source realisation of the HP Dexter project." Here's the skinny:

Today's Web-servers and Application Servers have a relatively flat interface which creates a hard boundary between Web and non-Web. This boundary defines the zone of URI addressable resources.

What if the REST interface (URI address space) didn't end at the edge of your external interface?

NetKernel uses REST-like service interfaces for all software components. The services are fully encapsulated in modules which export a public URI address space. A module may import other module's address spaces, in this way service libraries may be combined into applications. [NetKernel Essentials]

You compose primitive URI-based services like so :

Here's an example of a DPML [Declarative Processing Markup Language] 
instruction to perform an XSLT transform:
 
<idoc>
  <seq>
    <instr>xslt</instr>
    <operand>document.xml</operand>
    <operator>transform.xml</operator>
    <target>this:response</target>
  </seq>
</idoc>
 
Which the DPML runtime compiles to the active URI 
 
<code>active:xslt+operand@document.xml+operator@transform.xsl </code>

I never heard the phrase "REST microkernel" before, but I had an immediate expectation of what that would mean. An hour's experimentation with the system met that expectation. Wildly interesting stuff. Thanks for the pointer, Suhail!

I met Christopher Allen about a decade ago, when he ran Consensus Development, a company that made a commercial SSL toolkit. (Prior to that, he was involved in the startup of VeriSign, and in the development of the SSL reference implementation for Netscape.) I hadn't heard from him in a long time, and his recent essay, Security and Cryptography: The Bad Business of Fear, explains why. When he sold his company to Certicom in 1999, he signed a 5 3-year non-compete agreement. When it expired, he re-entered the security industry, expecting to find it much changed:

Internet time had still been moving fast back in 1999 and I wasn't sure how many generations had gone by in the security industry. One, two, more?

Actually, none, as it turns out.

Walking the floors of RSA last year, in the immense exhibit hall at the San Jose Convention Center, I did feel a sense of energy. The floor was still packed, and the carefully cut kiosks and the garish banners bespoke the millions put into the show by the exhibitors. The constant chatter was a deafening white noise, and whenever I veered too near a booth, there was a salesman very eager to tell me about his company's latest and greatest.

But, to a certain extent, that energy felt to me like a facade. There was nothing new; instead all the exhibitors were showing off the same technology that they were displaying five years ago. There was a bit of glitz and some extra chrome, perhaps a carefully redesigned product name, but beyond that there was a weird feeling of deja vu.

There were the same old tools that we've been using to deter hackers since the advent of the Morris Worm way back in 1989: products to detect intruders and safeguard your machines against them; firewalls; and VPNs. Maybe we've gotten a little better at figuring out expert rules, maybe we've improved our user interfaces, but these are slow, gradual upgrades, not quantum leaps.

To put it another way, we have been optimizing existing algorithms, not inventing new ones. The rest of this remarkable essay suggests what some of those new approaches might be. He considers the idea of insurance as a form of business risk management, something that Bruce Schneier has also been discussing lately. He notes that data security is not the same thing as data reliability: the latter is what we really want. And he suggests, finally, that alongside these approaches driven by fear, we need to develop new methods motivated by opportunity.

The possibilities are only limited by our imagination, if we can just think beyond current possibilities.

We have already seen the first wave of security technology; now we need to initiate a second, for I believe with the next wave the best is yet to come.

Well said. And welcome back, Chris!

Here's one popular definition of insanity: "Do the same thing, expect a different result." Now consider the following partial list of proposed standards for Web services: WS-Addressing, WS-AtomicTransaction, WS-Attachments, WS-Context, WS-Coordination, WS-Eventing, WS-Federation, WS-Reliability, WS-ReliableMessaging, WS-Routing, WS-SecureConversation, WS-Security, WS-SecurityPolicy, WS-Transaction, and WS-Trust. [Full story at InfoWorld.com]

In last month's installment I showed a simple search service that uses libxslt to reduce a file of XML content (my weblog writing) to just the elements matching an XPath expression. This month's challenge was to scale up to a database-backed implementation using Berkeley DB XML. [Full story at XML.com]

There's some sort of object lesson here. Lately I've grown extremely fond of the libxml2/Python combination. When I need to process XML, that's how I want to do it. But having developed this habit, it also becomes necessary to break it from time to time. Materializing the libxml2/Python combination, on a given platform, can absorb time and energy that may be better spent elsewhere, and it can even lead to compromises.

Case in point: my original implementation of this service used Jython to talk to the DB XML Java API. This was actually a great combination. It married Python's flexibility to a more robust and complete DB XML API than is available from the C flavor of Python. However, it lacked my new old friend, libxml2. So I wound up using an older version of DB XML (1.2, rather than the latest 1.2.1) in order be able to use C Python. Which, as it now turns out, was unnecessary, since DB XML supports both database-level and document-level querying.

It's amazing how one wrong or missing piece of information can wind up dictating a major architectural choice. And how one unexamined habit can make us vulnerable to that outcome.

Here's what Brent Simmons had to say about yesterday's item on news scanning and news reading:

1. NetNewsWire's Combined View works with channels, all-new-headlines -- and groups.

For instance, I have 145 subscriptions organized into 8 groups. When you view a group in the Combined View, you see all the unread headlines for that group. Like most people, I organize my groups by topic.

So I have a page for Macintosh news, a page for weblogs, a page for books, a page for baseball, etc.

2. Generalizing about NetNewsWire based on Steve Gillmor's use of it isn't fair.

For instance, I personally find Radio's batches-of-100-in-a-web-page to be awkward. With Radio I can't scan fast enough and I can't keep up. I developed NetNewsWire so I could keep up with more feeds with less effort. But everybody's different: different presentations work for different people. That doesn't mean that Radio's approach is better or worse than NetNewsWire's.

It's true that different folks will prefer different strategies for grouping and processing their feeds. But no matter which strategy you prefer, you need to harmonize two modes: scanning, and reading. And no matter which strategy you prefer, the same methods can be used to achieve that harmony. On the publishing side: untruncated feeds, containing HTML (ideally, but not necessarily, XHTML) markup, with a first element that can work standalone. This is often naturally the case, since a lead paragraph's job is to hook the reader.

On the consumption side: feedreaders that XHTML-ize content (in case it isn't already XHTML), use the first markup element to optimize scanning modes, and provide the full content for reading. Peter Eschenbrenner suggests this is already possible with FeedDemon:

You might want to check out FeedDemon by Nick Bradbury. While it comes with default style sheets, users are able to create their own XSL for efficiently processing the information. So, if you wanted to view just the first paragraph, you could create your own style or ask someone in the community to create one.

Radek, an active community member, has created styles that hint at what can be achieved with this combination, from rating your feeds in a database, to creating powerful MindMaps. [Peter Eschenbrenner: Note to Self]

Interesting! So far as I can tell, though, FeedDemon's XML data model corresponds -- as you'd expect -- to that of RSS. Which means the content is opaque. So while you can use XSLT to hack alternate presentations of channel-level and item-level dates and titles, XSLT can't see into the content. For example, the default stylesheet includes:

<div class="newsitemcontent">
  <xsl:value-of select="description"/>
</div>

Bottom line: blog content needs to become a first class citizen in the XML world. And as it turns out, it's more feasible than I thought to make that so. Most people won't be producing well-formed content anytime soon. But tools that produce and consume RSS can compensate, with the help of things like HTML Tidy, and there are compelling reasons to do so.

In his essay Birth of the NewsMaster, Robin Good writes:

I have seen and heard of people subscribing to hundreds if not to thousands of feeds inside their RSS aggregators.
Is that manageable? Do these people get better and more information than everyone else?
It is not. They don't.

Yesterday, for example, Steve Gillmor told me that he's feeling overwhelmed by thousands of unread items in NetNewsWire. Yet I never feel that way. I suspect that's because I'm reading in batches of 100 (in the Radio UserLand feedreader). I scan each batch quickly. Although opinions differ as to whether or not a feed should be truncated, my stance (which I'm reversing today) has been that truncation is a useful way to achieve the effect you get when scanning the left column of the Wall Street Journal's front page. Of the 100 items, I'll typically only want to read several. I open them into new Mozilla tabs, then go back and read them. Everybody's different, but for me -- and given how newspapers work, I suspect for many others too -- it's useful to separate the acts of scanning and reading. When I'm done with the batch, I click once to delete all 100 items.

As a user of NetNewsWire Lite, I don't have access to the combined view that enables items to be processed in batch rather than individually. The example screenshot suggests that there is still a per-channel interaction required, however I suspect that when Combined View is used in conjunction with Show Aggregated New Items, you can see -- and process -- everything at once. (If I've got that wrong, I'm sure Brent will clarify.)

If Steve and I have the same batch-processing capability, why do we feel so differently about the overload problem? Maybe because it's not the same. If I'm right about NNW's Combined View / Show Aggregated New Items, the difference may boil down to this: my aggregated view delivers batches of 100, whereas Steve's delivers either small per-channel batches, or very large all-channel batches. So, in other words, I'm seeing what roughly corresponds to a Wall Street Journal news summary, whereas Steve is seeing what roughly corresponds to a 5x or 10x bigger version of that page. (If I've got that wrong, I'm sure Steve will clarify.)

Either way, the content is an awkward mixture of truncated and full items. Both modes are useful, but they serve different purposes and they mix badly. Truncation is necessary for the Wall Street Journal effect, though where and how to truncate is a tricky question that I've just now changed my mind about. And of course you need the full view at some point, so you can actually read stuff.

Currently I provide two versions of my feed: truncated and full. And the truncated feed is intelligently truncated. Using a callback that Dave Winer added to Radio UserLand a couple of years ago, I select the first HTML paragraph (<p>) element. Knowing that this will happen, I put some thought into what that element will contain when I'm writing an item. In effect, the first paragraph element is the lead, or blurb. Sometimes it's just a plain paragraph. But sometimes it will contain an image, or a quotation, when these are appropriate and useful hooks. This query, which shows the first paragraphs from all my January items, illustrates some of the variation. The fact that I can issue this query against my untruncated feed shows that my truncated feed is really not necessary. What is necessary, or at any rate useful, is the extra bit of preparation, i.e. thinking about what goes into that first HTML paragraph.

Unfortunately the effect of all my careful preparation has mostly been wasted so far. When you process large batches of feeds, some of which use intelligent truncation, some of which use dumb truncation (i.e., just grab the first 250 characters and slap on an ellipsis), and some of which use no truncation, the result is kind of a mess.

All along, I've had the idea that feedreaders should be able to smooth out these differences. If you wanted a Wall Street Journal view across all your feeds, you could get one. And if you wanted a full-content view across all your feeds, you could get that too.

Playing around with my queryable feed database today, I realized we're within shouting distance of making that happen. And I'm reversing my former stance on truncation. Here is a Wall Street Journal view of all of my feeds so far today. And here is a full-content view of all of my feeds so far today. It includes this long item I'm now writing, which shows how a mixture of truncated and untruncated content is optimal for neither scanning nor for reading.

Here are my conclusions:

• Nobody needs to truncate feeds in order to enable front-page views (although some will still want to in order to drive traffic to websites).

• Everybody's content should be HTML (if not XHTML).

• Authors should think of the first HTML element (normally a paragraph, but could be a list or a blockquote or something else) as special: the lead, or deck, that will appear in a front-page view.

• Feedreaders should XHTML-ize what they read.

• Feedreaders should then offer a front-page view (e.g., just the first HTML element found in each item) as well as a full-content view.

By the way, in case it isn't obvious, the RSS/Atom controversy is irrelevant to this discussion. In both environments, the same principles could be applied in exactly the same ways, for exactly the same reasons.

Dare Obasanjo complains about being shut out of Steve Saxon's feed:

This afternoon I found out that Steve Saxon, the author of the excellent article XPath Querying Over Objects with ObjectXPathNavigator, had a Blogger.com blog that only provided an ATOM feed. Being that I use RSS Bandit as my aggregator of choice I cannot subscribe to his feed nor can I use a large percentage of the existing news aggregators to read Steve's feed. [Dare Obasanjo]

How? Just search Google for atom2rss. There are a bunch of translators floating around. The one I picked comes from the folks at 2rss.com. Here is their translator: http://www.2rss.com/software.php?page=atom2rss. And they are kindly making this service available for free. You can go to the site, plug in an Atom URL, generate the corresponding RSS URL, and subscribe to that.

Sheesh. The fact that we are now going to have a war over formats that are separated by a trivial XML transformation is almost as depressing as February in New England. This cheered me up, though:

I'd like to give a big "hats off" to all the work Jon Udell is doing to make the "semantic web" a reality by looking at ways to extract information from existing (X)HTML content, and also in proposing new ways of adding semantic information to the markup for new content. If you don't already read his blog then I can't recommend it highly enough. Anyone who's involved in the reading or writing of technical blogs must surely have at least a passing interest in this sort of stuff. (BTW, I'm surprised that his work does not seem to have received much attention from the "big" blogging sites and/or engines. Unless I've missed it, of course...) [Paul's Imaginary Friend]

Somebody was looking over my shoulder the other day as I was using the dropdown list of search plugins in Firefox (nee Firebird nee Phoenix nee Mozilla), and was surprised to see it. Which reminded me that in IE and Safari, the built-in search isn't extensible.

Now that we're all comparing Google and Yahoo!, it's really handy to be able to query one engine, then repeat the query in another engine frictionlessly. Here's what that looks like:

If you're using a Mozilla variant and haven't set up a Yahoo! plugin yet, it's installable from the mycroft page. Very convenient.

I wondered whether today's biggest tech news story -- Yahoo! dumping Google for its own search engine -- would show up first in the Sci/Tech category at Google. Sure enough, it does:

Cool.

Meanwhile, as everyone begins to dissect the capabilities of the new Yahoo search engine, The Shifted Librarian notes that RSS feeds associated with found sites are highlighted, and can be added to the My Yahoo feedreader. Except for Google's Blogger-created blogs, which don't bother to provide RSS feeds.

Uncool.

Update: Heh. I just rechecked, and now there's no sign of the Yahoo! story at news.google.com. Did I get today's lucky screenshot?

Further update: Now the Yahoo! story is back :-)

Still further update: And now it's gone again. I'm getting dizzy...

Tim Hodson's LibraryLookup bookmarklet broke when his library upgraded its OPAC. So he fixed it:

Have used your lookup many times, until our library service started to use a new OPAC! Talis Information systems have released a new OPAC which is called Talis Prism. For a while I thought my lovely lookup would never work again, but I have recently discovered (by changing their post form variables to gets with the marvellous firefox browser and a web developers toolbar) that a get version of their page works just as well.

If your game is enterprise software, you might regard all this library stuff as an odd diversion of mine. But ask yourself: can users of your ERP and CRM systems hack their own integration? If not, why not?

At ETech (which I unfortunately could not attend) there was a presentation entitled real world semantics that is close in spirit to my own recent experimentation. The presenters were Technorati's Kevin Marks and Tantek Celik, who fought the good fight to bring quality CSS support to Microsoft's now-abandoned MSIE/Mac. Phrases they use to define real world semantics: "emerging semantic (x)html", "adoption by 'real people'", "beyond academics and theoretical discussions." Exactly.

Meanwhile, over on Kingsley Idehen's blog, you can see another implementation of the kind of xhtml-aware search technology I've been playing with here. The advanced search feature uses the Virtuoso engine to perform not only XPath search, as I'm doing, but also XQuery search. Here's one of the provided examples:

for $i in node()//a return <p>{ string($i/@href) }</p>

Although random XHTML can be mined more fruitfully than you might suspect, I'm on the lookout for ways to naturally, and virally, enrich its semantic carrying capacity. The Celik/Marks presentation points to several such efforts, including GeoURL, which I use in my blog's header to announce my location (<META name="ICBM" content="42.93564,-72.27239">), and XFN, the XHTML Friends Network, which proposes using the REL attribute of links (<a href="..." rel="acquantaince">) to indicate relationships. This is the sort of thing that will make the search techniques Kingsley and I are demonstrating come alive. My hunch is that lots of XFN-like strategies will emerge, if we can close the feedback loop and connect the effort required to adopt such a strategy to an immediate reward.

"I feel like I'm at a Microsoft monastery here," wrote Rory Blyth from the most recent Professional Developers Conference. "I think I've seen about 2.5 females ... it's like they're an endangered species." The observation holds equally true for open source conferences.
...
If we expect social software to help rewrite the productivity equation, social skills and protocols become critical parts of the game. How can social software succeed if, in its development, half the population is so poorly represented? [Full story at InfoWorld.com]

I'm not a social scientist or a psychologist, and I was reluctant to touch either of these controversies. (As you might imagine, the column provoked some internal discussion at InfoWorld.) In the end I decided to go ahead precisely because both subjects make me uncomfortable.

The larger of the two issues, in my mind, is that of gender. Nobody seems to have any real answers, but here are some perspectives on gender and computing:

The percentage decline in computer science was much larger among women (51 percent) than among men (28 percent) from 1985 to 1995. [National Science Foundation]

Programming assignments are many times devoid of meaning and importance to people's lives, which tends to appeal more to boys. Girls, on the other hand, will be more attracted to technology, if it has some meaning or positive purpose in a real-world context. [Heather Makoski: Underrepresentation of Women in Science, Engineering, and Mathematics]

Girls and women are choosing, consciously or subconsciously, not to go into or stay in computer science. While one cannot rule out the possibility of some innate neurological or psychological differences that would make women less (or more) likely to excel in computer science, I found that the cultural biases against women's pursuing such careers are so large that, even if inherent differences exist, they would not explain the entire gap. [Ellen Spertus: Why are There so Few Female Computer Scientists?

On my way to work this morning I was listening to NPR, as I usually do, and heard a segment on the declining numbers of female students entering the computer science major. I'm sure they are correct in their observation that the numbers are indeed declining, I'm not going to argue that. I am however finding myself disagreeing with their reasoning behind this decline. One thing in particular that I felt was an erronous conclusion.. the amount of time young boys spend playing video games as opposed to young girls. I do agree that most video games are geared towards boys, I don't agree that this has anything to do with the probability of a child's future interest in computer science. [kasia in a nutshell

Additionally, for many females, computers are more meaningful and compelling if they are able to link them with other fields and are able to keep computer science's social context in mind. Margolis and Fisher (2002) call this appeal "computing with a purpose." However, computer science curricula has traditionally been oriented on the basis of the fascinations of male students, and the aspects of computers that females find interesting may not be emphasized. This lack of emphasis on certain characteristics may discourage women, allowing them to feel computers "aren't for them." [Maria Enderton: Honors Thesis, Women in Computer Science]

It's funny that Jon has written an article on social networking addressing the geek male perspective, when I've been thinking quite a bit lately that some of the best minds regarding efforts behind Social Networking are actually female. They just get the importance of relationships much better than guys. [Digital Squeeze]

I'd be interested in comments on these issues.

On a related note, I'm working on a story about enterprise social software. What that label means is, of course, open to discussion. If you're developing and/or using what you think of as enterprise social software, and want to talk about it, feel free to ping me.

Actually, it was in 1960, four years before the Beatles showed up on Ed Sullivan, that Doug McIlroy published Macro instruction extensions of compiler languages, which appears to be a seminal paper in the literature of metaprogramming. I mention this because a number of folks have responded to last week's item, Programs that write programs, pointing out that Lisp programmers have been there, done that:

"your note about code generation, and the referenced discussions - bits of which i'd already read elsewhere, left me with a really eerie feeling, that i might not be living in the same dimension with you folks. you see, there's a practice of code generation which extends back decades: lisp. code generations is a lisp programmer's bread and butter."

"In the lisp world, they call these macros. The idea is pretty widely known, though not too many languages implement them. Perl 6, and by extension the Parrot interpreter, will include macros, and they will thus be available to any language that gets implemented on top of Parrot (which currently includes Ruby, Python, (maybe) PHP, and, of course, Perl)."

Points taken. As it happens, I did at one time program in a variant of Lisp. From that experience I learned the value of incremental development, dynamic data structures (lists, dictionaries), code generation, and other techniques that later became available to me in languages like Perl and Python. But macros weren't part of the Lisp I used, so I didn't make that connection.

The cultural anthropology of programming languages is a fascinating subject. Recently, for example, I asked an accomplished developer with deep roots in the Microsoft programming culture to cite his favorite productivity aids in the .NET Framework. Regular expressions made his short list. That floored me, since regexes are just part of the atmosphere that Unix and open source programmers have always breathed. But a lot of Microsoft programmers didn't grow up breathing that atmosphere.

Of course it goes both ways. I'm likely to try out new technologies on Windows first, because the Windows culture groks packaging and installation -- even of open source software! -- better than the Unix culture does.

There's some truth to the oft-heard claim that there are no new software technologies. If we spent the next decade just cross-fertilizing what we already have, it would probably be a decade well spent.

Python hacker and OCLC chief scientist Thom Hickey has updated me on the xISBN project:

Just thought I'd let you know that we've put up a new version of the ISBN database. We've done a lot of work to pull works with variant titles together (which helps with The Innovator's Dilemma) and made the retrievals consistent, so that any ISBN in a group retrieves that same ISBN group (which also helps with I's D). We've learned a lot about how ISBNs are used (and misused).

There are a few caveats here. First, the one-to-many algorithm doesn't seem to be fully bi-directional. In the example above, we'd like to get from 0066620694, a paperback, to 0875845851, a hardcover. But although we can get from 0875845851 to 0066620694, we can't get from 0066620694 to 0875845851. [Jon's Radio: Multi-ISBN LibraryLookup]

My adaptation of LibraryLookup for xISBN, by the way, is here. An improved xISBN service makes it more interesting, but the real bottleneck will be the OPAC systems themselves. The LibraryLookup idea -- which gets a nice mention in this month's Technology Review (thanks, Rael!) -- works by splicing two Web contexts together. From a page at Amazon or B&N or AllConsuming, you go to a page on your library's Innovative or Polaris OPAC system. Now, with xISBN, you can present the OPAC with a list of ISBNs. Unfortunately, OPACs have no idea what to do with a list of ISBNs. The multi-window solution I tried is kind of lame.

I'd love to see the various OPACs take note of xISBN. We can imagine all sorts of fancy integrations involving Web services or WSRP, but the simplest thing, really, would be for OPACs to silently expand an ISBN to an ISBN group, search accordingly, and return a combined result.

I'll be on a panel at SXSW Interactive in March, entitled Streetwise Librarians and the Revolution in Public Information, which should be a great venue in which to explore these kinds of issues.

Following pointers from Ned Batchelder's recent excursion into code generation led me to another nice example of the power of dynamic languages. In order to streamline his use of C++, Ned wrote a little tool called cog which enables him to embed, in C++ programs, Python fragments that generate verbose and/or repetitive C++ constructs. He adds:

For more about code generation in general, try:

• Dave Thomas interviewed about code generation. Dave Thomas is one of the Pragmatic Programmers, and I find I agree with him almost universally. He forbids putting the output of code generators under source control, I encourage it. We agree that the output should never be edited.
• The Code Generation page on the c2 wiki. As will happen with a wiki, this fractures off in many directions, with many different viewpoints, both for and against code generation.

[Ned Batchelder]

In the interview Ned cites, Dave Thomas gives an example of a Ruby feature that I've heard of, but never had occasion to use. In a class definition you can write Ruby code to define a type. That means, as Thomas puts it, that "you can effectively extend the language at runtime from within." Statements like that have a tendency to alienate people. It can sound like the drug-induced fantasy of some idealistic tree-hugging Birkenstock-wearer who isn't living in the real world of Enterprise Software Development. But Thomas backs it up with a great example. In this case, he used Ruby's dynamic extensibility to wrap a database schema in classes that can either persist objects to the database, or create schema documentation, depending on how the methods that dynamically define those classes are defined.

Once upon a time Tom Christiansen gave me a great quote, which he attributes to Andrew Hume: "Programs that write programs are the happiest programs in the world." Templating and code generation are examples of this happy strategy. We've always known that dynamic languages are a great way to create "little languages" for specific tasks. But we don't yet fully appreciate that all programming is a continuous process of language invention. And we don't (yet) evaluate programming-language productivity on those terms. Dave Thomas:

I'm betting that languages such as Java and C++ will in the long term be seen as a curious branch in the evolution of computing. I'm hoping that somewhere out there some bright spark is coming up with a way of letting us write applications expressively and dynamically. Once this happens, the need for these kinds of code generators will diminish.

For example, I rarely (if ever) write a code generator that generates Ruby code: there's just no need, as Ruby is dynamic enough to let be do what I want without leaving the language.

A reader named Jemisa wrote last week with this proposal:

Just another proposition for

<pre class="code" lang="python">
...
</pre>

Why not use

<pre class="code python">
...
</pre>

I know it's less "semantic" than your experimental attribute, but it might be useful (to style python code with different color than perl code for example)

But then Jim Roepke sent me the exact same proposal, and set me straight on the first point:

You can specify more than one class name in the class attribute.... you could say:

<pre class="code python">...</pre>

I don't know if your XPath stuff can handle that, but it's a valid way to specify the class of an element. In terms of CSS, both the code and python classes will be applied to the element.

<pre class="code python">
import re
import sys
...
</pre>
 
<blockquote class="personQuote StefanoMazzocchi">
"..."
</blockquote>

blockquote.personQuote {
font-style: italic;
}
blockquote.StefanoMazzocchi:after {
content: 
  url(http://www.oio.de/public/web/stefano-mazzocchi.jpg)
}
pre.code {
border-style: solid;
border-width: thin;
padding: 10px;
}
pre.python:after {
font-weight: bold;
content: 
  url(http://www.python.org/pics/PythonPoweredSmall.gif);
}

This looks really promising. On the XHTML front, legal and even elegant. No extra namespace baggage. It's easy to write this way by hand, and tools that give people control over CSS styles could easily support this method if they wanted to.

On the search front, it's less than ideal. But the difference boils down to substring matching versus string equality. On a small corpus of XML content -- i.e., your own blog -- this won't matter, as I've already demonstrated to my own satisfaction. And on a larger corpus -- like the one I'm assembling now -- I'm presuming a database with indexing that can make a contains() query roughly as efficient as an equals() query.

Thanks, Jemisa and Jim! At the moment this looks like the winning strategy. I can't dive into this for a few days yet, but I'll watch for further feedback in case there's something else I've missed.

For a team of collaborators, Groove synchronizes both the sets of applications available in a given context (or "shared space") and the data written by those applications. If you drop your laptop on the floor you can effortlessly recover everything into a fresh instance of Groove on a new machine.

Of course this works only for native Groove apps. Browser history and bookmarks, Outlook settings, and a million other things are handled in a million other ways -- or not handled at all -- because desktop operating systems aren't Groove. A general solution would require OSs that work like Groove, and applications that send messages rather than write files. Well, come to think of it, why not? [Full story at InfoWorld.com]

But sometimes arguments cross a line beyond which everybody gets hurt, including the Net. I see that happening here. Even though I'm no technologist, it's clear to me that the Net has been improved, radically and fundamentally, by RSS and other standards like it (even if they come, as Mark claims RSS does, in 9 incompatible versions). [Doc Searls]

Oh, and by the way, what's up with this?

A Hungarian developer, Fejes Balazs, alerted me to a couple of his articles on XQuery -- the first a general introduction, and the second a walkthrough of XQuery transformation in BEA's WebLogic Workshop. Both are nicely done.

Given that so much more can be done with XPath and XSLT than is widely appreciated, I've been focused mainly on broadening awareness of what's possible. But I've been studying XQuery in parallel, and it recently struck me that one of the reasons XQuery is going to be important matches one of the reasons that dynamic programming languages are important: both let you play with data.

That phrase -- "play with data" -- comes to me by way of Jonathan Robie, who I met at XML 2003. Jonathan, co-author of XQuery from the Experts and co-editor of the XPath and XQuery specs, believes (as do I) that data is a substance, like clay, that you have to pound on, roll out, squeeze, mold, and generally get your hands dirty with, in order to discover its possibilities.

Over the years I've come to see that the ability to treat data like clay is a primary benefit of languages such as Perl and Python. If you had to finalize your data structures up front you'd never get anywhere, because they're emergent.

Now listen to Jonathan Robie on the subject of types in XQuery:

The type system of XQuery is one of the most eclectic, unusual, and useful aspects of the language. XML documents contain a wide range of type information, from very loosely typed information without even a DTD, to rigidly structured data corresponding to relational data or objects. A language designed for processing XML must be able to deal with this fact gracefully; it must avoid imposing assumptions on what is allowed that conflict with what is actually found in the data, allow data to be managed without forcing the programmer to cast values frequently, and allow the programmer to focus on the documents being processed and the task to be performed rather than the quirks of the type system. [XQuery From the Experts, Chapter 1: A Guided Tour]

There have been a number of thoughtful responses to my confession, the other day, about cheating on Web standards. Several folks recommended this approach:

<blockquote 
   cite="http://www.betaversion.org/~stefano/linotype/news/35/"
   title="Stefano Mazzocchi">
...
</blockquote>

<blockquote cite="urn:name:Stefano%20Mazzocchi">
..
</blockquote>

Of course that still left the other problem:

<pre class="code" lang="python">
..
</pre>

I think that for the next phase of this experiment, I should just bite the bullet and start writing nonstandard attributes -- such as 'lang' in this case -- into another namespace. For an author, as Jim points out, there's not a lot of extra friction or overhead. It could be as little as two extra characters:

<pre class="code" e:lang="python">
..
</pre>

It's time for a confession. I've been acting as though all this cool XPath search stuff I've been demonstrating for the past few weeks were based on plain vanilla XHTML. Well, it's not (quite) true. In general my point has been to illustrate two things:

1. That the XHTML equivalent of ordinary HTML content includes metadata (links, tables, images) that can be usefully exposed as XML.

2. That legal ways of enlarging the namespace used within HTML -- in particular, CSS class attributes -- can enhance this approach.

But in truth, as some have noticed, I've been cheating on XHTML a bit. Here's one cheat: in order to support this query -- Python snippets -- I've been writing HTML like this:

<pre class="code" lang="python">
...
</pre>

Here's another cheat. In order to support this query -- quotes from Stefano -- I've been writing HTML like this:

<blockquote cite="Stefano Mazzocchi">
...
</blockquote>

More generally, there are zillions of such use cases which I don't think we can know in advance of discovering them. So I can't imagine proposing any specific extensions to XHTML that would accommodate such discovery. I can think of two general approaches, though. One might go like this:

<blockquote 
  cite="http://www.betaversion.org/~stefano/linotype/news/35/" 
  X-who="Stefano Mazzocchi">
...
</blockquote>

Another might go like this:

<blockquote xmlns:exp="http://XHTML-Experimental"
  cite="http://www.betaversion.org/~stefano/linotype/news/35/" 
  exp:who="Stefano Mazzocchi">
...
</blockquote>

I've hesitated to even raise this issue because, in my experience, it's the kind of thing that can just get bogged down in endless discussion and debate. So I've gone ahead and cheated a bit on XHTML in the service of what I think is a proper ambition: to find some workable middle ground between the unstructured real web that exists all around us and the structured Semantic Web that exists only in our imagination. Or rather, to suggest how the latter can emerge from the former. But to those of you who've wondered: yes, I do feel guilty about cheating, and I'd like to come clean. Are there ways to enlarge the carrying capacity of the HTML namespace without doing violence to the spec? And without inventing mechanisms too complex for writers of ordinary everyday documents, or too far removed from existing writing tools?

At InfoWorld's 2002 CTO Forum, Google co-founder Sergey Brin threw cold water on the idea of instrumenting content for intelligent search. "I'd rather make progress by having computers understand what humans write," he said, "than by forcing humans to write in ways that computers can understand." Brin's pragmatic stance sharply opposes the idealistic view of the Web's inventor, Tim Berners-Lee, who continues to evangelize his vision of a Semantic Web full of carefully encoded content that we can precisely search and fluidly recombine. My own humble contribution to this debate is a prototype search engine, now running on my Weblog, that tries to steer a middle course between the Scylla of simple fulltext search and the Charybdis of unwieldy tagging schemes and brittle ontologies. [Full story at InfoWorld.com]

The NY Times has an article today by Steve Lohr, entitled Technology and Worker Efficiency, in which John Seely Brown makes the case for productivity very well. Here's a quote that sums up nicely what I also think is happening, and why I am optimistic:

John Seely Brown, former director of the Xerox Palo Alto Research Center, says he believes that recent changes in software technology could allow big gains in productivity and innovation. The opportunity, he says, is to move beyond the limitations of centralized systems for automating business operations, like enterprise resource systems. "Those systems are prisons," said Mr. Brown, who is scheduled to speak at today's conference.

The software plumbing of computing, Mr. Brown explains, is evolving, and so is Internet-based software for individual workers. Software systems built on Web standards, he said, can be used as pick-and-place building blocks, instead of the more formal hierarchical systems of the past.

Mr. Brown also points to the rapid development of what he calls "social software" like instant messaging, Weblogs, wikis (multi-user Weblogs) and peer-to-peer tools - all of which make it easier for workers to communicate and collaborate online, almost instantaneously.

The combined result, Mr. Brown said, is information technology that can amplify social interaction and enhance workers' understanding of what is happening around them. The benefit, he added, could be to increase their ability to "collectively improvise and innovate."

That is a key to productivity and peak performance, according to Mr. Brown. Business, he said, is a lot like soccer. In soccer, there are some set plays, but the best teams also display a wealth of effective improvisation based on the players' deep knowledge of one another. "It's the same in the best corporations or start-ups," he said. [New York Times: Technology and Worker Efficiency, by Steve Lohr

Kevin Werbach got the original sound bite on this: "Web services, Weblogs and WiFi are the new WWW." It was becoming clear in 2002, and is clearer now, that this is a recipe for the kinds of productivity gains that move the needle on the economic dial. However, it's frustratingly hard to be concrete about that squishy intersection between knowledge and collaboration.

Apologies to those of you whose feeds I incorrectly named in yesterday's (now updated) entry about RSS feed caching (or, rather, non-caching). I've revised the list. It does appear, though, that there is still a healthy percentage of my 200+ feeds that are not being cached.

It strikes me that the normal methods of checking whether a feed is or is not cached are way, way too geeky for ordinary users. Here's a thought: could/should the feed validator also report whether a feed is using one or another of the caching techniques, and warn if not?

Now that I'm accumulating my inbound feeds as XHTML, in order to database and search them, I find myself in the aggregator business, where I never planned to be. The tools I'm using to XHTML-ize my feeds are Mark Pilgrim's incredibly useful ultra-liberal feed parser and the equally useful HTML Tidy, invented by Dave Raggett, and maintained by folks like Charlie Reitzel, one of CMS Watch's Twenty Leaders to Watch in 2004 (along with yours truly).

Today I finally got around to using the ETag and conditional GET (If-Modified-Since) features of Mark Pilgrim's feed parser. (Apologies to my subscribees who, until now, have been treated impolitely by my indexer.) Of the 200+ feeds to which I subscribe, fifty 35 seem not to support either of these two bandwidth-saving techniques, which means they're probably getting battered unnecessarily by feedreaders. The victims are:

http://fieldmethods.net/backend.php
http://groups.yahoo.com/group/syndication/messages?rss=1&viscount=15
http://matt.griffith.com/weblog/rss.xml
http://nhpr.org/view_rss
http://royo.is-a-geek.com/siteFeeder/GetFeed.aspx?FeedId=43
http://safari.oreilly.com/NewOnSafari.asp
http://today.java.net/pub/q/29?cs_rid=47
http://today.java.net/pub/q/weblogs_rss?x-ver=1.0
http://usefulinc.com/edd/blog/rss
http://w3future.com/weblog/rss.xml
http://w3future.com/weblog/staplerFeeds/dubinko.xml
http://www.burtongroup.com/weblogs/jamielewis/rss.xml
http://www.eighty-twenty.net/blog?flav=rss
http://www.eod.com/devil/rss10.xml
http://www.fuzzyblog.com/rss.php?version=2.0
http://www.g2bgroup.com/blog/rss.xml
http://www.gonze.com/index.cgi?flav=rss
http://www.gotdotnet.com/team/dbox/rssex.aspx
http://www.gotdotnet.com/team/tewald/rss.aspx?version=0.91
http://www.intertwingly.net/wiki/pie/RecentChanges?action=rss_rc
http://www.lucidus.net/blog/rss.cfm
http://www.markbaker.ca/2002/09/Blog/index.rss
http://www.mobilewhack.com/index.rss
http://www.neward.net/ted/weblog/rss.jsp
http://www.newsisfree.com/HPE/xml/newchannels.xml
http://www.openlinksw.com/blog/~kidehen/gems/rss.xml
http://www.oreillynet.com/cs/xml/query/q/295?x-ver=1.0
http://www.pepysdiary.com/syndication/rss.php
http://www.photo-mark.com/cgi-bin/rss2.cgi?set_id=16
http://www.pipetree.com/qmacro/xml
http://www.testing.com/cgi-bin/blog/index.rss
http://www.voidstar.com/module.php?mod=blog&op=feed&name=jbond
http://www.xmldatabases.org/WK/blog?t=rss20
http://www.xmlhack.com/rss.php
http://www.zope.org/SiteIndex/news.rss

Update: This list is 15 shorter than it was last night. Greg Reinacker wrote to point out that his feed does emit the ETag header. I checked, and what I originally reported was feeds that were missing one or the other of two different ways to tell the client a feed hasn't changed. But so long as one is in effect, you're OK. Now the list should include only feeds that support neither method, and that as a result cannot return the HTTP '304 Not Modified' response enabling a feedreader to skip an unnecessary fetch of an unchanged feed.

Here's a brief summary of the two methods. First, a site that supports Etag (but not Last-Modified), namely Greg's:

1. First fetch of Greg's feed:
 
GET /gregr/weblog/rss.aspx HTTP/1.1
 
2. Etag response:
 
HTTP/1.x 200 OK
Date: Mon, 02 Feb 2004 14:17:01 GMT
Server: Microsoft-IIS/6.0
Etag: "632104748500000000"
 
3. Second fetch of Greg's feed:
 
GET /gregr/weblog/rss.aspx HTTP/1.1
If-None-Match: "632104748500000000"
 
4. 304 response:
 
HTTP/1.x 304 Not Modified

Now here's a site that supports Last-Modified (but not Etag):

1. First fetch of David's feed
  
GET /index.xml HTTP/1.1
Host: www.davidgalbraith.org
  
2. Last-Modified response
  
HTTP/1.x 200 OK
Server: Zeus/4.2
Last-Modified: Mon, 02 Feb 2004 02:02:55 GMT
  
3. Second fetch of David's feed
  
GET /index.xml HTTP/1.1
If-Modified-Since: Mon, 02 Feb 2004 02:02:55 GMT
  
4. 304 response
  
HTTP/1.x 304 Not Modified

And finally, here's a site from the list above, supporting neither method:

1. First request:
  
GET /syndication/rss.php HTTP/1.1
Host: www.pepysdiary.com
  
2. Response includes neither Etag nor Last-Modified
  
HTTP/1.x 200 OK
Server: Apache/1.3.19 (Unix) PHP/4.0.4pl1
Transfer-Encoding: chunked
Content-Type: text/html
  
3. Second request:
  
GET /syndication/rss.php HTTP/1.1
Host: www.pepysdiary.com
  
4. Unchanged feed sent again:
  
HTTP/1.x 200 OK
Server: Apache/1.3.19 (Unix) PHP/4.0.4pl1
Transfer-Encoding: chunked
Content-Type: text/html

If you're curious about which of these cases applies to your feed, one way to check is to use Mozilla's LiveHTTPHeaders extension, which is in fact how I took these snapshots.

Hats off to Paul Venezia for his exhaustive analysis of the Linux 2.6 kernel in this week's InfoWorld:

Will the new Linux really perform in the same league as the big boys? To find out, I put the v2.6.0 kernel through several real-world performance tests, comparing its file server, database server, and Web server performance with a recent v2.4 series kernel, v2.4.23. [InfoWorld: Linux v2.6 scales the enterprise, Paul Venezia]

Paul's not kidding, he went to the mat on this one. In a sidebar on the kernel development process, Paul notes that he twice went to the Linux Kernel Mailing List with what seemed to be -- and in fact were -- bugs. Here's the first LKML thread, and here's the second. Nice going!

Suppose that we bloggers, collectively, wanted to migrate toward HTML coding and CSS styling conventions that would make our content more interoperable. Since none of us is starting from a clean slate, we'd need to analyze current practice. Well, now we can. Here, for example, is a concordance of use cases for HTML elements with class attributes, drawn from the database I'm building:

With only a few days' worth of accumulated content, I wouldn't dare to venture any recommendations about these use cases. But as the picture develops over time, we might start to see opportunities for convergence.

Update: I've been hoping for some external validation of this approach, and Giulio Piancastelli provides it today. As part of a much longer posting with lots of detailed technical analysis of RDF-oriented techniques, he writes:

What Jon is searching for, I think, is a good balance between the cost of providing metadata and the benefits gained by working on the provided metadata, while trying not to entirely move away from the web world as we know it. In fact, this is probably the most important characteristic of Jon's experiment: he is working with what he is able to find right now, that is lots of HTML documents, which can be converted to be well-formed XML quite easily, and then searched by means of XPath. While these are ubiquitous technologies, it's difficult to find RDF files spreaded around as such: proving that the RDF world is query-enabled, stating that the right place where to put metadata are RDF files because you can probably get higher quality and more complete results is useless if there are little or no data to query.

From my personal perspective, I see those two worlds, one working with XML and XPath, the other messing around with RDF and RDQL, still very far from each other. Jon's experiment is helping us to become conscious of the fact we already are on a metadata path as far as web content is concerned: XML and XPath are probably the first steps in this journey, leading us to a more semantic web augmented with technologies which nowadays seems not to be successful, but that will hopefully prove to be useful when more complex needs arise. We can only hope the virtuous cycle will start to spin soon.

[Through the blogging-glass]

Amen. Thanks, Guilio!

I should probably get a life, but instead I can't stop myself from writing more new queries against my growing database of well-formed blog content. Here are some queries that find the following things in the last few days' worth of my inbound RSS feeds:

paragraphs containing links to apple.com

paragraphs that contain links to apple.com and mention 'XSLT'

paragraphs in items posted today that mention 'Orkut'

January items, posted by Joi Ito or David Weinberger, that mention mention 'Orkut'

items containing tables with cells that mention 'zipcode'

links to amazon.com that also contain images from amazon.com

Either I am crazy, or this is way cool. Or both.

The next phase of my structured search project is coming to life. For the new version I'm parsing all 200+ of the RSS feeds to which I subscribe, XHTML-izing the content, storing it in Berkeley DB XML, and exposing it to the same kinds of searches I've been applying to my own content. Here's a taste of the kinds of queries that are now possible:

quotes from Dare Obasanjo

links from Tim Bray

links from Brent Simmons to InfoWorld.com

books mentioned by AKMA

books, with XQuery in the title, mentioned by Michael Rys

The paint's not dry on this thing yet. I have yet to normalize the dates, and I'm still getting the hang of DB XML, but here are some things that become immediately obvious:

• Feeds that deliver only partial content are at a disadvantage.

• HTML Tidy is able to coerce a surprisingly large number of the feeds I take from HTML to XHTML.

• Once coerced, they're addressable in terms of the elements you find in HTML: links, images, tables, quotes.

Until now, I've thought the major roadblock standing in the way of more richly structured content was the lack of easy-to-use XML writing tools. But maybe I've been wrong about that. If it's going to be practical to XHTML-ize what current HTML writing tools, maybe we can make a whole lot more progress than I thought by working toward CSS styling standards that will also provide hooks for more powerful searching.

At the very least, this will be a nice laboratory in which to experiment with a growing pool of XML content, using a variety of XML-capable databases. My hope, of course, is to offer a service that's as useful to you -- the writers of the blogs I'm reading, aggregating and searching -- as it is to me. And ideally, useful to you in ways that invite you to think about how to make what you write even more useful to all of us. We'll see how it goes.

There's been some pushback recently, in the .NET blogging community, about Microsoft's habit of living in the future. For example:

It is abundantly frustrating to be keeping up with you guys right now. We out here in the real world do not use Longhorn, do not have access to Longhorn (not in a way we can trust for production), and we cannot even begin to test out these great new technologies until version 1.0 (or 2.0 for those that wish to stay sane). I know there's probably not a whole lot you can do, but this is a plea to you from someone "in the field". My job is to work on the architecture team as well as implement solutions for a large-scale commercial website using .NET. I use this stuff all day every day, but I use the 1.1 release bits.

Here's my point, enough with the "this Whidbey, Longhorn, XAML is so cool you should stop whatever it is you are doing and use it". Small problem, we can't. Please help us by remembering that we're still using the release bits, not the latest technology. [Michael Earls]

1. Easier deployment. The "end of DLL hell" was one of the early .NET battle cries. CLR metadata, enabling side-by-side execution, was going to make that problem go away. Well, has it? I hear a lot about ClickOnce deployment in Longhorn, but does the existing stuff work as advertised?

2. Unified programming model. It was obvious that wrapping years of crufty Win32 and COM APIs into clean and shiny .NET Framework classes, and then transitioning app and services to that framework, wasn't going to happen overnight. But, how much progress has been made to date?

3. Programming language neutrality. Here's a statement, from an early Jeff Richter article about .NET, that provoked oohs and ahhs at the time: "It is possible to create a class in C++ that derives from a class implemented in Visual Basic." Well, does anybody do this now? Is it useful? Meanwhile, the dynamic language support we were going to get, for the likes of Perl and Python, hasn't arrived. Why not?

4. Security. As security bulletin MS02-06 ("Unchecked buffer in ASP.NET Worker Process") made clear, not everything labeled ".NET" is managed. Still, there is a lot of .NET-based server code running now. Can we articulate the real benefits of .NET's evidence-based approach to code access security? And what have been the tradeoffs? For example, I've noticed that while .NET's machine.config adds a new layer of complexity to an environment, nothing is subtracted. You've still got Active Directory issues, NTFS issues, IIS metabase issues. How do we consolidate and simplify all this stuff?

5. XML web services. I'd say many of the original goals were met here. Of course the goalposts moved too. .NET Web Services, circa 2000, looked more like CORBA-with-angle-brackets than like service oriented architecture. But while Longhorn's Indigo aims for the latter target, it's worth considering how well the deployed bits are succeeding on their original terms.

6. XML universal canvas. I hoped the XML features of Office 2003 were going to deliver on this promise. But here, the jury's still out.

7. WebForms/WinForms. This is a tricky one. The original .NET roadmap charted two parallel courses for client-side developers, one for the rich client and one for the thin client. Or as we say lately: "rich versus reach." There wasn't a write-once strategy for combining the two -- and indeed, in Longhorn, there still isn't -- but it's probably useful to consider how the side-by-side strategy has played out.

8. Software as a service. Not much progress there, as Bill Gates acknowledged in a July 2002 speech in which he also lamented the failure of "building block services" -- what was envisoned as Hailstorm -- to emerge. What are the roadblocks here? Plenty of business and technical issues to consider.

9. Device neutrality. The Tablet PC has turned out to be a good platform for .NET apps. Phones and PDAs, less so, for reasons that will be interesting to explore.

10. User interface / personal information management. A bunch of important themes were sounded in the 2000 .NET rollout speech. Pub/sub notification. Attention management. Smart tags. Today, I'd argue, I'm getting a lot of these effects from blog culture and RSS. Going forward, Longhorn is the focus of the UI/PIM vision articulated for .NET. But living here in the present, as we do, it's worth considering which aspects of current .NET technology are making a difference on this front.

Over the next week or so, I'd like to have conversations with people on all sides of these (and perhaps other, related) issues. I'll be speaking with various folks privately, but here's a comment link (rss) for those who want to register opinions and/or provide feedback.

Here's a four-minute Flash movie containing three segments from an online demo of the latest version of Mindreef's SOAPscope. The presenter is Frank Grossman; a few others (including me) chime in occasionally. The segments are:

1. How SOAPscope integrates with the WS-I (Web Services Interoperability Organization) test tools.

2. How to invoke a WSDL service -- in this case, Microsoft's TerraService -- using SOAPscope to visualize inputs and outputs as pseudocode, and optionally modify and replay messages. You can try this yourself at XMethods.net, but the earlier version 2.0 of SOAPscope that's running there isn't as clever about converting enumerated types in the schema into picklists on the invocation form.

3. How SOAPscope 3.0 integrates with Visual Studio.NET.

Thanks to the Mindreef guys for playing along with this experiment, and to TechSmith for letting me test-drive Camtasia Studio. If folks think these off-the-cuff videos are useful, I'll try to do more of them. I'm involved in a lot of online demos, and showcasing them in this way is probably win/win both for the companies who present to me and for the readers of this blog.

Update: Just as I was noticing a playback problem, Frank Grossman wrote to report the same thing. Camtasia uses a secondary .SWF file, launched from this HTML, to control playback. Evidently, the idea is to make sure the movie plays at the correct screen size. But what I found, as did Frank, is that after the first time through, progressive playback of the video doesn't work on subsequent playbacks. So now I'm pointing directly at the primary .SWF file which, if you're running at greater than 1024x768 (the resolution of the demo) should work fine. If you're running at 1024x768, though, you'll want to use F11 to maximize the Flash player.

Test-driven development does require a lot of time and effort, which means something's got to give. One Java developer, Sue Spielman, sent a Dear John letter to her debugger by way of her Weblog. "It seems over the last year or two we are spending less and less time with each other," she wrote. "How should I tell you this? My time is now spent with my test cases."

Clearly that's a better use of time, but when up to half of the output of a full-blown TDD-style project can be test code, we're going to want to find ways to automate and streamline the effort. Agitar Software's forthcoming Java analyzer, Agitator, which was demonstrated to me recently and is due out this quarter, takes on that challenge. [Full story at InfoWorld.com]

E-forms, a technology that's been around for a long time, is now a hotbed of activity. Microsoft's XML-oriented InfoPath, which shipped with Office 2003 in October, is now deployed and in use. Adobe plans to ship a beta version of its PDF-and-XML-oriented forms designer in the first quarter of this year. And e-forms veterans such as PureEdge and Cardiff, whose offerings are built on an XML core, are lining up behind XForms, the e-forms standard that became an official W3C recommendation in October 2003. [Full story at InfoWorld.com]

The genius of Jon Udell's work is not sheer technical innovation (not that TransQuery amounted to anything like that either) but rather the ability to make sense of how such technologies can be used in simple but powerful ways over compelling content.

And not getting lost in the trees.

[Evan Lenz]

Co-incidentally I'm now reading XQuery from the Experts, and am having a curiously mixed reaction to the book. The geek in me is irresistably drawn to this Swiss-army-knife query language that so ambitiously straddles the realms of typed and untyped, hierarchical and relational, declarative and procedural. And I can't wait to use the corpus of XHTML blog content that I'm assembling to explore XQuery implementations, along with the XPath/XSLT techniques I've used so far.

On the other hand: so what? If I can't paint a picture of the forest that people can relate to, then planting a few more trees won't help. The notion of dynamic categories comes closest to answering the "so what?" question. But not close enough. When you work publicly, in blogspace, as I have been doing, reaction to your work is exquisitely measurable. And when I take the pulse of that reaction it's clear that I'm miles away from proving three points:

1. Ordinary Web content is already full of metadata,

2. which can enable powerful queries,

3. which, in turn, can motivate us to enrich the metadata.

With the release of MySQL 4.0, the licensing policy of the wildly popular open source database underwent a subtle change. The code libraries that client programs use to access the native MySQL API, formerly licensed under the LGPL (Lesser General Public License), were converted to the GPL. The LGPL was designed to exempt "nonfree" programs that link against open source libraries from the GPL's strong requirement to release source code. The purpose of the LGPL, according to the Free Software Foundation, is "to encourage the widest possible use of a certain library, so that it becomes a de-facto standard." And indeed, MySQL has become the database pillar of the so-called LAMP platform, whose acronym expands to Linux, Apache, MySQL, and the trio of Perl, Python, and PHP. [Full story at InfoWorld.com]

This week's column is only partly about licensing, though. It's also a cautionary tale about getting locked into database-specific access