While Microsoft isn't expected to talk about Longhorn at this week's SpeechTEK show in New York, the Redmond software maker is working to add speech and other "natural-user-interface" (NUI) technologies to its next-generation Windows operating system.

Microsoft's Natural Interactive Services Division (NISD) has been working on a "NUI Platform" that is designed provide users with "rich interaction" (speech, handwriting, natural language and even machine learning). The NUI Platform is expected to debut in Longhorn, Microsoft's Windows client due to ship in 2005+.

NISD includes folks plucked from a variety of teams across Microsoft. The 200 or so folks reporting to NISD corporate VP Kai-Fu Lee Lee include individuals from Microsoft Research's speech team; Microsoft's Entropic acquisition (who are familiar with the telephony-applications space); Microsoft's Office and Windows speech-API and speech-engine teams; Microsoft's natural-language group (which has pioneered Microsoft's grammar and spell-checker products); Windows and MSN Search groups; and programmers from Microsoft's Help and authoring-technology units.

Without running afoul of the Longhorn police, Lee talked to Microsoft Watch a bit about NUI and how it will manifest in Longhorn.

"Our approach is not to replace the GUI (graphical user interface)," Lee explains. "It is to augment it. We want to find new places where natural language can add value."

To do this, Microsoft is building a common foundation: an application programming interface, engine and platform that will find its way into lots of products, starting with Longhorn.

"Initially, there will be differences in the foundation platforms. There will be one for speech, one for gestures, one for handwriting," Lee says. The ultimate goal, however, it to build a least-common-denominator NUI platform that will work across all kinds of natural computer-human interactions.

Originally, Lee's group was almost entirely desktop-focused. In recent months, however, many of those individuals have been reassigned to work on Microsoft's evolving Speech Server platform and on embedded devices, Lee admits.

Read More About Speech Server Here

"In the long term, we are extremely bullish on speech," Lee says. "We believe it's the most natural way to have computers communicate. But in the short term, people don't want to talk to their computers. That is why most of our speech team is currently server-focused, not desktop-focused.

"We don't expect more than a few users to talk to their PCs, even by the time Longhorn ships," Lee continues, "other than a few hobbyists and (people) running certain applications, like CAD (computer-aided design), games, medical dictation and those needing it for accessibility."

Does that Longhorn will have fewer speech-recognition abilities than Windows XP Tablet Edition? Not at all, says Lee. But think about speech as an adjunct to elements such as the new Longhorn search and help systems, Lee says, instead of a replacement for keyboard entry.

In other words, if you want to tell your Longhorn computer to find a file, you will have the option of saying the word "Find," evoking a dialog box and/or drop-down menu on your screen.

This kind of capability is predicated on a new type of file system, Lee says.

"First, we need a file system that is more of a structured database," Lee explains. "You can't reason with everything being a file type. That's why we need WinFS," the Windows File System at the heart of SQL Server "Yukon" (and the data-store component of WinFS that will be embedded in Longhorn), Lee says.

"Then, we need to ask, if the richer (data) store lets you reason about nouns, why not verbs ... like 'format,' 'delete,' 'print'?" Lee continues. "If you think about 'find' as a verb, then everything fits into a reasonable taxonomy."

In the latter half of this decade, users will see more-advanced NUI technologies debut in future Windows releases and other Microsoft software, according to Lee.

An edited version of this article originally appeared in August 1, 2003, issue of the Microsoft Watch newsletter.