According to his official biographer, Steve Jobs went ballistic in January 2010 when he saw HTC's newest Android phones. "I want you to stop using our ideas in Android," Jobs reportedly told Eric Schmidt, then Google's CEO. Schmidt had already been forced to resign from Apple's board, partly due to increased smartphone competition between the two companies. Jobs then vowed to "spend every penny of Apple's $40 billion in the bank to right this wrong."
Jobs called Android a "stolen product," but theft can be a tricky concept when talking about innovation. The iPhone didn't emerge fully formed from Jobs's head. Rather, it represented the culmination of incremental innovation over decades—much of which occurred outside of Cupertino.
Innovation within multitouch and smartphone technology goes back decades—the first multitouch devices were created in the 1980s—and spans a large number of researchers and commercial firms. It wouldn't have been possible to create the iPhone without copying the ideas of these other researchers. And since the release of Android, Apple has incorporated some Google ideas into iOS.
The "Digital Desk" in action, showing a pinch-to-resize gesture in 1991. Pierre Wellner
You can call this process plenty of names, some less than complimentary, but consumers generally benefit from the copying within the smartphone market. The best ideas are quickly incorporated into all the leading mobile platforms.
The current legal battles over smartphones are a sequel to the "look and feel" battle over the graphical user interface (GUI) in the late 1980s. Apple lost that first fight when the courts ruled key elements of the Macintosh user interface were not eligible for copyright protection. Unfortunately, in the last 20 years, the courts have made it much easier to acquire software patents. Apple now has more powerful legal weapons at its disposal this time around, as do its competitors. Together, there's a real danger that the smartphone wars will end by stifling competition.
Multitouch in the lab
High-tech innovations are often developed by laboratory researchers long before they're introduced into the commercial market. Multitouch computing was no exception. According to Bill Buxton, a multitouch pioneer now at Microsoft Research, the first multitouch screen was developed at Bell Labs in 1984. Buxton reports that the screen, created by Bob Boie, "used a transparent capacitive array of touch sensors overlaid on a CRT." It allowed the user to "manipulate graphical objects with fingers with excellent response time."
In the two decades that followed, researchers experimented with a variety of techniques for building multitouch displays. A 1991 Xerox PARC project called the "Digital Desk" used a projector and camera situated above an ordinary desk to track touches. A multitouch table called the Diamond Touch also used an overhead projector, but its touch sensor ran a small amount of current through the user's body into a receiver in the user's chair. NYU researcher Jeff Han developed a rear-projection display that achieved multitouch capabilities through a technique called "frustrated total internal reflection."
Jeff Han demonstrating his multitouch work in 2006. TED
While they refined multitouch hardware, these researchers were also improving the software that ran on it. One of the most important areas of research was developing a vocabulary of gestures that took full advantage of the the hardware's capabilities. The "Digital Desk" project included a sketching application that allowed images to be re-sized with a "pinch" gesture. A 2003 article by researchers at the University of Toronto described a tabletop touchscreen system that included a "flick" gesture to send objects from one user to another across the table.
By February 2006, Han brought a number of these ideas together to create a suite of multitouch applications that he presented in a now-famous TED talk. He showed off a photo-viewing application that used the "pinch" gesture to re-size and rotate photographs; it included an on-screen keyboard for labeling photos. He also demonstrated an interactive map that allowed the user to pan, rotate, and zoom with dragging and pinching gestures similar to those used on modern smartphones.
Commercializing multitouch
In 2004, a French firm called Jazzmutant unveiled the Lemur, a music controller many consider the world's first commercial multitouch product. The Lemur could be configured to display a wide variety of buttons, sliders, and other user interface elements. When these were manipulated, the device would produce output in the MIDI-like Open Sound Control format. It debuted in 2005 and cost more than $2,000.
The market for the Lemur was eventually undercut by the proliferation of low-cost tablet computers like the iPad. But Jazzmutant now licenses its multitouch technology under the name Stantum. It raised $13 million in funding in 2009.
Jeff Han also moved to commercialize his research, founding Perceptive Pixel in 2006. The firm focuses on building large, high-end multitouch displays and counts CNN among its clients. The DiamondTouch also became a commercial product in 2006.
Microsoft says its researchers have worked on multitouch technologies since 2001. Microsoft's Andy Wilson announced Touchlight, a multitouch technology using cameras and a rear projector, in 2004. Touchlight had an interface reminiscent of Minority Report — a three-dimensional object would be displayed on the screen and the user could rotate and scale it with intuitive hand gestures.
Wilson was also a key figure in developing Microsoft Surface, a tabletop touchscreen system that used a similar combination of a rear-projected display and cameras. According to Microsoft, the hardware design was finalized in 2005. Surface was then introduced as a commercial product in mid-2007, a few months after the iPhone was unveiled. It too used dragging and pinching gestures to manipulate photographs and other objects on the screen.
Another key figure in the early development and commercialization of multitouch technologies was Wayne Westerman, a computer science researcher whose PhD dissertation described a sophisticated multitouch input device. Unlike the other technologies mentioned so far, Westerman's devices weren't multitouch displays; they were strictly input devices. Along with John Elias, Westerman went on to found FingerWorks, which produced a line of multitouch keyboards that were marketed as a way to relieve repetitive stress injuries.
Fingerworks was acquired by Apple in 2005 and Westerman and Elias became Apple employees. Their influence was felt not only in the multitouch capabilities of the iPhone and the iPad, but also in the increasingly sophisticated multitouch capabilities of Mac trackpads.
Touchscreen phones
IBM's Simon, introduced in 1993, is widely regarded as the first touchscreen phone. It had a black-and-white screen and lacked multitouch capabilities, but it had many of the features we associate with smartphones today. Users dialed with a onscreen keypad, and Simon included a calendar, address book, alarm clock, and e-mail functionality. The e-mail app even included the ability to click on a phone number to dial it.
IBM's Simon, introduced in 1993 TekGadg
The Simon was not a big hit, but touchscreen phones continued improving. In the early 2000s, they gained color screens, more sophisticated apps, and built-in cameras. They continued to be single-touch devices, and many required a stylus for precise user input. Hardware keypads were standard. These phones ran operating systems from Microsoft, Palm, Research in Motion, and others.
April 2005 saw the release of the Neonode N1m. While lacking the sophistication of the iPhone, it had a few notable features. It was one of the few phones of its generation not to have a hardware keypad, relying almost entirely on software buttons for input. It supported swiping gestures in addition to individual taps. And it employed a "slide to unlock" gesture, almost identical to the one the iPhone made famous.
More sophisticated touchscreen interfaces began to emerge in 2006. In October, Synaptics unveiled the Onyx, a proof-of-concept color touchscreen phone that included a number of advanced features. While it may not have been a true multitouch device, its capacitive touch sensor included the ability to tell the difference between the user's finger and his cheek (allowing someone to answer the phone without worrying about accidental inputs) and to track a finger as it moved across the screen.
The Onyx's phone application had an intuitive conference calling feature, and the device included a music player, an interactive map, and a calendar.
LG Prada in March 2007 giiks
That December, LG announced the LG Prada — beating the iPhone to market by several months. The two devices shared several common features. The Prada dispensed with a traditional keypad, relying on software buttons for most input. It included the ability to play music, browse the Web, view photos, and check e-mail.
The iPhone was finally unveiled in January 2007. LGaccused Apple of copying its design, saying it was disclosed in September 2006 in order to compete for an IF Design Award (which it won). The accusation doesn't hold much credibility, however. Although the phones have undeniable similarities, the iPhone features a more sophisticated user interface. For example, the iPhone used the flick-to-scroll gesture now common on smartphones; the LG Prada used a desktop-style scroll bar. The two phones were likely developed independently.
So is Android a stolen product?
Great artists steal
In a sense, the answer is almost certainly "yes." It's hard to imagine how Google could have prevented some iPhone innovations from seeping into Android design. The iPhone was the talk of Silicon Valley in 2007 and 2008. It would have been practically impossible for the Android development team to avoid learning about iPhone features. Once Google's engineers were exposed to the concepts Apple pioneered, they couldn't help but be influenced by them.
But if Google is guilty of using Apple's ideas, Apple is equally guilty. Many researchers and companies invented technologies that predate the iPhone but made it possible. As Microsoft's Buxton points out, Wayne Westerman (the multitouch researcher who sold his startup and became an Apple employee in 2005) cited the work of numerous early multitouch researchers in his 1999 PhD thesis. The iPhone incorporated key innovations pioneered by Bob Boie, IBM, Jazzmutant, Jeff Han, and others.
Indeed, what made the iPhone such a great product was precisely that Apple drew together a number of innovations already developed separately—touchscreen phones, capacitive touchscreens, sophisticated multitouch user interfaces—and combined them in a product greater than the sum of its parts. This pattern of combining and refining of previous innovations is the rule, not the exception, in innovative industries. Android is simply the latest example of the pattern.
While Android clearly adapted some important iPhone innovations, Google's engineers also added refinements of their own where they felt Apple's approach lacked them. For example, the early iPhone notification system left much to be desired. As we put it last year:
Every new SMS or push alert showed up as a modal dialog box in the center of the screen, forcing the user to make a decision immediately. The system also offered no way to see multiple notifications at once.
Google came up with a more flexible notification scheme. Rather than presenting them in a modal dialog box requiring immediate input before a user could proceed, Android displayed notifications briefly at the top of the screen, then added an icon atop the screen to signal existing notifications. When a user was ready to respond, she simply pulled down the notification bar to see a list of all pending notifications.
The Notifications Center in iOS 5, 2011 Apple Inc.
Apple revamped its notification system in iOS 5, introducing a Notifications Center that was strikingly similar to Android's notification scheme. Apple added its own refinements, such as the ability to add widgets displaying the weather, stock prices, and other frequently-updated information. But the basic approach—notifications displayed at the top of the screen accessible through a pull down gesture—is virtually identical to the approach Google invented.
Users benefit from this kind of copying. Google's notification scheme was better than the original iPhone notifications, so it is in iOS users's interests for Apple to copy the idea. The alternative—a world in which companies scrupulously avoid using each other's ideas—would be much worse. It would become impossible to buy a smartphone incorporating the best innovations from across the industry.
Inventing in the dark
Legally, the question is whether Google infringed on Apple's patent, copyright, or other possessions. Google appears to be on safe ground from a copyright perspective. Android is built on Linux and uses a Java-like virtual machine; iOS is built on Darwin and uses NeXT-derived Objective C frameworks. We don't know of any allegations that Android was developed with literal copies of iOS code.
But whether Google infringed on Apple's patents is a harder question. And it would have been especially difficult to answer as Google was creating the first versions of Android.
Patent law generally gives a firm like Apple one year from the public disclosure of an invention to file for a patent on it. Apple unveiled the iPhone in January 2007, so the filing deadline for iPhone-related inventions would have been in January 2008. After filing, there is an additional 18-month delay before applications are made public. So if Apple filed an iPhone-related patent application on the last day before the deadline, Google wouldn't have learned of its existence until July 2009—almost a year after the first Android phone hit the market !
And even after patent applications are made public, it can take several more years for the patent office to make a decision on them. There's also no certainty about what a granted patent covers or whether it will stand up in court.
In short, Eric Schmidt's Android development team would have had no idea in 2008 which ideas were, legally speaking, Apple's ideas. The only foolproof way to avoid infringing Apple's patents would have been to avoid a multitouch phone OS at all.
History repeating
Steve Jobs eventually made good on his threats against Android. While Apple has not sued Google directly, Cupertino is now locked in legal battles with a number of major Android vendors. And unlike Microsoft, which has focused on signing licensing revenues with its patents, Apple seems genuinely determined to drive Android devices off the market.
Xerox Alto, which pioneered the graphical user interface in the 1970s Marcin Wichary
This isn't the first time Apple built a new user interface based on the ideas of others, then sued competitors for using those same ideas. The graphical user interface now standard on desktop computers can be traced back to the invention of the mouse by SRI's Doug Engelbart in the 1960s. The ideas were refined at Xerox PARC in the 1970s, where Steve Jobs famously led a group of Apple engineers to visit in 1979. Five years later, Apple introduced the Macintosh. Those ideas then found a much larger audience.
Microsoft scrambled to catch up, releasing the first version of Windows in 1985. In 1988, Apple filed a lawsuit accusing Microsoft of stealing the "look and feel" of the Macintosh. Xerox got involved in 1989 with a lawsuit accusing Apple of stealing the ideas behind the Macintosh from Xerox researchers. The courts eventually ruled key user interface concepts behind the Macintosh were not copyrightable, removing the legal cloud that had hung over early GUIs.
In many ways, today's smartphone litigation is a sequel to that far-off "look and feel" fight. But there's an important difference: a series of court decisions in the 1990s effectively legalized software patents. As a result, incumbents like Apple have more powerful legal weapons to use against would-be competitors.
Google has spent billions on patents to help it fight back against legal actions by Apple, Microsoft, and others. The search giant may have deep enough pockets to buy the patents it needs to defend itself, vindicating its right to compete in the mobile computing market.
But firms without billions may not be so lucky. The failure of Apple's original look and feel lawsuit cleared the way for smaller firms (like Jobs's own NeXT) to compete in the desktop computing market. In contrast, if a firm today has a great idea for a mobile OS but lacks thousands of patents or the billions of dollars it takes to acquire them, it will likely be defenseless against angry patent lawyers. Apple's demand that companies "stop using our ideas" will have real teeth.
That's bad for firms that want to get into the mobile OS market. It's also bad for users, who may be deprived of innovations these firms can bring to the market. But it's great for patent lawyers.
What do I think about it?... If Android is a "stolen product," then so was the iPhone. PERIOD.
- Total editing time of this post: 17 Hours and 55 Minutes.
