Virtual Maestro

Music man: Wang plays theme song to Legend of Zelda video game using Ocarinaapp, with speaker “gloves” amplifying sound. Toni Gauthier

Jeff Smith had enjoyed plenty of success in the world of Internet start-ups. In 1993, just a few years after graduating from Stanford University, he had cofounded Tumbleweed Communications, a software company that specialized in email security, catering to large corporate clients. Over twelve years, he’d expanded the company, slowly acquiring other software firms, and eventually taking it public on the Nasdaq.

But by 2005, he was ready to move on. An avid pianist and composer in his spare time, Smith decided to leave the business world and return to graduate school at Stanford to follow a different passion: computer music. Early on, Smith was particularly inspired by a course on synthesizing sound, but even more so by the course’s instructor, first-year professor Ge Wang ’00. A talented programmer who specialized in computer-generated sound, Wang had written a new computer language dedicated to music performance. In his first semester at Stanford, he had founded a pair of novel ensembles: the Stanford Laptop Orchestra and the Mobile Phone Orchestra, both of which featured students composing and performing music on electronic devices.

“When I met him,” Smith says, “it was my conclusion that this guy was going to change music, he was going to change what it meant to [the world].”

As is often the case with those who have proven themselves successful in Silicon Valley, Smith was still routinely in touch with his former associates. It was early 2008 when a former investor asked Smith to review some new business ideas.

The previous fall, Apple had announced the upcoming release of the iPhone SDK, a software development kit that would allow third-party developers to create programs for the iPhone platform. In June, the company would debut an “App Store,” where applications could be sold directly to consumers. Though Smith says that most of the ideas presented in the investor meeting were duds, the resulting conversation concluded with a tentative plan for Smith to look into the new market for iPhone applications, with a particular focus on developing music-based programs.

Smith shared the idea with Wang, whose programming and music skills would lend themselves well to the project. The timing wasn’t perfect for either one of them—Smith had his Ph.D. work to keep him busy, and Wang was rushing to wrap up and defend his own Ph.D. thesis and adjusting to his new teaching duties—but after much discussion, they decided the opportunity was too good to pass up. “Wow,” Wang recalls thinking about the iPhone. “This is going to change how people do music, this device. But someone will have to actually be there to effect that change. And we might as well be part of that.” That summer, the pair launched SonicMule (later shortened to Smule), a start-up dedicated to developing interactive “social/sonic media.”

Over the past three years, the growing start-up has released nearly a dozen music-based apps for the iPhone and iPad, almost all of which have been unquestionable successes, commercially and critically. Collectively, its apps boast more than ten million active users. The staff, originally a bare-bones team of six, has grown to twenty-five, with a wave of additional new hires expected this fall. And it’s part of a growing industry. In April, Forbes reported that the mobile app market totaled about $2.2 billion last year, up 160 percent from the year before.

But Wang hopes to accomplish much more than simply establishing a successful company. His goal, audacious as it sounds, is to help change the way that music is produced, listened to, and shared around the world.

“I think the future of music-making is one where we might see the relationship of who is producing music versus who is consuming music actually changing,” Wang told the BBC last year. “Instead of a model where you have a few performers performing for many audience members, it actually might be a model where it’s many to many.”

“I think mobile devices [represent] a wonderful way to actually move in some sense towards that vision,” he added. “It’s perhaps the most personal and intimate computer that we’ve ever had.

And we have these types of devices in the hands of tens of millions, and soon more, people.”

Original schema for what would become the Ocarina.

On a Wednesday afternoon in late April in Smule’s Palo Alto, California, headquarters, Ge Wang is trying to explain some practical uses of the company’s latest iPhone app.

He pulls his iPhone out of his pocket and cues up the app, Magic Piano. The app turns a smartphone into a sort of musical instrument. “If something particularly epic is happening in your life,” Wang says, “you might play something like this,” and as green dots begin to float down the device’s touchscreen, he follows them with his fingers, tapping out the triumphant opening notes from Chariots of Fire.

“On the other hand,” Wang says, “if you’re feeling down, you might play something more like this.” Again, his fingers follow a series of descending green dots, but this time, the music that comes from the iPhone’s speakers is “100 Years,” pop band Five For Fighting’s soulful ballad about the passage of time. Over the past two weeks, staff members have been working long hours to get the new release—adapted from the original iPad version—just right. It was submitted to Apple yesterday, and today, in the wake of the storm, things are unusually quiet, save for Wang’s performance.

The office, located on the second floor of a two-story building just off Stanford’s campus, consists mainly of one large room with a wall of east-facing windows and desks in groups of four arranged in what Wang describes as “ninja-star formation.” Wang’s desk is in one corner, Smith’s is in the opposite.

The walls are decorated with colorful drafts of design documents used to build past apps, photos of staff members, and “Ahnold Film Festival” posters featuring Wang’s face Photoshopped in place of the former California governor’s.

One of two conference rooms features a long table, comfortable office chairs, and a big screen; it doubles as a site for weekly business meetings and a gaming studio. An X-box console, as well as faux instruments used in the popular game Rock Band, are stashed along one wall.

Tuned in, turned on: Rehearsals for Stanford Laptop Orchestra and Stanford Mobile Phone Orchestra concert at the university’s Dinkelspiel Auditorium in June.
Toni Gauthier

It seems possible to trace Wang’s path to computer music back to childhood, though you could also say that his interests weren’t all that different from that of the average boy growing up in the 1980s.

Born in Beijing, he spent most of his childhood in Kansas, where he grew up on classic video games like Mario Bros., Donkey Kong, and The Legend of Zelda. His first musical instrument was an accordion, a gift from his grandparents; his second was an electric guitar, which his parents bought him unprompted when he was thirteen. “In retrospect, that seemed like kind of an unconventional thing for parents to do,” Wang says. “To preemptively invite an instrument of rebellion and decibels into your home.” He took lessons from a teacher at a local music store and was soon jamming to Metallica and Guns N’ Roses. He loved experimenting with sound.

“My parents have always encouraged me to follow my interests,” he says. “They never hard sold me on anything. They waited to see what stuck. And music stuck.”

At Duke, Wang (who went by “Gary” at the time) studied computer science, excelling in programming courses, spending many late nights in the Teer Building’s computer lab, and often serving as a de facto teaching assistant and debugger for friends. But he also balanced out his schedule with music courses: composition, theory, music history.

One course in particular stood out for him: “Electronic Music,” taught by music professor Scott Lindroth, who is now Duke’s vice provost for the arts. It was during that class that Wang first heard a recording of “Table’s Clear,” an experimental piece of computer music by composer Paul Lansky. The piece begins withseemingly random clanks and bangs, the sounds of kitchenware being handled roughly. But as the piece goes on, the sounds begin to organize themselves into a musical groove. “This was the first piece of computer music that moved me musically,” Wang says. “It was like, I want to do that, or I want to help people do that.” The course also gave him his first opportunity to compose and record his own computer music.

Until this point, Wang, like many of his computer-science classmates, had envisioned a career in programming, either with a software giant like Microsoft or maybe with a videogame design company. But now a second path appeared. Lansky, an acquaintance of Lindroth’s, taught at Princeton University, which has a renowned sound lab. Wang applied to the graduate program there and was accepted.

Wang’s first foray into the start-up world also came during his time at Duke. With four friends he hatched a plan to launch an Internet site that would compile, summarize, and synthesize online reviews for a wide array of consumer products. This was in the late 1990s, when reviews of this type had just begun piling up online. The five packed their things into a U-Haul and headed north to Cambridge, Massachusetts, where they rented a townhouse and got to work on their site.

“The Internet has gone through peaks and valleys in terms of levels of excitement” that it raises among investors and entrepreneurs, says Matt Killingsworth ’00, the group’s leader. “At this point it felt like anything was possible, that this is the future.” But after a few months, Killingsworth says, “we began looking at the financial assumptions we and others in the Internet industry were making.”

“In the process of raising round one of financing,” he says, “we became increasingly skeptical. The rate at which we were assuming we could turn visitors into revenue seemed less and less feasible.” When the time came to decide whether they should request an additional semester’s leave of absence from Duke, they decided to pack up and head back to campus. The following spring, the dot-com bubble burst. (Killingsworth is now pursuing a Ph.D. in psychology at Harvard University. His thesis involves using a smartphone app he developed to more accurately measure and understand happiness in humans.)

In graduate school, Wang worked under renowned composer and computer musician Perry Cook, who had a joint appointment in Princeton’s computer science and music departments. Wang immersed himself in computer music.

But the programmer in him was always tinkering. Since his time at Duke, Wang had spent a lot of time thinking about the nuances of computer languages. His experience debugging friends’ programs made him “appreciate when software…was designed in a way that makes people’s lives easier.”

There are thousands of computer languages out there, including dozens designed specifically for composing computer music, but despite a great deal of experimentation, Wang couldn’t find one that met all of his needs. “One day I came to Perry,” Wang recalls, “and I said, ‘Perry, I know there are a lot of programming languages out there for music. I think I want to build yet another one.’”

He explained the basics of the new language he was proposing. Cook took one look, and said, “Okay, that sounds pretty insane. Go for it.”

The resulting language, dubbed ChucK (after the verb meaning “to carelessly throw”) and released as open-source software in 2003, addressed several problems that Wang saw as inherent in its competitors. Most significantly, it allowed programmers to modify code on the fly, changing and embellishing their compositions during performances.

One early test for the language came in 2005, when Wang, along with fellow graduate student Scott Smallwood and professors Cook and Dan Trueman, founded the Princeton Laptop Orchestra (PLOrk). The orchestra, originally designed as a freshman seminar course, was intended to teach students a variety of skills, including programming, music composition, and live performance.

“People learned programming because they had to go create a musical instrument and a performance, and they were going to perform it in front of the class or in front of an audience,” Wang says. “The programming becomes a tool and not the end goal.”

Though the class comprised fifteen college freshmen, none of whom had any significant programming experience, the experiment was a great success, Wang says. “They rocked it. We were scared. They were not. They did it, had tons of fun with it.”

When he came to Stanford, he brought the idea with him, founding the Stanford Laptop Orchestra, or SLOrk, during his first semester on campus. In the course, now designed for upperclassmen and graduate students, orchestra members learn to perform pieces first created at Princeton, as well as compose their own. Wang’s other courses are similarly project-based; in one, students design and build mobile apps similar to those developed at Smule.

His hands-on approach to teaching was one of the big things that attracted faculty members at Stanford to him. “He has a really outgoing demeanor and a way of privileging innovations,” says Chris Chafe, director of the university’s Center for Computer Research in Music and Acoustics. “And not just his own. He’s really interested in others’ work.”

When Wang first at Stanford, he had no intention of founding any sort of company. But when Smith came to him talking about mobile music apps, it got him thinking. He already had some experience programming for smartphones. In 2007, in collaboration with Georg Essel, an assistant professor of computer science at the University of Michigan, and Henri Penntinnen, a Finnish researcher, Wang had launched a new project: the Stanford Mobile Phone Orchestra (MoPhO). As with SLOrk, the student members of MoPhO composed and performed electronic music, but instead of making music through keystrokes and trackpads, they played on touchscreen smartphones, the output of their Nokia 95s amplified by custom-made gloves outfitted with speakers.

Compared to the old Nokia phones, programming with the iPhone SDK was a breeze. The popularity of the new iPhone also offered them the advantage of a large-scale audience for social applications. “I realized that in the vein of ubiquitous computing, just building something is not enough,” Wang says. “It needs to be used. It needs to be in the hands of not hundreds, or thousands, but millions, or hundreds of millions. When you reach those different scales, different things become possible, socially and musically. And as a researcher, that is irresistible. I felt like both academia and the commercial world had something to offer this.”

Their first product was not music-based, or at least not obviously so. It was a virtual cigarette lighter of the sort that concert fans have become fond of waving. But it was different from the scores of other lighters in the App Store. The app features a clean black screen with a remarkably realistic flame that shifts when you brush your finger across it and actually seems to singe the edge of the screen if you tilt it too far one way or the other. You can also extinguish the flame by blowing into the phone’s microphone, or, using the flamethrower feature, pass the flame to another iPhone that has the app. This is the sonic part of the app. The phone is responding to soft audible cues passed from speaker to mic.

On a whim, Wang added a social feature to the app, a moving image of a globe that users could pan across, seeing where in the world others were using the app. The app was social, but it did not rely on the same explicit links as other social networks. It preserved a sort of anonymity that Wang thought was powerful. “This is here to show you that you’re not the only one. Like you, these other people around the world have also paid a buck to get a fake lighter on their iPhone.” The globe has now become a recurring feature in the company’s apps.

Their next project was more ambitious. They would turn the iPhone into a flute-like musical instrument inspired by an ancient wind instrument featured in one of the early Legend of Zelda video games, the ocarina. For this product, Wang pulled out all of the stops, making use of every iPhone feature he could think of. The design is colorful. Users blow into the device’s microphone while fingering virtual holes that appear on its touch screen. Tilting the device, thereby alerting the built-in accelerometer, changes the pitch of notes, and the phone’s GPS makes performances plottable on a map. Wang called the instrument Ocarina.

Electric: Ge Wang makes sure all systems are go before final concert of the season.
Toni Gauthier

Ocarina proved to be a massive success, commercially and critically. It earned raves from critics at tech-focused publications like TechCrunch and PC Magazine, but also caught the attention of general-interest publications like Newsweek, Scientific American, and The New York Times. Times technology critic David Pogue invited Wang onstage at the MacWorld 2009 Expo to demo the Ocarina. Accompanied by Pogue on keyboard, Wang played The Beatles’ “Yesterday.”

The app racked up 400,000 downloads in its first month and quickly became the number-one music app in the U.S. and in twenty other countries. Since 2008, it has been downloaded more than five million times.

Wang hypothesizes that the Ocarina, as well as other Smule products, have been successful because they allow people to be expressive. Users have created a catalogue of tablature for thousands of songs on the Ocarina. From his desk at Stanford, he pulls up the program on his desktop computer. He clicks on a recently uploaded Ocarina piece, and notes begin to trickle out.

“This is coming from.…” He squints at the screen, trying to divine international borders on the blue and green globe. “Is that Russia?”

Since the app was first released in November 2008, people have listened to each other on the app more than 50 million times. Users have left comments on Smule’s site saying that listening to the Ocarina has replaced TV as their regular dinnertime entertainment.

In the wake of Ocarina’s success, Smule has released several instrument-based apps, including Magic Piano, Magic Fiddle (inspired, supposedly, by a symphony encore in San Francisco during which pianist Lang Lang played the classical piece “Flight of the Bumblebee” on an iPad), and Leaf Trombone.

They have also teamed with singer T-Pain and the producers of the hit Fox show Glee to produce two branded karaoke-style apps.

T-Pain’s app was especially fun. The singer behind the tracks “Buy U a Drank” and “I’m N Love (Wit a Stripper)” is known for his constant use, or overuse, of Auto-Tune, a digital pitch-correction software that Time magazine once likened to “Photoshop for the human voice.” Used sparingly, the software smoothes out vocals, eliminating off-pitch notes; used as T-Pain does, it makes the human voice sound mechanical and fake. The T-Pain app adds Auto-Tune to the iPhone’s microphone and lets users record themselves singing along with T-Pain songs or just about anything else.

“The irony is that [T-Pain] doesn’t actually need Auto-Tune to be good,” Wang says. “He uses it purely as an artistic gesture. He’s certainly pushed the Auto-Tune to its boundaries in a way that Auto-Tune was never intended to be used.” At this, Wang smiles, as if he’s found a kindred spirit.

Ang and Smule have had an impressive three-year run. According to cofounder Jeff Smith, the company brought in about $1 million in revenue in its first full year of operation, 2009, selling apps for between $1 and $3. Last year, they made $4 million. This year, they hope to double that.

But they’re not resting. In addition to developing new apps, Wang and Smith say that they are trying to further develop the game-like aspects of their existing apps, for example, by including challenges and levels that users must pass in order to earn awards, in this case a currency tentatively titled “Smoolah.”

They are also looking at converting some of their apps to a “freemium” model, meaning they would allow users to download the app itself for free, but then charge for add-ons. The iPod edition of Magic Piano, released in May, represents their first attempt at this strategy. Initially, they will charge for extra songs; in future editions of the game, they hope to be able to offer different sounds—for example, a harp or a synthesizer—as well.

For the foreseeable future, Smith adds, Smule apps will only be available on Apple devices—other companies, he says, “are not interesting from a commercial standpoint today”—though that may change as Google’s Android is integrated into more phones.

In some ways, all of these new music apps seem like a classic example of the Internet’s ability to promote over-sharing. Glee karaoke, for example, lets millions of anonymous users belt it out for the world to hear. It does for music what blogs, Facebook, and Twitter did for opinion—it lets individuals embrace their need, their right, to be heard. David Pogue, the Times reviewer, wrote, “The result—professional backup band, you processed to sound gorgeous and perfect—is exhilarating, no matter how rotten a singer you are.” But a quick sampling of tracks uploaded by users indicates that not everyone is cut out to be an American Idol.

At the same time, the apps do encourage unprecedented, and unexpected, types of musical collaboration. In March, in the wake of the massive earthquake and tsunami in Japan, a young woman from Tokyo turned on her iPhone and pulled up Smule’s Glee karaoke app. Using the phone’s built-in mic, she recorded herself singing along to the song “Lean on Me.” She uploaded the song to the Glee karaoke website, where it could be heard by other users, and invited a few friends to chip in vocal tracks.

The response was astounding. Over the next week, requests poured in from complete strangers around the globe, asking for invitations to contribute their own voices to the project. Within ten days, the track featured more than 1,500 voices, most perfect strangers, all singing in support of Japan.