Drones take research to new heights

Is it…a balloon? An airship of some sort, tethered to the turf? A children’s pool, blown onto its side?

“This is an inflatable whale,” says David W. Johnston, whose position as assistant professor of the practice of marine conservation and ecology almost never brings him to the Nicholas School of the Environment on West Campus. Instead, he spends his time at the Duke Marine Lab, on Pivers Island in Beaufort, where he investigates ecology and the effects of people and climate change on the needs of marine vertebrates.

Like, for example, whales, though he usually focuses on ocean-dwelling mammalian ones, not inflatable ones on loan from the North Carolina Aquarium at Fort Fisher and strapped to the ground on a day when the wind is gusting to 20 mph. One of the ways he gathers information is through the use of drones—“unoccupied systems” in the parlance. Drones can explore and map territory. They can gather images after a storm and compare them with historical ones. And they can provide information about the number, the range, and even the health of animals like sharks, turtles, and whales.

Enter the inflatable whale, which is set up as an example. All morning drones from the lab’s squadron have buzzed it, taking pictures of the island that they will stitch together into high-definition mosaics that will include the whale. (At other times things like homemade wooden models of sharks and turtles have played the role of the inflatable whale.) The various drones—some rotor-style craft that hover like helicopters, others lightweight foam fixed-wing planes—track their altitude when each image is taken. Then applying trigonometry to measurements of the whale and the camera’s internal organs, Johnston and his crew of graduate students can calibrate the cameras so that subsequent images will yield completely trustworthy measurements of objects below.

It’s nice for them to have something to do, given that the drones themselves do pretty much everything else.

THIS IS AN ISSUE WORTH CONSIDERING. Drones are already astonishingly capable and complex: The drones Johnston’s group uses take care of the hardest parts of launch and landing, check on the wind, use the conditions to organize the course they need to trace, and decide to come back to base if anything gets out of whack. They don’t lick a finger, hold it up to the wind, and get a thoughtful look on their face while they cogitate, but that’s about all they don’t do. “As much as this hurts me as a pilot,” says Ph.D. student Everette “Rett” Newton, “it flies a lot more accurately than me.” And Newton, mind you, for twenty-eight years flew for a little organization called the U.S. Air Force, where he piloted things like the F-15E Strike Eagle.

Johnston’s group uses drones for photography, mapping, and other oceanographic research and observation needs; it also provides similar services to other academic departments, like classical studies (for which drone patrols have facilitated discovery at archaeological sites). Other labs at Duke explore just how much autonomous systems—drones, robots, vehicles—can do instead of us, without us, and even with us.

CONSIDER THAT DRONE—an eBee, a fixed-wing, airplane-style mapping drone made of foam that has a wingspan of about a meter, weighs a pound and a half, and can fly on its own for almost an hour. Of course, that’s not all it can do on its own. Johnston shows how the drone, working with a program on a nearby computer, responds to a boundary called a geo-fence—a border Johnston sets up on a map onscreen, in this case covering the island point where the whale sits. The drone uses its GPS system to orient itself with respect to the geo-fence.

Then Newton takes the eBee out near the whale, points it into the wind, and shakes it three times to wake up its internal motion sensors. Propellers whir, Newton flings the drone up, and it flies, tentatively meandering around while it gets a feel for the wind (if the wind hits 29 miles an hour, the drone comes home like a kid with a curfew; it does the same thing if it somehow accidentally crosses the geo-fence). Then it starts covering the territory. Johnston, watching the computer, listens for shutter clicks as the drone takes the pictures it will stitch together to make the single survey image of the territory. And once again, it does this on its own. It chooses its own route. When it gets done, it orients back into the easterly wind, heads home, slows down, and drops lightly to the ground at Newton’s feet like a good dog.

“It figures out what’s the best overlap and footprint,” Newton says. “I just tell it I want this area surveyed with this camera at 2 cm per pixel, and it does the rest.” They keep an eye on it the whole time, of course—it has a brief interaction with an eagle, which appears to want to challenge it for territory before losing interest—but mostly they just monitor as it buzzes back and forth, making a sort of dentist-drill whine when it’s overhead. “It’s windy, so it’s crabbing a little bit,” says Johnston, but he echoes Newton. “For it to be able to stay on course, the autopilot does a much better job than we can do. That autonomy part really helps with our sampling to make sure we get really good coverage.”

“And here’s the cool thing,” Newton adds. “We use that same autopilot for an autonomous ground rover, an autonomous boat, an autonomous submersible. The best of all possible worlds is when we can tie all those systems together for science.”

That is, wherever they need to put eyes, they can put them—and the eyes will, on their own, provide that coverage. Which is good enough that the Marine Lab has ceded Johnston half its iconic boathouse as a home for the new Marine Conservation Ecology Unoccupied Systems Facility, which focuses exclusively on the use of various autonomous systems to perform marine-science work like mapping coastlines, counting species, examining ecosystem health and productivity, measuring animal health and activity, and even doing marine archaeology. It’s not only the first facility the FAA has approved for drones in marine research. Right now there isn’t another university that has a place designed for using drones for marine ecology, Johnston says.

As a biological oceanographer, Johnston is interested in all the ways marine species and habitats interact with people. He looks into big-picture issues like climate change and localized ones like harassment of animals.

As he walks from the island point to the room where he’s preparing equipment for the new facility in the boathouse, Johnston explains. “So for me the drones are just a tool,” he says, “one of many tools we use to measure species.” They put tags on animals to gather data on their travels. With the capacity to measure and record position, speed, and depth, the tags can gather weeks’ worth of data on, say, a seal. The boathouse room is still being completed, so he shows off his workroom, a large lab filled with computer guts, whiteboards, and eight large screens on the wall. He picks up a blob of clear plastic with an antenna, a battery, and some electronic components inside: “This is a tag that we would glue on a seal,” he says. It has a cell-phone antenna, so when the seal comes near enough to shore to be picked up by a service, the tag automatically uploads its data.

“So you essentially have a data plan for a seal,” he says, smiling. And he can envision further streamlining— say, instead of merely uploading its data, the tag sends an automatic text message that turns on a nearby drone, which wakes itself up, checks for flying safety, and then goes out to take photographs or video of the seal. “You can imagine a future where you have one robot telling another robot what to do. It’s kind of like there’s no ceiling for what technology can help us do now.”

Photos. Videos. (The facility Twitter feed, @ MarineUAS, is a thing of astonishment and beauty.) Infrared photos. Underwater sound recording. Lasers that not only measure altitude with greater accuracy (altimeters commonly use air pressure, but that changes with wind speed) but also help with mapping tasks. He’s even flown drones through whale spouts, collecting what scientists call “exhalate” but you and I—and Johnston, to be honest—call snot. To check their health, because like every daycare provider knows, if the snot is icky, there’s something going on. With all these tasks, the drones gather enormous amounts of data.

Which is where the people come in. Graduate student Alex Seymour joins the discussion. “I’m just in the process of building the new computer; it’s part of the core structure that helps us manage the drone data.” He pulls an image up on the screens of how overwash from Hurricane Matthew affected Bird Shoal, the spit of land just offshore from the Marine Lab. The lab made drone flights over the shoal just before, just after, and a month after the hurricane and could measure changes in its structure. There were 2 meters’ worth of erosion on the sea side, new sand on the island’s shore side. “And on the recovery flight, interestingly enough, the area [on the front] is filling back in.”

The point is that the drones take these images cheaply (flights with a pilot would be enormously expensive), quickly, and routinely. “And we start to understand process at that point,” Johnston says. “It changed this amount because of this storm with this amount of wave energy.” More, the stitched-together images the drone create have such high resolution that in the resulting 3D images you can zoom in on a footprint or piece of trash.

Again, that’s a lot of data to manage, and the drones and computers can enable people to step out of the equation there, too. One Marine Lab team used drone-mounted infrared cameras to photograph seal rookeries in the North Atlantic. The result: images, covered in blobs, each blob a warm mammal on a cold shore (small blobs are babies; blobs of a certain size are groups). Ask grad students to count those blobs, and you’ve got twelve to fifteen hours of work per image; plus, the students might fall asleep. Seymour worked out an algorithm that counts them in two to five minutes and doesn’t get bored and stop to check Facebook. Plus, give three different grad students the same image, and you might get three different counts; tests have shown the algorithm doesn’t much differ from people’s counts. “And Alex’s algorithm does it the same way every time,” Johnston says.

The drones and their attendant computer programs save enormous amounts of money and time and free their human coworkers to do other things—like design research, of course, but also teach. The center hosted a workshop in 2015 that drew attendees from companies, universities, and federal agencies and had a workshop on West Campus last year as well as a summer 2016 course on unoccupied-system use in research. And Johnston and his charges remind anybody that they write the programs, design the problems, choose the research. Even once the systems are in the air, people aren’t completely out of the loop. But as Johnston says, “We’re mostly just monitoring at that point.”

WHICH, IT TURNS OUT, IS NOT REALLY WHAT WE’RE GOOD AT. This from Victoria Nneji, a Ph.D. student in the Humans and Autonomy Lab (which has the playful acronym HAL). She’s studying the development of autonomous aerial vehicles or “personal transportation drones,” which will carry people rather than cameras. Instead of doodling pictures of flying cars, Nneji is thinking infrastructure. “What kind of command center is necessary?” she asks, and she has a point. If the sky is full of drones carrying people— and packages, and cameras—no matter how good the programs are, doesn’t some actual person need to be keeping an eye on things? Yes, she says, and that’s the problem. Keeping an eye on things bores us.

“Compare with trains,” she says. Engineers on busy commuter trains stay busy; those on long, straight-line freight runs lose attention, because instead of actively controlling their vehicle, they’re simply monitoring it. “We’re not the best at monitoring tasks where a lot of boredom comes in.” She spent last year working with the Federal Railroad Administration on what is called “positive train control,” a way to protect against that human boredom factor that in many ways resembles the geo-fencing of Johnston’s drones. Trains would receive information about speed limits, workers in the area, other trains, and so forth, and could automatically avoid collisions and other problems that inattentive operators might miss, and that would be catastrophic if missed.

Regrettably, she says, “with that work I realized just how far behind our train system is here in the United States” compared with others worldwide.

The fact that the very computer systems that will manage the drones will keep an eye on most of the conflicts perversely creates the same problem the trains have: It makes people even less likely to be aware when trouble develops. “That’s why it’s important to recognize what humans are good at, compared to computers,” Nneji says.

Put a computer in charge of watching out a train windshield for hours and stopping the train if some concern arises, and it’ll do great; it has no attention span; its mind can’t wander. On the other hand, in an environment not limited to whatever’s straight ahead between the rails, computers are still in the infancy of their capacity to truly recognize what they’re seeing. Nneji sweeps her hand around a building lobby filled with chairs, tables, columns, windows. “We’re good at looking at that window and understanding what’s in front of it and behind it,” she says. We get how windows work. Computers just see objects and boundaries, still wrestling with things like windows or gestures. “Humans are better at making inferences.”

And leaps. If despite all the management issues they’ll raise you’re still wondering about flying cars, she notes that several companies are working on prototypes; some look like overgrown drones, with a dozen or so rotors. Don’t expect them to come to your yard any time soon (noise issues, for one thing), though parking garage top decks offer an opportunity. As for others? “Well,” she says. “Have you seen The Jetsons?”

Yes, we fall asleep when we’re trying to manage trains, but we’re on the road—sky?— to personal flying cars. She emphasizes that the time of autonomous drones carrying people around, when it arrives, will require above all human management, and human understanding of how to manage. Air traffic control? A national network? “What sort of people should be doing this? Pilots? I don’t think this would be the same as air traffic control,” she says about the way drones will travel. She suspects that what will emerge will be something closer to the way Uber works, with light, user-created management in many different places and people watching the systems to make sure they work. And Nneji uses the phrase “management by exception” to explain how computers and people can work together. The computers keep an eye on things as long as, according to algorithms the people write, everything looks okay. When the systems pick up something out of the ordinary, they alert the person in charge, who presumably pauses a Youtube video and checks things out.

HAL director and professor of mechanical engineering and materials science Missy Cummings, too, issues a reminder that as rapidly as autonomous systems—in cars and elsewhere— are advancing, they’re obviously not yet ready for introduction into the real ecosystem of human beings. The 2016 crash of a Tesla vehicle operating semiautomatically in Florida that caused the death of its driver bears that out. That subsequent investigations have blamed the crash on human error emphasizes Nneji’s point. “Even the Mars robot still has to have someone to supervise it,” Cummings says. You also have to remember that people bring an unpredictability factor. “They’ll find a way to cover up the cameras,” she says of the way people will likely turn driverless cars or buses to their own uses.

If you don’t think people are already figuring out how to spoof cars—confusing them by sending them false GPS signals, for example— you’re wrong, she adds. “This potentially could spark a bunch of new behaviors.”

BEHAVIORS THAT OTHER DUKE RESEARCHERS ARE TRYING TO ADDRESS even now. Kris Hauser, associate professor of electrical and computer engineering and of computer science in the Pratt School of Engineering, runs the Intelligent Motion Lab. He’s doing research into several ways in which people and robots interact.

Hauser has a team working on the DARPA (Defense Advanced Research Projects Agency) Robotics Challenge, working on getting a generally human- scale robot (it has two legs and arms and looks like it’s made of an Erector Set) to climb a ladder. Videos show it doing well in the lab, but a high wind was too much for it outdoors. Climbing a ladder, though, is simply repetitive motion: If the robot has one routine, it can just repeat it to complete the task. Hauser is more interested in complicated motions, like rock climbing, where a robot would have to figure out each next step on its own.

Wait—rock climbing? We need robots to go rock climbing? The point, Hauser says, is to address complex and unpredictable motions. “Think about getting into an airline cabin,” he says. “Contorting yourself around people, luggage, seats,” in a different way each time. “We’re trying to achieve versatility in robot locomotion.” Something like rock climbing goes to extremes, of course. “By focusing on the far end of extremes, we’re hoping everything easier can be done,” he shrugs, “for free.” As a rock climber himself, Hauser knows how complex the decision-making is, and if robots can learn to do that, climbing stairs and getting into and out of cars will be nothing. “We’ve gotten two limbs up on the wall so far,” so there’s a ways to go, though at least his RoboSimian, as the robot is called, is working.

RoboSimian was developed by NASA, and it uses a lithium-ion battery. There was one more working version until one at NASA, incorrectly hooked up for recharging, suffered the same fate as phones with lithium- ion batteries have suffered recently. (Find the video; it’s worth it.) Applications for the RoboSimian, once it can clamber to its heart’s content, will include, besides getting into and out of airplanes, search and rescue and space exploration.

If you’re wondering how it would feel to have a RoboSimian squeeze past you on an airplane to get to the window seat, you’re thinking about one of Hauser’s main interests. One place he’s pursuing that is with the Tele-Robotic Nursing Assistant, or Trina, a robot nurse that shares a lot of physical characteristics (as well as a job description) with Baymax, the inflatable robot in the movie Big Hero 6. Trina will someday, Hauser hopes, be able to do everything from bring food to patients to change their bedding and even draw blood. With obvious benefits not only to free up nurses but to tend to patients suffering dangerous communicable diseases, she’s gathered a lot of notice. She may not look human, but she’s generally vertical, has two arms, and has a screen where her face would be, which shows the face of the person working Trina. “Right now we do have a nurse behind the wheel, driving, so to speak,” Hauser says.

The problem is, where the characters in the movie almost reflexively hugged Baymax, Trina gets a different response. Engineers try Trina out with nurses and people who act as patients. “The fake patients tell us they are threatened by the robot,” Hauser says. Even though they know that there’s a nurse behind there, and they even see the nurse’s face on the screen, “they still feel this surrogate body the nurse has is still quite threatening.”

Nurses like to put hats and things on Trina and give her pet names to try to humanize her, but Hauser says, “this whole boundary between humans and robots is one of the great unsolved mysteries.” Trina has a long way to go before she’s ready for patient care anyhow— she’s pretty clumsy, and she’s slow—but that challenge gets attention elsewhere in Hauser’s lab, too. In a project on cooperative motion- planning, he works on the way people and robots work together. Not in the standard, I’ll-tell-you-what-to-do sense, but together, perhaps on an assembly line. “Robots that work alongside humans without safety cages and so on,” he says, illustrate the idea of “collaborative manufacturing.” He leads the way into what looks like an assembly table. “We’re exploring a mock industrial setup here,” he says, “where the robot is doing something around the human.”

The point? Taking the role of robot, he takes a step forward, uncomfortably close to a visitor, who instinctively steps back. “You see?” he says. “You’re already backing up.” That showed “avoiding behavior” to his “aggressive behavior.” He’s interested in how different people behave around robots, when the robots go about their business. Do people back away when the robot gets close, or do they push back in frustration? Do different personalities respond in different ways? “We also want to see what people do if the robot explicitly transgresses cultural norms,” he says. “It’s really interesting to take things out into the wild and see what normal people do.”

Whether it’s drones, autonomous cars on the ground or in the air, nurses, or coworkers, that sounds like a description of the next decade. As for whether people are even necessary, you can relax. Everybody thinks we still are.

If only to manage the robots.