Recently T3 travelled to Nissan's high-tech design centre in Paddington, London, to talk with Dr Maarten Sierhuis. Maarten not only currently heads-up Nissan's Silicon Valley-based research team but, in a 12-year career at NASA, he built a computer language that was used to develop an intelligent system for all communication between Mission Control and the International Space Station. Today, he utilises that expertise to drive forward Nissan's pioneering work in autonomous systems.
The resulting interview, which is authoritative and refreshingly candid, gives a snapshot into his visionary work.
T3: Maarten, you have likened a person’s car to a spacesuit. Could you explain what you mean by that?
Dr Maarten Sierhuis: Sure. Well that is really an anecdote from working with astronauts at the Johnson Space Center with NASA. There I was working on autonomous technology in their spacesuits and, to put it bluntly, the astronauts don’t mince their words. There was one guy that said to me, “it’s my home, it’s my work environment and it’s my life support system - don’t F it up!”
"The car is your home, your work environment, and it is your life support."
So, while I was at NASA I did a lot of work with external robots, robots that I interacted with as a physical entity, and - directly linking to the work I’m undertaking at Nissan - an autonomous vehicle is like a robot in that sense. But, crucially, it’s a robot that you are inside of. It creates a different relationship, a relationship between human and technology that is akin to a spacesuit. The car is your home, your work environment, and it is your life support. It needs to be safe. So that’s, for me, what the analogy is really about.
The Nissan IDS Concept car designed to show off autonomous systems
T3: So, immediately picking up on that point of the car being an important daily life support system, where would you say vehicles are currently failing in that regard?
Dr Sierhuis: Well, many things pop into my head when you say that. From the home perspective, if you think about car share, I don’t like sharing my bedroom or living room with four strangers. My home is my personal space. A car, I feel, is also your personal space.
With the analogy of the home it is interesting, as we have anthropologists in our office that are studying how people behave in traffic and how they communicate. We had this video where a person was driving and the researcher, who was sitting in the back, needed to charge their phone and asked for a charger. The driver just went, sure no problem, and reached over without looking and grabbed the charger. In that sense, the car is just like your home - you should be able to instinctively know where things are.
“I’ve come to realise that braking is a stupid activity”
Work is a different matter. Right now we don’t really have cars where you can work. Maybe if you are passenger, but if you are a driver then not at all. It’s funny really, I used to work for a phone company before I went to NASA and every morning I was driving with my colleague and we would do research together in the car. The one who was not driving had his laptop open and we we’re working. And we always would say that the phone company is getting a lot more out of us than it is paying us for! We would commute 90 minutes in each direction, so they were getting almost an extra three hours out of us each day!
So, work in the car, is a nice thing to try to achieve with autonomous systems, which is my main area of research at Nissan.
Maarten, when not gunning it in his Nissan GT-R Black Edition, drives an Infiniti Q50
T3: So, jumping forward to that work you are undertaking at Nissan, could you provide me with some examples of how you are making car environments better and the vehicles themselves safer and more autonomous?
Dr Sierhuis: Well, making the vehicle safe is an easier one to address I think. From the research side we are working mostly on the autonomous city driving. So, from the city driving point of view, I can answer the question in two ways. We are working on making the car as safe as possible from the point of view of reacting and stopping for anything that comes in front of it. We’re making sure the car doesn’t hit anything, that it drives as smoothly as possible to ensure a nice experience for the person in the car, and I think that with autonomous sensors and systems we can be faster to reacting to problems than humans can.
“As you make systems more intelligent, it is really hard to know where the limits are for that intelligence.”
With the Infiniti Q50 that I drive, I’ve been driving it four years and I travel with the Digital Adaptive Steering and Predictive Forward Collision Warning system always on. The reason is because it is amazing how much safer it is in braking than I am.
So, I’ve recently been reflecting on this point as I’ve been driving the car for so long. These autonomous systems are designed, primarily, for highway driving. But I drive mostly in the city, as I live in San Francisco. In city driving I only have to look for a second to the right when somebody comes from the left and I don’t see it and something can go wrong. But this system can see in 360 degrees, right, and in those situations it will brake for me. And what I now have noticed is that if I drive in a car where I have to brake, I’ve noticed how cognitively difficult that is. Like if I’m talking to someone next to me, I notice now that I have to pay attention way more to what happens in front of me. I’ve noticed this in the last couple of months.
Now if I’m in a car that doesn’t have those sort of autonomous systems it really bothers me. I’ve come to realise that braking is a stupid activity. It is not something that a human should be doing. I’ve way more important activities I could be doing other than braking. And it has been fascinating for me to feel that way myself. So I’m like, give me that no hands on the steering wheel thing anytime now because I can do other stuff!
This is what the challenge with autonomous systems is all about. As you make systems more intelligent, it is really hard to know where the limits are for that intelligence. Humans have to trust the system, they need to know what the system is doing before we feel relaxed - and that takes a while.
At Nissan we have a group of social scientists at my centre that study how people drive and they’ve recently started studying how people drive in the Infiniti Q50. The reason why we can do that so easily is because so many people in our office have a Q50. So we study ourselves. And, you know, different people use the system differently. It is dependent on how people really drive. Each person’s driving style determines how the system is used.
T3: So talking about getting people used to autonomous systems in cars, do you feel a staggered approach is best?
Dr Sierhuis: Well, we hope that we can build interfaces and interaction that makes it intuitive. But, you can’t guarantee that will always happen. The iPhone could be used as a good example here. It takes us a very little time to figure out how to operate one of those phones. So there are ways we can make something that is very complex very intuitive. Now the one thing with the iPhone is that it is a pretty stupid device. If I swipe, it does what I think it should be doing. If the same system was doing all sorts of things behind the scenes while I swipe that I don’t understand just from the motion, now it becomes very hard to understand exactly what that means.
“No matter what technology people use, people create a conceptual model in their brain about what they think the system is doing.”
No matter what technology people use, people create a conceptual model in their brain about what they think the system is doing. That’s just human nature. An elevator, my toaster, the Nest Thermostat - each never does what I think it should be doing. That’s because my mental model of that technology is almost never correct. The more intelligent the system is, the more my mental model about what that system is doing differs from reality. It’s almost like, the more I’m trying to think about what you’re thinking about, the more I know I’m wrong about what my assumptions are.
So this is the problem with making systems more intelligent. Because, the problem is that not only will a human create a mental model of what the system should be doing, but the system also has to create a model of what the human is doing.
For example, with the Nissan IDS Concept car - which I think is an incredibly interesting and problematic and research-worthy topic - what interested us most was the question of how does the car communicate to the outside world? So everybody is focussed on how does the driver and the autonomous system interact. Now that is an obvious problem, but what about the pedestrians and the cyclists? How do the pedestrians know who goes first and who goes second? If I want to cross the street as a pedestrian, how do I know the car is going to stop?
Maarten's team heavily contributed to the Nissan IDS Concept car
Now humans communicate with each other in a variety of ways. We look at each other and wave each other on for example. But how does the car know that the human is going to go across the road? And, even if it does know, how does it then communicate to the human that it has seen the human and it is going to stop?
“How does the car know that the human is going to go across the road? And, even if it does know, how does it then communicate to the human that it has seen the human and it is going to stop?”
So one thing that we have at Nissan is what we call “communicating with intent.” What I mean by that is that when I want to communicate something to you I attempt to do so by communicating my intent by talking, by waving my hands and making eye contact. Now the car’s system needs to do the same thing, if it has seen the pedestrian it needs to communicate to that person that it has seen them. That way, the person knows that the car has seen them. With a human driver we look at each other’s eyes and that is how we confirm communications of intent. So in the IDS, we came up with a concept that the car will show light to help communicate intent. Of course though, that then leads to the problem of what happens if there are 10 people around the car?
So, with the IDS, we decided that the car is not just communicating intent to one individual but is broadcasting its intent to the area around the car. So if the car sees people and is going to stop, then it shows a green light. The pedestrians then infer green means I can go. So we, as the car, are communicating not what we think your intent is, but what our’s is. Naturally, when we’ve waited a long time and it is then our turn to go, the light slowly decreases and that again communicates the car’s intent that it is about to move off.
T3: So if communication needs to occur both to and from the car, which one is harder for you to implement in autonomous vehicles?
Dr Sierhuis: The first one, and I’ll tell you why. The sensor technology on the car needs to be really good to pick up the complex detail and minutiae of human gestures and movements. For example, if I’m a pedestrian and I make a small movement to leave the sidewalk and cross the road - literally my foot moves a foot or so over the curb - the car’s sensors need to be able to pick that up and infer the pedestrians intent to cross the road. The car has to recognise the all the various motions humans make. It has to understand what an arm swiping gesture means for example. And that is just for one person, what happens if there are 10 people again around the car? The car then needs the sensors and computational power to recognise and process all of those motions and gestures.
“I put down the hypothesis that there is a universal language that we use as humans to drive.”
T3: From your experience, how do people’s relationships with their cars change depending on which country they are in?
Dr Sierhuis: That’s a fascinating question and is one that I have asked myself many times. I put down the hypothesis that there is a universal language that we use as humans to drive. The reason I came up with the hypothesis is because if you are from the UK and go to California and land at San Francisco Airport you can rent a car and drive around. It might take you a little bit and if you come to four way stop sign you might be thrown out for a while, however after that you figure it out pretty fast. Now if you go to India, the driving again is different, but people soon get the hang of it and adapt.
There are ways that we communicate on the road and this is what we are trying to figure out - what is this common language? If we can find this common language and teach the car the basics, so the syntax and the grammar so to speak, then we can then use that car anywhere. Because, and our research back this up, how people drive and communicate differs massively depending on local. And that doesn’t just mean from country to country. During our research, for example, we discovered that people drive and communicate differently in the Stanford University area than half a kilometre down the road in University Avenue in Palo Alto. The driving behaviour was incredibly localised, so we need to figure out how we can reduce the core driving language down to the basics we all utilise.
T3: Talking of figuring it out, let’s talk about the future of autonomous vehicles. Where do you see the field heading in the near future?
Dr Sierhuis: Well, Nissan has a roadmap, and that is what I love about the company, we can sit down and talk it through. So we say in 2020 we will deliver to the masses autonomous city driving. And then we have different stages leading up to that point. So we start with highway driving in lane, then we start with making lane changes autonomously, and then we’ll go to city driving. And, hopefully, by 2020 we will have that capability.
T3: Is autonomous city driving the Holy Grail for you and your team?
Dr Sierhuis: It is the Holy Grail in terms of being the most difficult, sure. As soon as you enter that realm you need a lot more AI and far greater capability. On the highway we’ve had cruise control for a long time, for example. We have it and it was easy to implement as we know that everybody on a highway drives in the same direction. There are a lot of constraints that we have put in place on a highway that makes driving easier, because otherwise you’d find it difficult to drive at such speed.
I mean it is just an easier environment all round to drive in. There are no pedestrians, no cyclists and everyone is driving in one direction in a handful of lanes. Now I won’t say that by 2020 we’ll have a car that will be able to drive anywhere. I break down city driving into various subcategories, such business park driving, suburban driving and full on, downtown-style driving. That last category is the Holy Grail, the ability to drive something like 5th Avenue in Manhattan autonomously. There are gradations in autonomy. So by 2020 maybe we have a system that can handle some of the easier city driving environments.
“I have this autonomous concept where the car’s autonomous system would know when it can’t drive anymore, handing back control to the driver”
You see I have this autonomous concept where the car’s autonomous system would know when it can’t drive anymore, handing back control to the driver. That is where I think we will go first. There will be places where the system communicates to the driver “you can let go now as I can take over and drive this”, and other times the system will say “this is a little too hard for me so you need to take back over”. The trick with that of course, is to know far enough in advance to hand over smoothly.
Beyond 2020, we will try to push more and more to make it even more autonomous. As time goes on I change my views on this, however right now I still have a real big question mark over whether we’ll ever get to fully autonomous driving. When you ask people, “what do you mean by autonomous driving?” then what they mean is the ability to sit on the back seat, maybe lie down and sleep, while the car drives you say from Buckingham Palace to Stonehenge! What I would say is this, if we don’t allow any other humans on the road then maybe this is doable.
T3: A lot of people enjoy the act of driving, so do you actually think there is a desire for that level of autonomous systems?
Dr Sierhuis: This is how Nissan is viewing this. We still want to supply this beautiful driving experience. And the driving experience, for some people, is being able to drive yourself. On the Sunday afternoon, when the sun is shining, and I’m on Route 1 down the California coast it is pretty impressive. I can tell you there are some nice curves to drive in my GT-R.
That’s why I refer to this as “sliding autonomy”. On the edges of this continuum you have humans driving completely by themselves, to autonomous systems driving completely by themselves. Nissan want to cover that entire spectrum. It is not an either-or.
“I still have a real big question mark over whether we’ll ever get to fully autonomous driving”
T3: Lastly then, forgetting about autonomous vehicles, if you could drive one car through one area on Earth, what and where would that be?
Dr Maarten Sierhuis: So there are two places that I have been that I really enjoy driving. One is the Alps, where I can remember driving when I had just got my licence and my father and I were going skiing in Switzerland, with snow all around that was really fun. And speaking of Route 1, that is good but there is one road you can take to get to it through the mountains that takes about 20 minutes to run and boy that is fun in the GT-R.
As for the car, well the GT-R is fun, however my perfect car is still my Deux Chevaux that I had in Holland when I was 18 years old. Citroen made just 300 of them and mine was a yellow Deux Chevaux 2CV. Back home in Holland we called the car “the ugly duck”! James Bond drives one in For Your Eyes Only though and for those 300 made they included bullet hole stickers you could use. Performance wasn’t great mind, as I remember back when I was a teenager me and my friends would get the car driving up a hill and then get out and walk faster than it!
Maarten's first love, a yellow Citroën 2CV, as used in the James Bond film For Your Eyes Only
That car was so basic but you could take the back seats out and roll down the sardine can-style roof and it was great. It even had a start button just like my GT-R! That, to this day, is still my favourite car.
T3: Thank you for talking to us Maarten. Good luck with your future work.
Dr Sierhuis: Not at all. It has been my pleasure.