In 2021 in the United States, 23 children died from heat stroke after being left unattended in vehicles according to the child safety organization Kids and Cars. Depending on circumstances, studies have shown that the inside of a vehicle can reach 125 degrees Fahrenheit in minutes, even with outside temperatures as low as 60 degrees.
In light of this, Toyota Connected, a software and innovation centre has recently introduced its ‘Cabin Awareness’ concept technology. In practice the technology uses millimetre-wave high resolution 4D imaging radar to detect vehicle occupants, including pets, who have been left behind in the vehicle and may be subject to danger.
Go deeper with GlobalData
The concept technology has the unique ability to detect micro movements such as motion, respiration and heartbeat of the occupants. The platform is also able to classify vehicle occupants according to size, position, and posture.
We spoke to ‘Cabin Awareness’ managing engineer Simon Roberts, to find out more about the concept.
Just Auto (JA): Could you tell me a little about your job role and what this involves?
Simon Roberts (SR): I am the managing engineer of the cabin awareness team. This team was formed out of a hackathon at Toyota a couple years back and it kind of grew organically. The title of the challenge was: Safety security and social good.
My team – we won the hackathon and the outcome of that was Toyota saying: let’s see if there’s anything to this idea and let’s see if we can expand on this idea a little bit more over 36 hours. There’s only so much you can accomplish in that time.
How well do you really know your competitors?
Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.
Thank you!
Your download email will arrive shortly
Not ready to buy yet? Download a free sample
We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form
By GlobalDataSo, we got a little bit of flexibility from the leadership team who said spend an extra few hours here and there, and a couple of us started working on that. Then over the coming months it really started to solidify that this idea had potential.
Then fast-forward to today where we’ve really started to solidify the concept and the concept is what we’ve demonstrated on our promotional video. So now I lead a team of around half a dozen engineers, designers, and electrical engineers on the Toyota side as well.
Every day is different, and that’s one of the things I love about Toyota Connected in general. We’re encouraged to find solutions to problems and that leads you down all kinds of paths; it’s a fun role.
Could you explain the technology concept and how it works?
It doesn’t have to be exceptionally hot for this to be an issue. I am in Texas where it gets really hot – 100 plus degrees Fahrenheit on some days. But it doesn’t have to be that sort of temperature, it could be as low as 60 degrees, so someone is just in the sun for a few hours.
I’m sure you’ve got into a car on a sunny day and thought ‘oh my gosh, that is so hot’. For children, they don’t regulate their body temperature as well as adults. It’s the same for pets; dogs pant as they can’t sweat. So, this can be a real problem when they are in a hot car.
We evaluated all kinds of technologies. There are other systems on the market today such as weight sensors that are in the seats. I’m sure you’ve had the experience of throwing a backpack or something in the backseat and then getting the ding ding ding as they can’t tell the difference between your backpack and a child.
We have door logic today, rear door logic, which is kind of a good proxy, a check-up, for something that because you open that door someone might be back there, but it doesn’t cover everything. We wanted to expand on that.
We looked at cameras as well. From a Toyota perspective, we looked at that, but first and foremost there’s a privacy concern. Personally, I don’t really want a camera pointed at me and my family in the car, and I know the Toyota is not going to do anything with that, but it’s also a psychological thing.
Plus, there’s the compute cost, it’s intensive, you can’t necessarily put a camera in the rear-view mirror, and someone might have a rear facing infant seat meaning the camera can’t see the baby. Same problem if you cover the baby up with a cloth because it’s sunny outside, and they’re sleeping – it’s just problem after problem.
Then there’s ultrasonic technology. There are vehicles with ultrasonic in them that can’t differentiate between people and objects and that basically led us towards this millimetre wave radar. I was challenged by this particular executive to find as many use cases as possible for this sensor because this is a big deal for the industry, and you want to basically try and find as much as much utility as possible. This millimetre wave radar really allows us to do that.
Real world testing is currently underway, could you discuss what that involves?
We have partnered with a robotaxi or autonomous mobility as a service (MaaS) company called May Mobility and we have several vehicles of theirs. The Sienna minivans are equipped with the Cabin Awareness concept system. The technology gives the autonomous vehicle operator an interface on an iPad app that we created.
We have our sensor in the vehicle, and it’s essentially connected to an iPad app. For every passenger seat in the vehicle in the second and third row for those vehicles we have for every seat an option that says what is the status? Is it unoccupied or occupied by a child or an adult? Is there anything in the footwell or anyone in the footwell in front of you? Do they have the seatbelt on?
There are visuals on there for them to answer all of those questions, and we gave them an interface for the driver for them to say: is this correct? Yes, or no? Then we log that data for every single trip that they do, and we get that feedback.
So, if they do 1,000 drives in a month and 900 of them are good then we’ve got 90% accuracy. But if they’re not good, let’s say behind the driver’s seat every time you put an infant in that seat, it’s always coming up as an adult, they would say no, that’s wrong. Then they would put in there what that is, that it’s actually an adult and that’s the nature of the information collected.
The technology doesn’t collect any personal information or anything like that. It’s also super-fast.
We record the radar data and that gets uploaded to our system at the end of the day. Then our engineers, in partnership with the VR engineers, can analyse those results and say, okay we’re seeing a trend on that seat, we’re always classifying a child as an adult. With that data we can train the neural network to fix that.
What ideas are there surrounding how the system would issue an alert that someone was in trouble in the vehicle?
We have the Toyota app and of course an app with the Lexus brand as well. The vision with the app is in the hot car scenario. Let’s say I’m driving, and I have my child in the back. The car is going to know that; it’s going to know that there’s just me and my child in the car, and where we are in the car.
When you turn the ignition off you get that little message pop up on the dashboard screen that says check the rear seat. The idea is, let’s change that message, let’s be contextual about that message. If we know that nobody ever got in the backseat, and you didn’t even throw your backpack in there it would be great to not have that message. So we can be more contextual on that, maybe pop something on the infotainment screen as well saying, don’t forget your child is back there.
Let’s say you get out of the vehicle; we want to allow some period of time for you to do things such as filling up your car with fuel. If after that period of time you still haven’t got the child out and you’ve maybe locked the door and it is clear that you’re no longer at the vehicle, we want to flash the lights, beep the horn and try and get your attention.
These are just ideas, because in those periods of time there’s a lot of variables at work there. What is the temperature of vehicle? What do we need to do? Do we need to alert them within five minutes, ten minutes? We’re trying to figure those things out.
After that, if you’ve got the Toyota app or the Lexus App, if you’ve configured notifications, you could potentially get a push notification saying there’s a child still in the car. If you’re in the middle of a busy meeting or somewhere where you can’t check your phone, we could have some escalation path as well. Let’s say you’ve added the emergency contacts to your profile, they can get a notification that something isn’t right.
If you’ve got smart home, smart home lights, Google Home, some smart home devices, we could also interact with those if you’ve linked them and we could throw messages on that and wake your entire smart house up, take over your television, that kind of thing. Potentially we can really grab your attention if you still haven’t done it after a period of time and things are potentially getting critical.
We have the Toyota Safety Connect system on all vehicles; they are equipped with connected vehicle technology, so we could leverage that and that’s a call centre, at least in the US and North America. There are call centres that if you get an accident today it’ll trigger the SOS system and it will connect you with a Toyota agent and they can dispatch emergency services for you. So, the thought is, well, we could potentially connect to them as well. This is the escalation path that we are envisioning.
Do you want to share anything else with our readers?
I’ve been on this for two and a half years; it’s a daily driver for me, this is this is near and dear. You read the stories and it’s heart-breaking.
I was just talking with my doctor’s office yesterday and I was chatting with the technician. He’s just had a baby and he said, in response to leaving a child in a car: ‘I’d never do that’. That’s a common attitude. I don’t fault him because that’s a super common response: if you’re a responsible adult, why would you leave your baby?
It’s more complex than that, though. Data from the experts in that industry say that it happens, for example, due to a change in routine; you’ve just had a baby and normally mum takes the baby to day-care, but this morning she’s got something else going on. Suddenly dad has to do it and it’s outside of his routine. Maybe the baby was up every two hours in the night, he didn’t get much sleep; it’s all of these compounding things. That’s a real story that happened.
Toyota is trying to do its part and hopefully lead and guide the industry in that. I want to share that 100%. I think it’s also critically important that everyone partner together in this as an industry, sharing that story about how this can happen. Even if it might be like tipping it down with rain, it can still happen.
I think sharing that with the readers is not to shame anyone, it’s more about raising awareness.
