People in the car can adjust how much of your attention they are taking based on their observations of road conditions. Someone on the other end of the phone cannot.
This is why it’s dangerous to talk on the phone in the car; it’s the lack of attention, not the holding of the phone to your head. And people who are in the car with you are able to stop bothering you when it matters, or even call your attention to the road when needed, unlike someone on the phone who could be at the best part of a story or tell a great joke right when you need to make a critical decision.
Makes sense to me. ::