Oh, I couldn't disagree more. I find audiobooks less passive than almost any other medium out there, but then again, I'm listening to every word, every sentence, every paragraph, critiquing, evaluating, considering, punctuating ...
Unlike the written word, you don't get the opportunity to reread a line (unless you rewind) so it teaches you to take it in quickly and analyse it quickly.
I'm not talking about listening to them for entertainment, I'm talking about listening to them educationally.
I read a huge amount of nonfiction, and any time I have the TV on, it is usually science/history/nature/food related. I process information differently and at a rate uncomfortable for a lot of people. Audio book pacings do not mesh well with my sensory input process, so while they are a huge boon for a majority of people they are a huge distraction and sensory dissonance for me personally. I'm not knocking the format, I'm just saying that they do not appeal to everyone for various reasons.
When I pick up a book, I commit my attention to it and filter out auditory input because visual and tactile sensory inputs are fully engaged. I have a literal mute button filter that blocks out all verbal input, coping mechanism in the wierding of my brain. (It is a visceral shift when I bring focus back to a different gear and tune into verbal cues again.)
To understand something correctly. I need the written context and punctuation as a visual cue. I need the construct to organize the ideas and concepts in a linear format. Much akin to NT brains processing verbal infliction, rhythm, cadence, and facial expressions...all things I struggle with.