Why do people squint when they can't HEAR something clearly or well?
Could be for all sorts of reasons. Depends on what is being said, any context to the conversation, the situation, the relationship between the people.
Could be they're a student sitting in the back of the class, can't hear the teacher very well so are squinting in order to read shape and motion of lips and/or body language more clearly to compensate, instead of shouting out "what!?" in the middle of class.
Could be that they're in a situation or listening to something that is triggering some level of fear response and it's an automatic response to seek out the specific danger.
Could be they're in a social situation and they've learned that squinting while unable to hear indirectly communicates to the person speaking they need to speak louder, without interrupting them which would be rude.
Or they've learned it's a means of conveying some level of not wanting to hear, or incredulity to, what is being said and they have simply shortcut don't want to hear it and can't hear it in some pavlovian fashion.
Could be they heard everything on a not entirely conscious level and their natural response is to squint to convey a feeling of danger, fear, intolerance, to whatever is being said, but consciously they're focusing on doing something else. The situation and script is making them squint more than the lack of hearing. e.g. you're filling your tank and a homeless drug addict person comes up and goes mumbling incoherently into their story about not wanting money and you need to figure out if they want to beg, steal, or kill you.
Could be they don't really use their head that much, they control the environment more than themselves, so their specific muscle control isn't that great. So in turning their head slightly to angle their ears towards what they want to listen to they also trigger the muscles around the eyes and squint a bit, they're just randomly flexing all of their muscles.
Could be the reason they can't hear that well are distracting movement and light surrounding the person and they're attempting to block it out.
Could be they realize they can't hear what is being said very well, don't want to interrupt, and don't really care about the relationship or conversation, so use indirect communication to show that they are listening and paying attention, focused, when they really aren't, in the hopes of offsetting any further questions that would prove they weren't really listening or couldn't hear.
but is there some underlying primitive instinct that causes people to sometimes squint when something is difficult to hear?
You mean like soft rustling in the grass or slithering in the tree causing people to squint to obtain better information in determining if there is danger?
Yes.
But the face is also the most expressive area of the human body.
And it's used as such for social reasons too.