I have been plagued by this problems for months. I frequently searched for a solution but found nothing until today.
I am signed into hangouts on my Android phone and use Gchat (the chat client inside the web version of Gmail). Throughout the day I use Gchat. At the end of the day, when I look at my phone, there is a hangouts notification for the first message of each separate Gchat communication. I have to manually clear the hangouts notifications on my phone, even though I have both seen and responded to the messages in Gchat.
I can’t claim credit, but buried on this Google Groups page, I found the answer. I have no idea why such an important solution is buried so deeply inside a groups post.
The solution in that post is hard to follow, what you want to do is “enable hangouts in gmail.” Google does provide a nice instruction set for enabling hangouts in gmail.
It works! When I get a new hangouts message, I get a notification on my phone and in Gmail. If I click on or respond to that message in Gmail, the notification is cleared from my phone.
The only downside, is that the new interface for hangouts in Gmail is significantly different than the old Gchat interface. It is too soon to say whether I think this is a good or bad thing.