You should be able to fix this by playing with the mic speech floor. It happens when to much ambient stuff slurps in.

It's actually gaslighting you, you don't say that many ums and ahs ;)