Does a 401 INVALID_TOKEN answer count against my address?
Yes. Each one counts as a failed authentication of the client address. After
100 of them in 15 minutes, every bearer request from that address answers
429 until the window ends, valid keys included.
More: Limits, Rates