In western music there are 12 notes. That means you have a one in twelve chance of picking any note you hear and correctly naming it. Those are pretty good odds.
But learning by ear is really hard. Why? Is it hard to find the notes? Not really, you could try one note at a time and you wouldn’t have to try more than 12 before you found the right note. If you do that for each note you’re guaranteed to learn anything, so long as you can hear what each note sounds like.
Learning by ear is hard for the same reason that anything is. It requires patience and attention, and when we don’t give those things then we get worn down. I don’t believe we get worn down when we really do anything. It’s the moments when we aren’t really giving something our full attention, or when we aren’t really listening, that we get worn down.
If you really listen, and you’re really patient, then learning something by ear will actually excite you. It will give you energy instead of wearing you down.