It is definitely not every client that we get to work with for a long time. There are times when we have to let go due to reasons that are beyond our control as freelancers.
Some might be because of payment, some character, and other reasons.
So many more than I can remember. I've let go more than 20 clients over the years and it doesn't bother me at all. The most important thing is being able to get new clients from time to time.
There are many clients whom I just worked with only once and that's it. I'm happy we worked together but I will never start fussing about them not coming back again.
I've actually had some clients made inquiries and promise to order for my services but after a while they are no longer responding to my inquiry. I'm very comfortable and okay with them not ordering for my services.