Of course these are legitimate concerns as you want your music to sound alive, not contrived.
And at this point you may decide that you need to construct the drum parts yourself, hit-by-hit, slice-by-slice, using single-hit drum samples. Admittedly, you’ll probably get a better result this way as you at least have control over specific cymbal crashes, adding bass drum notes, fills, etc.
There is a big problem to this approach however: It takes up huge amounts of your precious time.
If you have been through this before, I don’t need to tell you: putting together “custom” drum tracks using single-hit drum samples will take up so much of your time that you are likely to compromise your artistic needs in the end anyway. You will put in so many hours, and the tracks will sound better than just using factory loops, but you will probably give up realizing that they will never really, truly sound and feel like custom drum tracks that were played by a real person for your song.