Get a washcloth (several) and a scrub brush and make sure you hit the areas you can’t reach with your bare hands. I like cotton. Don’t go so hard that your skin feels raw afterward, you can use the same pressure you normally use with your towel. the mechanical action of the cloth/loofah/sponge on your skin helps break stuff off in a way that soap+bare hands won’t get. That’s likely why your towel is getting stuff off of you even right after a shower.

Also it takes some trial and error to find soaps (and shampoo, and conditioner, and skin lotions etc.) that work and don’t make your skin feel bad. Try something different from what you usually get, if you can afford it you might benefit from doing some research and getting fancier cleansers than just plain bar soap or body wash. A lot of them are snake oil, especially the ones that are super expensive or aggressively advertised, but different people respond better to different compositions of cleansers, and it pays to figure out what works for your skin.