I'm sure your tort had not ill intentions. Keeping a tort is not like keeping a puppy. It's not realistic to expect to 'socialize' a tortoise. Some don't mind humans, some do. Some trust you over time, some don't. Reptiles in general are more observation pets and although some will tolerate head stroking and being picked up, they may never truly like it.
Turtle's and torts like to explore terrain whether it's grass, woodland or your arms & legs. When they seek rest, they will often find a comfortable 'form' whether it's under a log, in a tuft of grass or under your shadow.
When they need to pee or poop, they do it. Doesn't matter if they are eating, drinking or walking. If they are scared or made nervous, sometimes they do one or both.
Sometimes we want really bad for our animals to know us and interact with us, however it's best to let them do what they do best...be a tortoise. I'd suggest handfeeding because they get reinforcement for approaching you and paying attention. Crawling on you may make your animal nervous or insecure (or simply, he just had to go potty).
Bottom line...appreciate the animals beauty and give him/her the space, environment and time it needs. In time, your animal may reward you by craning it's neck to look at you when you are near, crawling to you and maybe even begging for food. I have many torts in my group that do this, but know also that some, even after several years, want nothing more than to get away from me except at feeding time. I don't take it personally, have even have gotten to the point where I think it's comical. It's kinda like having grumpy grandpa and nice grandpa. You love them the same.