trimming the nails by hand is fine, providing you take care not to clip into the 'quick' of the nail. This is where all the bloodvessels etc of the nail is. As BTS nails are pretty clear, shouldn't be difficult to avoid the quick. If your blue tongue happens to have dark nails, just clip the very tip of the nail. (not sure if other species of BT's have dark nails, I do know my Northerns have clear nails).
Sharp cat or human nail clippers work best.
Rough surfaces in their cage certainly will help them naturally wear down their nails. A large flat rock, packed earth to burrow in, wood surfaces, fine sandpaper (200 grit or finer (the higher the number, the finer the sandpaper)) stapled or glued around wood, or other objects, ceramic tiles etc, will all help wear down your skink's nails. Just make sure there are softer surfaces around so your skink isn't forced on rough surfaces all the time.
-----
PHLdyPayne