where is this dirt? in the plants? Chams do not need any substrate, so if you are using any I would suggest removing it. If it is in the plant pots you can cover it with a piece of plastic canvas (this way air and water can get through but not your cham's tongue) or large stones.
You should always repot any plants with fresh soil that does not contain perlite (the white pieces) or synthetic fertilizers, etc. Plants should also be washed as they are probably sprayed with pesticides at some point. If the soil he got was clean it should not pose a problem this one time. Frequent snacks of soil are not good as he could become impacted - they do occasionally miss 
-----
Chameleon Help & Resource Info
0.1 Veiled Chameleon - Luna
1.0 Beardie - Darwin
0.2 felines - Kyndra and Lita
0.1 African Clawed Frog - Skipper
0.3 Mad. Hissers (2 died ;(
0.1 Chilean Rose Hair Tarantula - Rosa Leigh
0.1 Goliath Bird-Eater Tarantula - Natasha