I agree.
I know a man who had a foal that hung himself on some piping by having a halter on unsupervised.
A horse can get into quite a pickle with one of those on!!
If you insist on keeping one on, I suggest a leather halter with a break away strap.