The possibility of birth defects is higher in a related pair vs. a non related pair. This is because the related pair have more similar DNA than the unrelated pair. This increases the odds of them sharing a defective gene locus, which in turn can lead to birth defects. Continued inbreeding or line breeding even further reduces the amount of variability in a population, and thus the number of alleles possible for a gene locus.
However for genetic birth defects to occur the genes responsible must be present. So you should be careful even when out breeding as you can accidentally introduce a defective gene into an otherwise healthy population.
The best advise is to inbeed for 1-2 generations to obtain the desired trait. The out breed to increase heterogenetity in your population. This gives you your desired trait plus the hybrid vigor that a healthy population should have.
BTW - Hybrid vigor is a term related to increased heterogeneity in the alleles of a population and has nothing to do with hybrids. The term for hybrids would be genetic swamping, which is bad.