Yeah, there are some areas of the world where there are problems with human/snake encounters in villages and towns and even in people's homes.
In most of these places, you have to consider some pretty big differences in the environment. Places such as Sri Lanka, India and other parts of Southeast Asia have frequent encounters with Russell's Vipers and Cobras. That is largely due to the fact that most of these villages are rice or grain farming villages and because of the huge stores of grain and way they store the grain, they have huge populations of rodents. That attracts the snakes. And, yes, several thousand of them die every year. But compare that to the tens of thousands that are bitten. They are mostly bitten while walking BAREFOOT through snake infested areas, many times at night while these snakes are on the prowl looking for food. I've been to these places and seen it for myself. They do very little to prevent these encounters. I'll guarantee you that if you step barefoot on a Russell's Viper, you are going to get bitten. The way these people live is a matter of necessity and economics, unfortunately. It's not that way in the United States. On another note, in India the cobras are considered sacred. The people protect them and recognize that they help keep the rodent populations down. They realize that if they didn't have snakes around to control the rodent population the losses of food from being eaten or contaminated by rodents would be staggering.
Guam has a problem with Brown Tree Snakes because they were introduced into an area with ample food supplies and no natural predators. In the U. S. there are natural predators to most of our venomous snakes. That's one reason their populations don't get out of control, in addition to the fact that there is balance with regard to available food supplies because we don't have huge open stores of rodent food laying around all over the place. There's only enough prey animals around to support a certain number of snakes. The danger isn't in introduced exotiics exponentially multipying and taking over the country, but in replacing native species in their habitat by out competing them for the available food.
I don't think that you are silly at all for worrying about it, I just think that the worrying is for the wrong reasons. Yes, there is some small amount of danger to humans asociated with the scenario of introduced venomous animals, but it is very small. The biggest danger is in the damage to the ecology and native species.