It's in this chapter. The elders couldn't have gone across the desert and the mountains, but a sea voyage would have been possible (with the usual tsunami risk). That Damien doesn't like ships much doesn' t have to mean that no-one else could have come.

So who knows, maybe those letters really were a bit irritating?