You are confusing a genetically diverse population (or species) with a stabilized variety, which is just about the same thing as a clone of an individual. With a stabilized variety you will see neither adaptation nor inbreeding depression (decline due to inbreeding), all of that potential variability is weeded out during the selection process.
Breeds of animals are different from horticultural varieties of vegetables because it is nearly impossible to "stabilize" an animal breed because of the high number of complex gene interactions, low number of offspring for each generation and because they do not "self pollinate". Most animal breeds are full of hidden deleterious recessive genes, which constantly have to be weeded out through continued selective breeding. Plant varieties can get bred past that point rather quickly (5-10 generations). Some lab mice and fruit flies used in research are "purified" to the level of vegetable varieties and are pretty much clones of each other, but your basic German Shepard or Quarterhorse is comparatively very different from its brother or sister.
Here is a quick single gene example. I'll use a gene for leaf shape and call it 'El' (L or l) for leaf, but it could be one of dozens of flavor components, disease resistance etc.
Each gene has two replicates, one from the mother (Lm), one from the father (Lp) p for paternal or pollen.
A stabilized regular leaf variety will be homozygous dominant (L,L)
If it self pollinates (including pollination from any other individual of its vareity) the only option for that gene is regular leaf (Lm or Lp)= (L,L). Thus there can be no change (except for mutations, which are literally, one in a million seeds per gene).
Any potato leaf variety will be homozygous recessive (l,l), and similarly cannot change.
Practically all of the genes are this way (homozygous) for a stabilized variety (A,A, B,B, c,c, D,D, e,e, f,f, G,G....). Both parents can only contribute ABcDeFG to the next generation so the next generation cannot change.
What if you did want to chang the variety some how?
A hybrid of a potatoleaf and regular leaf will be (L,l) and since L is dominant the plant will look regular leaf. This is genetically unstable because if/when it self pollinates the possiblities are (L,L), (L,l) and (l,l)...This is genetic diversity. If bugs or fungal disease prefered regular leaf (L,L, or L,l) and killed most of them off then the population would shift to potatoleaf (l,l), and adapt to the local selective pressure, but in doing so a random mix of other gene combinations would end up in the survivors so flavor and production would be off. That is why it is so hard to get a good tasting supermarket tomato. They are bred for a handfull of disease resistance and storage characteristics and it is very difficult to get good flavor gene combinations to follow along.
It is quite funny to see the survivalist seed people pushing heirlooms, because it is the hybrids which are full of disease resistance gene components and are also heterozygous (mixed) for other genes, which could hold the genetic diversity needed for local adaptation. The heirloms will either sink or swim with local conditions year to year while the offspring of the hybrids will show a range of success from which survivors/high producers can be chosen at each location/year. They may taste worse than the parents but poor flavor is better than starvation
One reason that some people say to collect heirloom seed from multiple fruit and multiple plants is so that when there is crossed seed there will also be a large amount of true-to-type plants growing from those seeds for comparison. If you only save seed from only one plant and that one happened to be from a crossed seed then you have lost that variety. Likewise, each fruit has its own level of cross pollination based on what pollen was covering the bees that hit it.