Whiteflies are a very prolific pest once they get established. They thrive on the sap of your fruit and plant. The have a short incubation period, so new batches are hatching all the time. For this reason, you need to give repeated treatments to your plants over a period of time.
Insecticidal Soap spray should work, as it prevents more eggs from being laid. But there will be eggs already laid that will hatch, so you need to keep spraying until you no longer see any whiteflies. Persistence is the key!
Definitely do not just let it be, the plant and fruit may not be affected now, but they will soon see the effects and become skeletonized sad versions of their prior selves.
Hope this helps and you enjoy your tomatoes!