When we lived in NC Passion Flower grew in a neglected pasture: full sun, well-drained, pH maybe 5.5 or less. In ten years our growing season was only long enough once to ripen the fruits - and they are even better than the flowers.
They were also native or naturalized in the mountains in Hawaii. And again - open fields, humid, alternating wet and dry, well drained, benign neglect, long season.
I don't know if they freeze back to the root or are annuals.
I hope they work for you!