Sunflower, great going on your garden plot!
There are ways to compost given your situation of limited space apart, and garden some distance, but don't beat yourself up over it. Don't let it be the tail wagging the dog.
A suggestion if it works in your space. Any small area, dig hole or trench and add your peals, stems and cores, coffee grounds, cover with dirt. Next trip there, another hole, then another. As a crop finishes and you aren't replanting same space, use that area, on going around the garden plot. If you have a few worms, this will make them happy and they will make more worms.