Is this a raised bed or in ground garden?
If it is a raised bed and you used some kind of barrier under it it may be a few years before the roots from the black walnut invade the bed.
I would look to relocating the bed in the future or if you have a lot of rocks issues build a raised bed farther away. Most tree roots will travel laterally as far out as the unpruned canopy and black walnut can travel twice as far.
Black walnut are infamously toxic to a number of plants but not all of them. There are a few of them that can coexist.
Many plants such as tomato, potato, blackberry, blueberry, azalea, mountain laurel, rhododendron, red pine and apple may be injured or killed within one to two months of growth within the root zone of these trees. The toxic zone from a mature tree occurs on average in a 50 to 60 foot radius from the trunk, but can be up to 80 feet.
You can get around this by planting the sensitive plants in large pots instead or well away from the walnut tree.