I have squirrels too, up to 9 at a time in the garden, but they've never bothered my veg garden (so far, knock on wood, etc.) They do steal my pears and apples though... Anyway, I'm going to assume that squirrel and crow predation is a given in your garden.
Now, first obvious considerations is that you're going to have to group your plantings by mature height. Tomatoes, for example, are going to get pretty tall unless you're planning to grow shorter determinate varieties. I guess another option is to meticulously train the growing stems -- the cage will probably provide a handy support/ trellis UNLESS you're planning to lift the entire cage off.
In my garden, I find 2'H fences sufficient to keep out rabbits -- my main concern. Cats can easily jump over them and GroundHOGS climb over them, but that's another story. 2'H fences are low enough to reach down and plant, work, or harvest, as well as step over as necessary.
My idea would be to build a semi-permanent 2'H surround, then build 2'H extension units to fit over that, possibly secured in place with easily removable fastenings such as latch plate-over-eye-bolt. If removability is not a concern during the growing season, I guess you can just fasten them together with tie-wraps or something like that. I would build rectangular frames of wood, cross braced as necessary depending on length. Hinge 2 frames together into L-shape, and use hook-and-eyes (or tie-wraps) to attach the 2 L shapes into rectangles. This way, they can be folded away.
My idea for the top would be 2'W frames that are hinged together to fold up out of the way, or accordion fold if they are wider than 6'.
If you decide to use the PVC pipes, you could probably use tie-wraps for similar hinging effect, but I'm not sure what is available for easily removable but secure fastening. Oh! I know! The loop-and-hook (e.g. velclo) strips used by electricians to bundle cables together! BTW I mention tie-wraps because I find them to provide a good secure hold for the season (they do weather and become brittle after a full year) but they tend to be rather VISIBLE and so not as elegant as something more discreet.