My "favorite" place to go for garden supplies is a local hardware store with an excellent garden center -- premium bagged potting soil and soil amendments as well as good selection of organic supplies. This one is ACE (all the small hardware stores around here are affiliated/franchised with ACE or True Value). A particular benefit is that I can look for things on the ACE's website and then go to the store andif they don't have it, I just ask them to order it. They will also consider ordering/getting other things if I ask.
I also go to a grower's co-op where they have excellent started plants and premium bagged soil mixes. Another couple of large and small local garden centers, a largish roadside farm stand where they have great and unusual started plants in spring, and a small herb farm for herb starts.
I do a lot of catalog perusing, but I try to buy locally unless they are unique items or not available. I love Lehman's catalog that Eric mentioned. Others that I constantly dog-ear pages is Lee Valley Tools/Garret Wade and A. M. Leonard. Gardener's Supply is like a yuppy, specialty gardening catalog. I like looking through that one a lot -- it's amazing how "dressed up" some basic tools can become. I occasionally spend the money on well-designed/convenience items.
For seeds, I tend to go to some favorite on-line/catalogs, though I occasionally find specific varieties I'm looking for on the local seed racks. Johnny's Selected Seeds is one of them and they have some serious tools in the back of their catalog. I think some of the specialty tools must be professional grade because they tend to be expensive and I have to think twice before buying anything, but what I have bought are high quality.
I generally go to the big box stores for things like inexpensive garden/wire fencing, rebar, bagged gravel/sand, and common DIY supplies and cheaper hardscape materials, hoses and sprinklers, etc. and farm supply or feed store for hay, straw, alfalfa, and more sturdy animal fencing.