Welcome to The Helpful Gardener! Hummingbirds are attracted to the color red, so red flowers or flowers with red or a red 'eye' will attract them. I know of one woman who tied red ribbons all over her garden to get the hummingbirds to come to her garden. The are also attracted to purple flowers as well. They have long bills so they prefer flowers shaped like trumpets. I can think of many plants, but your hardiness zone will be limiting. Here's some that are hardy to zone 3. You don't say what the sun conditions are so I'm giving you a mix of plants that like sun (6 hours or more), part sun (4 to 6 hours), part shade (2 to 4 hours) and shade (2 hours or less of sun).
Columbine has several cultivars, but the most common one seen in the wild is the native Aquilegia canadensis and it's a red and yellow flower, sometimes called granny's bonnet. It's a short lived perennial but seeds around the garden and isn't invasive. I love mine when it appears in places I'd never thought to plant it. It's easy to pull out where you don't want it and the leaves are distinctive when the little plants emerge. They remind me of curly parsley. Let the seed heads dry on the plant and harvest the pods as soon as they open. Scatter the seeds where you want them so you'll always have some. Don't purchase the double hybrids as the hummers can't get to the nectar and they don't produce as much. These will grow in sun, part sun and part shade.
Lobelia cardinalis is a native they love. It does well in part sun to shade.
Heuchera - coral bells - native clump forming for part sun to part shade. Most are rated to zone 4, but some sites I've looked at say zone 3. Many with pretty leaves. Try and select ones with pink or red flowers. Actually after looking at the site below, try and select one! If you see ones you like at this site and want to see the color of the flowers, go to google and put in the botanical name and click on 'Images'. You might get some pics with it in flower. Example of how to put in the botanical name:
Heuchera 'Lime Rickey'
You will also see Heucherella at the Terranova site. It's a cross between two natives - Heuchera (coral bells) and Tiarella (foam flower). It's sterile and dosen't produce seeds and doesn't do well in less then pampered conditions. So don't even bother with them.
There are some native Tiarella - foam flower that bloom in pink. Some have pretty colored leaves. They like part shade to shade.
Tiarella 'Pink Brushes'
Tiarella 'Pink Pearls'
Lilium canadense - Canada Lily is a native woodland bulb. These get tall and the bulbs can get expensive if you will be moving and leaving them behind. You can buy seed but it can take 5 years before they will bloom from seed. They come in shades of oranges and some sites list them as only hardy to zone 4.
Impatiens capensis - Jewelweed is an orange flower and can seed around the garden.
There are several cultivars of the taller Penstemon - beardtongue, but not all are hardy in your zone. Here's more info on Penstemons. Btw, Penstemon 'Husker Red' has red stems and white flowers. Not what you'd expect.
Phlox paniculatta - tall garden phlox will also attract them and is fragrant. You'll need to plant reds, pinks and/or purples. Here's a few examples. These also tend to get powdery mildew, so choose carefully. This first one Phlox 'David' is white, is the most mildew resistant and is listed as a hummer attractor. I grow this in my garden and have never seen hummers go to it. There is a lavender sport of 'David' too. It's the second link.
Monarda - bee balm is another native that will take full sun to part shade. There are several varieties of these natives and they come in different colors. It's a member of the mint family (they have square stems) and like mints, they spread, so give them plenty of room when you plant them. Not all are hardy in your zone. This is the only one I can find that will grow in zone 3.
Monarda fistulosa - wild bergamot
Salvia will be good for them and there are annuals and perennials, but I can only find the perennials that are hardy to zone 4. If you find any that are hardy in your area give them a try. They usually come in reds and purples.
Lonicera sempervirens is a native honeysuckle that won't take over the world and smother trees in the environment like the Japanese honeysuckle does. I have two and the hummers love my Lonicera sempervirens 'Blanche Sandman' better then the yellow Lonicera sempervirens 'John Clayton'. Native honeysuckle is great for hummingbirds but I'm not finding any hardy farther north then zone 4. Most repeat bloom all season.
Lonicera sempervirens 'Blanche Sandman'
If you decide to mail order plants or if you just want to search for nurseries near you (you can search by country), keep this site for references of nurseries with good a reputation. You can even search by plant material.
Here's my favorite hummer feeder. I prefer this one because it's the easiest to clean and refill. I find the 8oz feeder is large enough in my climate. In the hottest days of summer I have to clean and refill it every 2 to 3 days. My yard is very small and I don't' get that many hummers. I also grow plants that feed them so they don't rely as much on the feeders. I usually make a large batch and freeze it in small baggies that I label so I don't have to cook up a bunch every couple of days.
Save this site so you can track the progress of the spring migration. In my area the hummers usually arrive around April 10th. I put my feeders out on April 1st. These maps are for Spring 2007 and are for the eastern US and Canada. For the western US there are links at the bottom.
Lots of helpful info here about hummers and migration. The last one is migration by state, with Canada at the bottom.