Aphids are usually pretty easy to control, being slow and stupid. They will just sit there and let you squish them. I don't like to do it bare handed, so just use a kleenex or paper towel. Go all over the undersides of leaves and the stems. The squished bodies of aphids also act as a deterrent to other aphids (maybe they aren't totally stupid!
Other than that, Neem oil is recommended, or a mineral oil and water mix with a little soap. This forum would have recommended soapy water spray. Your Joy might be detergent not soap, which would be why it damages the leaves. Look for something like Dr. Brunner's pure castile soap. With the soapy water spray again you have to be sure to coat the underside of the leaves. Or use a hot pepper spray of garlic and peppers blended in water. I and others have posted several versions of this recipe. Do a search for it.
Here's an article about aphid control:
Predators: Green lacewings, ladybugs and their respective larvae have a voracious appetite for aphids. Larvae from the syrphid fly also consume aphids. Hover flies and praying mantis feed on aphids. [RG note: you can buy many of these, especially ladybugs]
Repellent plants: Anise, chives, coriander (cilantro), garlic, onions, petunias and radish. Nasturtiums act as a trap crop. Aphids definitely prefer yellow flowers.
*Squashing a few aphids around the infested plants releases a chemical signal that makes the other aphids drop from the plants and leave.
*To foil aphids: flatten a square of aluminum foil around the base of plants to bounce light on the undersides of leaves. This also helps the plants in giving them more light.
*Try a barrier of powdered charcoal, calcium dust or bonemeal to keep them away from your plants. [RG note this also keeps the ants away]
*Stinging Nettle Spray: Aphids & Thrips - Cover 1 quart nettles with water, cover and ferment for 3 weeks. Mix 1 part nettle tea with 7 parts water. Spray.
*Spread out a barrier of tansy around the base of the plant to stop those ants. [RG note: or mint or anything else strong smelling]
*Use a spray made from a tea of tomato or potato leaves and water.
Chop 12 or so tomato leaves and 1 chopped onion in 1/2 cup of of 70% isopropyl alcohol for a few minutes. Apply the mixture directly on aphids with a cue tip or paintbrush. [RG note or put the tomato leaves and onion in the hot pepper spray]
*A forceful spray of water is often enough to knock the aphids off the plant and may discourage the ants, well sometimes.
*Put a bright yellow plastic pan in a strategic spot in the garden. Fill it a third of the way full with water. Aphids are drawn to the yellow color, land on the water, sink and drown.
*A soap spray can be used to strip them of their protective wax coating, dehydrating them. Mix 1 tablespoon of Castile soap to 1 gallon of water, spray.
*Garlic oil spray can kill aphids and other soft bodied pests.
A dusting of diatomaceous earth is lethal to aphids. Wear a mask when using DE.
*Place banana peels at the base of infested plant. The peels give them a shot of potassium too!