I'm guessing you're referring to Angel's Trumpet, of the Datura family. Yes, all parts of those plants are poisonous. If you think your child or pets might eat them, or even chew on them, I would say plant them in an area where they won't be accessible. Otherwise, you probably shouldn't try to grow them.
I have always had poisonous plants in my house and yard, and none of my pets ... dogs or cats ... have ever bothered them. My son never bothered any of my plants, either. But some children do taste everything, and some dogs will chew on anything. Only you know the habits of your pets and your child, so you'll have to be the one to determine how great the danger of poisoning might be.