Being a celebrity has its perks but, like most things, there are drawbacks that come with fame. Giving the world the opportunity to judge everything you do—and don’t do—is one of these downsides. So when there’s a celebrity who also happens to be a parent, the chances of opening a “mommy-shaming” can of worms are pretty high.

Hilary Duff, mom of precious three-month-old baby Luca, looked like she was enjoying a night out with her husband, Mike Comrie, last Friday night when she was photographed holding a lit cigarette (you can see the photo here).

The photo landed on the web and as a result, the former Disney actress has received backlash. “Does Hilary Duff Puff?” and “Hilary Duff Smoking While Breastfeeding Newborn” headlines started popping up throughout celebrity gossip sites and parent blogs. Some bloggers have dubbed the incident as “a classic new mom mistake”, while others are assuming Duff picked up the habit in an attempt to lose her post-baby weight.

Duff quickly addressed the photo via her Twitter account on Monday once the story broke:

“Yikes me holding a cigarette for my friend might have been a bad idea! pretty bummed about the photo.. Sorry guys, looks worst than it is!” (sic)

So, what’s the big deal?

Yes, smoking isn’t good for you, and it’s especially bad to do it in front of your little ones. And we can’t forget about the dangers of second and third hand smoke. But while it’s very easy to judge someone’s parenting skills when they are doing something we don’t agree with, maybe we need to cut Duff some slack. Her excuse might be the oldest one in the book but in Duff’s defense, the photo only shows her holding the cigarette—not smoking it—and little Luca is nowhere to be seen. Beyond that, all that’s left is our own (possibly incorrect) assumptions. We can’t help but wonder, are we too quick to judge other parents without knowing all the details?

What do you think about the photo? Do you judge other parents before considering the circumstances?