Ben Napier Almost Got Into a Fight with a ‘Home’ Town Fan in NYC After a Misunderstanding – ryan

Ben Napier almost threw hands with a fan.

On the most recent episode of The Heirloom Podcast with Josh Nowell and Jim Rasberry, the Home Town woodworker discussed his affinity for storytelling and his experience with fame — both of which led to a surprising tale about a run-in he had on the streets of New York City.

Ben, who co-stars on his HGTV show alongside his wife, designer Erin Napier, recalled a moment from the early days of Home Town, before the show became super popular. He was visiting Manhattan, a big change from his typical small town surroundings in Laurel, Miss., when he encountered a man who he thought was looking for a fight.

“We’re walking down Fifth Avenue. We’re in a sea of people and we are (crossing the street toward) another sea. You know, the hand (on the crossing signal) comes up and then it turns to the guy walking and we start to cross,” Ben recounts.

“We started walking across, and a guy — who’s covered in tattoos on his arms, wearing dark-rimmed glasses and a Brooklyn hat — pops me on the shoulder, like stiff-arm.”

Surprised by the harsh physical contact, Ben says his shock quickly turned into a defensive instinct.

“So I’m like ‘here we go,’ and I turn, in this fight-or-flight mode,” he says. But when the tough-seeming stranger spoke, Ben was grateful he hadn’t taken action.

“He says, in his accent, he says ‘Home Townlove your show man,’” Ben recalls.

While these days, he’s more used to fans approaching him and Erin in public, at the time, the show was newly released and less widely known, leaving Ben in somewhat of a state of shock at the comment. “I was like, ‘Thank you! Wow! This is really cool!’” Ben says.

No longer on the offensive, he naturally struck up a conversation with the man before they parted on much friendlier terms.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

The most recent season of their Home Town spinoff, Home Town Takeover, aired its finale on Sunday. The show saw Erin and Ben in Sebring, Florida, where they’ve been surprising deserving individuals with home makeovers and helping business owners revitalize their storefronts.

“You could tell there are people who really love it and they care about it, and parts of it are doing well,” Erin told PEOPLE in March. “But the downtown, specifically, was fifty or sixty percent shuttered.”

The series previously helped revitalize the small towns of Wetumpka, Ala. and Fort Morgan, Colo.