As Sylvester Stallone made his breakout performance in Rocky, there was one person he couldn’t quite turn into a fan.

While discussing his memoir The Steps, releasing Oct. 23, the 3x Oscar nominee recalled things on set getting “really nasty” after giving his real father Frank Stallone Sr. a cameo as the bell ringer in the 1976 boxing drama’s climactic match.

“It was like: I just want you to love me for a little bit,” Sly explained to The New York Times. “Like, look what I did! Share it with me. I know you thought I was a fool my whole life, but here we are.”

On the day of filming, Frank Sr. “didn’t ring the bell properly,” as his son “kept fighting and fighting and fighting. Everything’s choreographed. It’s one, two, three, four, bell. Not six, seven, eight, nine, ten. Carl Weathers is like, ‘What’s going on?'”

Stallone continued, “And of course it ended up in an altercation by the end. It got really nasty. A really bad Italian drama. But for a moment I thought, Jesus, he’d be proud.”

In the interview, Stallone explained that his father “was very, very cold and tough” until they reconciled upon his death in 2011.

“It would be these constant bombastic explosions and then total silence,” recalled Stallone of living alone with his dad. “If he and I were talking and I said, ‘I don’t know if I agree with that,’ he’d start spinning his fork and suck his teeth, and you knew what was coming. The next thing you know, wham! He had very big hands. You learn to duck. I didn’t put a lot of stuff in the book because I didn’t want a pity party, but it was insane.”

In the biopic I Play Rocky, premiering Nov. 6 in theaters, Anthony Ippolito plays a young Stallone as he writes the first Rocky movie and fights to star as the titular boxer.