The man who went toe-to-toe with Adrian Peterson at a poker game last month tells TMZ Sports it was actually the NFL legend’s mouth that caused the fists to fly.

Joe “Baby Joe” Castaneda said on Tuesday his May 27 brawl with A.P. at a social club in Houston, Texas all went down because the future Hall of Famer hurled an insult at one of his buddies.

Castaneda — a professional poker player — couldn’t recall exactly what quip Peterson fired off … but he said after he stood up for his pal, “that’s when the melee kind of broke out.”

The footage cut out before the two were formally separated — but Castaneda tells us once they were pulled apart, he had some damage on his face.

“My nose was leaking really bad,” he said. “It was definitely a lot of blood. It was kind of gory.”

Somehow, he says he and Peterson, 40, were able to bury the hatchet after the dust settled … and he has zero plans to go to cops or to file a lawsuit.

The 47-year-old also told us he and the ex-Vikings running back have been playing together for years … and he’s looking forward to getting back on the felt with him some time soon.