The '127 Hours' actor, who features on the star's rumoured list of 36 sexual conquests, which also includes 'Fifty Shades of Grey' star Jamie Dornan and Justin Timberlake, insists he has never slept with the actress but has written about her and other celebrities in a new poetry book.
In an interview with Los Angeles magazine, James said: "No, I didn't write anything bad about them. And Lindsay herself has told lies about me with her people-she's-slept-with list! So I feel like what I said is much less than what she's said."
The 27-year-old star reportedly left a list of all the men she has slept with behind in a LA hotel bar after writing it to show off to a group of friends.
Although five of the names have still not been revealed, the list includes actor Colin Farrell Maroon 5 frontman Adam Levine, Zac Effron, Ashton Kutcher and the late Heath Ledger, as well as British stars and entrepreneurs.
James, 35, mentions Lindsay in his book 'Directing Herbert White', which took him four years to write, and claims poetry is a "big part" of his life.
He explained: "I know that my life as an actor obviously casts a shadow over what I'm doing with the poetry, but it allows me to get readers that maybe normally wouldn't read a book of poetry.
"It also attracts skepticism and contempt prior to investigation, but I can't think about that. It's something I love, and if people want to judge my poetry through the lens of my acting then there's nothing I can do about that."