The '127 Hours' actor, who was recently named on the 'Mean Girls' actress' leaked list of celebrity sexual conquests, claimed that she once broke into his hotel room at the Chateau Marmont in Los Angeles, but insisted they've never slept together.
He said: "I don't know what to say other than we were at this hotel during a very dark period of her history and I think she liked, it seemed pretty damn clear that she liked me.
"She even broke into my room one time, because she was so at home there I guess they just gave her a key. I was on the couch and I opened my eyes and there's Lindsay in my room at 3am."
But the 35-year-old star also told Howard Stern on his radio show: "I will swear on anything that you ask that I have never had sex with Lindsay Lohan! Here's the problem: I bet you if we brought her in here and you asked her to her face 'Did you have sexy with James?' she'd say yes.
"I can't believe she put me on that private list. She's so delusional!"
When Howard suggested there must be some truth to the list, James confessed he locked lips with the 27-year-old actress while filming 'Spider-Man 2' in 2004.
He said: "Alright, we maybe kissed...It was lame."
He added: "When we made out it was so long ago. I was like a nice guy."