The 35-year-old actor - who is currently in a relationship with British actress Tamsin Egerton - claims that romancing some of his former co-stars has had a negative impact on his career and made him "a lot of enemies" in Hollywood.
Asked about finding love on set, the 'Penny Dreadful' star told Elle magazine: "I don't recommend it to young actors. You can make a lot of enemies in the business that way. But when you work with somebody every day, it's like trial dating. You develop a fantasy about them. It doesn't always work out, does it?"
Josh dated Scarlett for almost two years until 2007 after meeting her on the set of 'Black Dahlia,' and also dated Julia Stiles in 1999 after meeting her on the set of 'O.'
He split from 'Les Miserables' actress Amanda Seyfried, 27, in June 2012 after six months of dating, and also briefly romanced Kirsten Dunst, Sienna Miller, Rihanna and Penelope Cruz.
The actor also suggested jealousy has been a problem for some of his famous ex-girlfriends.
Discussing the secret to a happy relationship in Hollywood, he said: "Let them be on set whenever they want, with whichever co-star you're working with. Never say the scene is too sensitive. They get very suspicious."
But the actor - who began dating Tamsin "long after" he met her on the set of 'Singularity' in 2012 - admitted that none of his friends are willing to listen to him complain about dating some of the world's most beautiful women.
He said: "They want to hear all about what you're doing, but they don't want to hear any complaints. Back when I dated a few very beautiful, very famous girls, I said something once - it wasn't really a complaint - and my buddy said, 'Oh my God, my diamond shoes are too tight.' He coined the phrase."