A study in humans has shown a compound in soya called genistein sabotages the sperm as it swims towards the egg.
Professor Lynn Fraser, from King's College London, said even tiny doses in the female tract could burn sperm out.
She told a European fertility conference that avoiding soya around women's most fertile days of the month might aid conception.
Genistein is present in all soya-containing products such as soya milk and many vegetarian foods, as well as some pre-packed meals and pizzas.
It is also present in vegetables such as peas and beans, but the researchers say it is not yet clear if levels if the compound in these foods are significant.
Professor Fraser tested what happened to human sperm exposed to the compound in a dish in the lab.
The compound kick-started a reaction in a large proportion of the sperm that gives them the ability to fertilise an egg.
In real life, this does not usually happen until the sperm have been inside the female for some hours and are close to completing their long swim towards the egg.
Therefore, if women have genistein in and around the womb this could hamper conception by making sperm peak too soon, believes Professor Fraser.
This could mean they would not be able to fertilise the egg, she told the annual meeting of the European Society for Human Reproduction and Embryology.
In mice she found it took higher doses of genistein to cause the reaction, but in humans very small doses were enough.
"We were really surprised. Human sperm proved to be even more responsive than mouse sperm to genistein, responding to very low concentrations - well within the amounts that have been measured in people's blood."
She said it was not yet known how much soya might be a safe amount to avoid this effect.
"It's not a question of completely stopping eating products containing soya.
"But it might be best for a woman to avoid them for a few days around the time she is ovulating."
Professor Fraser's previous work in mice showed that compounds similar to genistein - one found in hop-based products like beer - affect sperm in the same way.
These compounds are all weak oestrogens, but Professor Fraser does not believe that their action on sperm is the same as the female sex hormone.
Instead, genistein seems to trigger the production of a signalling molecule in sperm called cyclic AMP.
Dr Allan Pacey, Senior Lecturer in Andrology at Sheffield University and Secretary of the British Fertility Society said: "It's early days, but clearly if what happens in the laboratory also occurs in the woman's fallopian tube as the sperm make their way to the egg, then there would be the potential for fertilisation to fail."
A spokeswoman from the Vegetarian Society said: "For anyone struggling to become pregnant, avoiding soya products for a few days a month is worth a try if there is even a slim chance that it will help increase fertility.
"Obviously many vegetarians and vegans use soya in their diet, however as there are lots of vegetarian and vegan alternatives to dairy, milk and meat on the market, it shouldn't pose a problem."