No, unfaithfulness is a personal choice. A man willingly chooses to go outside the marriage that he started and he is the head of.
If a man kills a person, will he accuse the wife that she led him to kill someone because she stressed him? Of course not. The court will hear none of that. The wife will not be prosecuted for the murder committed by the husband, so why does a man in the case of sexual crime want to accuse the wife?