The Howard League for Penal Reform says it will spend two years looking at all elements of the issue, from consensual to coercive sex in jails.
The investigation in England and Wales will also look at what can be done to improve the sex education of adolescent and teenage inmates.
The UK has a ban on conjugal visits for prisoners - although more than half of European countries allow them.
There is virtually no reliable information about the extent of consensual and coercive sex in jails, although ministers recently told Parliament there were almost 140 sexual assaults in 2010, up on the two previous years.
The Howard League said it wanted its independent commission to shed light on the issue, which had never been properly discussed.
The study has partly come about after the charity got involved in the cases of three teenagers who said they had been raped by other inmates.
The research team will try to assess the scale of sexual offending in prisons - but will also look at whether the ban on consensual sex should go.
Sex is unlawful in prisons because cells are deemed to be public places - but other countries take a completely different view.