It wants the number of hospital units cut to ensure 24-hour access to care from senior doctors and says more midwife-led units are needed for women with low-risk pregnancies.
The National Childbirth Trust welcomed the report but says the proposals do not go far enough.
NHS managers said maternity care desperately needed to be reorganised.
'Serious complications' Too many babies are born in traditional hospital units, says the college, which also warns the current system is neither acceptable nor sustainable in its report on maternity care.
RCOG president Anthony Falconer told the BBC that most out-of-hours care was being provided by junior doctors.
"You need the right person, as senior person, there immediately"
Previous attempts to re-organise maternity care around a smaller number of hospital units have proved controversial, but Dr Falconer said if women could be convinced of the greater safety they would be prepared to travel to have their babies.
The need for change would be largely in cities or large towns, because in rural areas it might be more important to support smaller units.
The report estimates that across the UK there are 56 units with fewer than 2,500 deliveries of babies a year.
In order to take the pressure off busy hospitals, the college is also calling for an increase in the number of midwife-led units.
'Joined-up care' Midwives have welcomed the report, saying it could improve the experience for about a third of women who have straightforward deliveries.
The proposals for maternity are part of a wider vision of delivering all women's gynaecology and obstetrics care in networks, similar to the model which has helped improve cancer treatments in England.
The National Childbirth Trust said the idea of having a network to provide joined-up care for women was one it could support but it would prefer care during pregnancy and maternity to be concentrated in one NHS organisation in each area.
The NHS confederation, which speaks for managers, described maternity care as a classic example of a service which desperately needed to be reorganised.
Chief executive Mike Farrar said politicians needed to be prepared to speak up for change.
"Where the case for change is clear, politicians should stand shoulder-to-shoulder with managers and clinicians to provide confidence to their constituents that quality and care will improve as a consequence of this change."
That has not always been the case, with two ministers in the last Labour government campaigning against the closure of units in Greater Manchester.
Hundreds of people turned out to a rally to oppose the closure of maternity services in Salford last autumn. After a review under the coalition, the NHS is pressing ahead with plans to reduce the number of units across the area from 12 to eight.
Although Scotland has reorganised some of its maternity services, there are likely to be pressures for change elsewhere in the UK.
In North Wales maternity care across three hospitals is expected to change after an initial review recently concluded improvement was needed.