Pressure is mounting on Prince Andrew to give up his 30-room mansion after not paying rent for 22 years.

But amid the clamour to evict him from Royal Lodge, property experts said it would be 'impossible' to get rid of him as he has a 'cast iron lease'. Senior Tory Robert Jenrick declared it was 'about time Prince Andrew took himself off to live in private' as 'the public are sick of him'.

It comes after a posthumous memoir by Virginia Giuffre alleged three sexual encounters with the prince – who gave up his titles last week over his relationship with Jeffrey Epstein, and his links to an alleged Chinese spy. Andrew has always denied the accusations. Andrew has lived for two decades rent-free at Royal Lodge, paying just 'one peppercorn (if demanded) per annum' – according to the extraordinary terms of his lease made public by the Crown Estate, which hands its profits to the Treasury for the benefit of the nation's finances.

Parliamentary committees could now look into the Crown Estate's handling of the grandiose home set in 98 pristine acres of Windsor Great Park. Dame Meg Hillier, chairman of the Treasury committee, said: 'Where money flows, particularly where taxpayers' money is involved or taxpayers' interests are involved, Parliament has a responsibility to have a light shine upon that, and we need to have answers.' 

At one point yesterday, Whitehall sources believed the spending watchdog, the National Audit Office, could launch a probe because it was in 'the public interest' to examine whether Royal Lodge was 'value for money' to taxpayers. But one source admitted they had concluded Andrew's lease on the property was watertight, adding: 'There are no plans to look into this now. 'That might change at some point in the future. There is a lot of political pressure.'

Mr Jenrick, the Tory justice spokesman, told Radio 4: 'It's about time Prince Andrew took himself off to live in private and make his own way in life. I don't see why the taxpayer, frankly, should continue to foot the bill at all. The public are sick of him.' Trying to get Andrew, 65, and his ex-wife Sarah Ferguson, 66, kicked out of the property would be impossible, according to top property lawyer Mike Hansom of BLB Solicitors. Estate agent Henry Sherwood agreed they would be unlikely to get him to leave against his will – unless lawyers were able to dig up an obscure ancient law giving senior royals the power to oust him.

Under the terms of his 75-year lease, signed in 2003, the prince had to pay £1million upfront and then agree to spend £7.5millon on urgently needed renovations. Unless he breaks the terms, he and his family will have the run of the mansion until 2078. According to the lease, the 'nominal' annual rent on the property was £260,000. But the £8.5million initial outlay is equivalent to £113,000 per year, if Andrew or his family remained in the property for 75 years, less than half the supposed market rate.

Mr Sherwood suggested: 'The yearly rent could be as much as £1.2million per annum if it was on the open market.' This works out at £17million over the years. Ultimately there is no break clause in the contract, which means the King – who has repeatedly tried to convince his brother to move out – cannot summarily throw him out.

The public accounts committee said its own programme of inquiries was 'full up until the new year', but it would 'decide in due course' whether to probe the Crown Estate's accounts and annual reports. At the weekend, it emerged that Andrew had asked his police protection officer to investigate claims his accuser Virginia Giuffre had a criminal record, which has prompted Scotland Yard to launch a probe.

And in her memoir, published yesterday, Ms Giuffre claimed that Andrew's team tried to hire 'internet trolls to hassle' her in an attempt to avoid being served court papers. Andrew has always denied all the allegations against him.