You've all focused upon efficiency and value for money. But the most important argument in favour of the private sector taking over roles from government is innovation. Basically government is usually much slower to bring in transformative technological change. By transformative I mean that the entire activity is changed - for example shifting from renting physical video tapes to streaming, or from landline telephones to mobile.
Of course governments can innovate, but they have big problems with transformative change as it usually involves large numbers of people no longer having a job (eg all the employees of video rental stores). The people who lose out can block innovation or limit its extent.
In the private sector, a company that doesn't innovate (eg Blockbuster) gets replaced by one that does (eg Netflix). That's sh.tty for all the people who worked at Blockbuster, but overall society benefits from much greater choice, better TV shows, lower costs for regular users, and far less use of physical resources (and so carbon emissions). On the latter just think of all the car journeys to the local Blockbuster store that no longer happen.
That doesn't mean that all sectors should be replaced with private firms. Natural monopolies are probably better off in the public sector. There also appears to be inherent limits to how much innovation could transform, for example, education or healthcare. Both are still based upon an expert interacting with a small number of people. So incremental change may well be the best we can hope for and so the innovation argument is weaker.* We do though rely upon private companies for aspects of healthcare where innovation is very important, such as developing new drug treatments.
*It does though seem conceivable that future advances in AI could transform education and healthcare, for example if everyone could have a personal robot doctor or teacher we wouldn't need humans to do that (or at least not as many of them). But that's a long way off, if it it could happen at all.