Hopes an EU summit on Wednesday would unblock Brexit talks have taken a blow, with agreement still to be reached on what happens at the Irish border.
The UK and the EU had hoped that enough progress had been made to call a special summit in November to finalise the terms of the UK's exit.
European Council president Donald Tusk said he now had "no grounds for optimism" on a breakthrough.
But Theresa May told her cabinet she still hoped to get a deal.
The UK's prime minister told a three-hour cabinet meeting on Tuesday morning she could not agree to any deal which created a new border between Northern Ireland and the rest of the UK - or one which locked the UK into a customs union with the EU indefinitely.
She says this would be the consequence of the proposals currently on the table from the EU.
The latest Brexit developments:
- A group of Brexit-supporting ministers met in the Commons on Monday night - apparently over pizza - to discuss the way forward, but the BBC has been told that no resignations are imminent
- The Democratic Unionist Party - whose backing Theresa May needs to win key Commons votes - is warning that it is prepared to "paralyse" her domestic agenda if she undermines Northern Ireland's position within the UK in the Brexit negotiations
- The Austrian foreign minister, along with several senior EU officials, have told the BBC they feel the chances of Wednesday's summit being "the moment of truth" for Brexit have diminished significantly
- German Chancellor Angela Merkel said Berlin wants the UK's withdrawal next March to be orderly "but not at any price", while Irish premiere Leo Varadkar suggested a deal may be delayed as late as December
Mrs May used Tuesday's cabinet to rally support for her position among senior ministers, amid reports eight of them had met on Monday to discuss their concerns about it.
Downing Street said none of the eight - Dominic Raab, Jeremy Hunt, Michael Gove, Penny Mordaunt, Chris Grayling, Liz Truss, Andrea Leadsom and Geoffrey Cox - had threatened to quit at Tuesday's cabinet meeting and it was clear that she had strong support.
Skip Twitter post by @bbclaurak
Sounds like no one pushed for a decision, so no pushed for a confrontation over what to do if the EU doesn't budge + there was this appeal to Cabinet from PM -
“I am convinced that if we stand together and stand firm we will achieve this“ - that is a BIG if
3:39 PM - Oct 16, 2018
Twitter Ads info and privacy
End of Twitter post by @bbclaurak
The cabinet had discussed a "mechanism" that would avoid an indefinite customs union if a full trade deal cannot be agreed by the end of the 21-month transition period that is due to kick in after the UK leaves on 29 March - the so-called "backstop" plan.
But it was "not a decision-making cabinet", Downing Street said.
The prime minister told ministers progress had been made in Brexit talks and although there would be challenging moments ahead a deal was within reach, Downing Street said.
"I am convinced that if we as a government stand together and stand firm we will achieve this," she told ministers.
Both sides in Brexit talks are hoping that a deal on the UK's withdrawal from the EU, including the Irish border question, will be agreed by mid-November in time for it to be ratified by EU members and for MPs at Westminster to vote on it.
But senior EU officials have indicated that there'll be no special Brexit summit in November, unless Mrs May puts forward what they consider to be an acceptable plan for dealing with the border between Ireland and Northern Ireland.
Media captionWhat will become of the Irish border when the UK leaves the European Union?
The EU's chief Brexit negotiator Michel Barnier, who is travelling to Luxembourg to brief EU leaders ahead of Wednesday's summit, said he hoped a deal with Britain was possible "in the coming weeks".
"We are still not there.
"There are still several issues which remain unresolved, including that of Ireland, and therefore what I understand is that more time is required to find this comprehensive deal and to reach this decisive progress which we need in order to finalise these negotiations on the orderly exit of the United Kingdom."