Vaughan, who is himself sidelined as he continues to battle to overcome a hamstring strain, said the self-doubt permeating the team was "cancerous" and a real threat to England's hopes in the upcoming World Cup.
The 32-year-old said England were working hard to turn their form around, ahead of a crucial triangular series One-day international against New Zealand at Perth's WACA ground on Tuesday.
A loss to the Black Caps would all but end England's hopes of reaching the tri-series finals and Vaughan said the biggest challenge for his side was shedding the mental frailties so evident in their two losses in Adelaide, when they made just 120 and 110.
"At the end of a long, hard tour like this one, it is a mental problem," Vaughan admitted here Sunday.
"Tuesday's game will be a real big game mentally for the guys, to see if they can come back from the disappointment of Adelaide.
"In eight years in the England team, that was as low as I have ever seen players feel.
"That is what tours like this can do to people.
"You are getting beat up most days for the whole tour. It has to have some kind of effect."
Vaughan, who again backed besieged coach Duncan Fletcher, has missed the past three England games with his injury and admitted it was hard to watch the side play so poorly from the stands.
He said England's dismal batting reflected their muddled thinking after a string of defeats on Australia soil.
"You can't defend the way we have batted... we have to work out what is going wrong with our thought processes in the middle," he said.
"A lot of the times we are getting out, it is the thinking that is wrong and that is purely mental. It is not technique."
Vaughan dismissed suggestions the English were already marking time and simply waiting to go home, saying it was crucial they left Australia in improved form.
"We need to change the momentum fast on this trip and make sure we try and win a couple of these last three games to get into the finals," he said.
"You can't keep performing to the standard we have in the last week or so and expect to go to the World Cup in good shape, that is nonsense."
He added: "The individual standards of players have to improve and, if they do that, the results tend to follow."
Vaughan said he was shocked by England's poor efforts in Adelaide.
"The two games in Adelaide came a bit out of the blue, to be honest," he said.
"I just didn't see those batting performances coming. I watched the guys in the nets and they were all hitting it pretty well."
Vaughan continues to struggle with his injury and looks unlikely to play in Tuesday's match against New Zealand at the WACA.
He is desperate to play again, but said he felt a twinge in his left hamstring during training on Sunday.
"I'd have to play a full part in all training tomorrow," he said.
"We'll have to monitor how I wake up tomorrow."
Opening bowler James Anderson and Jon Lewis both remain in doubt for Tuesday's game, failing to train on Sunday.
Vaughan said wicketkeeper Chris Read had a chance to be recalled for Paul Nixon, while Ravi Bopara was in line to make his international debut.