All passengers on board the Norman Atlantic, which was travelling from Greece to Italy on Sunday when the fire broke out, have now been rescued.
Showjumper Nick Channing-Williams, 37, said the fire was "very scary" and at one point he thought the situation was going "the wrong way".
The 422 passengers aboard also included a British family of four.
The Italian car ferry was travelling from Patras, in Greece, to Ancona, in Italy, also with 56 crew members on board, when a fire broke out on its car deck, north-west of Corfu.
Five people are now confirmed to have died, according to the Greek coastguard. Their nationalities have not yet been confirmed.
Mr Channing-Williams, who lives in Greece, is a competitive show jumper and trains young riders.
He was on the boat with his 33-year-old Greek fiancee Regina Theofili, who was rescued and taken to hospital.
His mother Dottie Channing-Williams, from Berkshire, told BBC Breakfast the wait for news was "just a rollercoaster, up and down".
Mr Channing-Williams, 37, who was travelling to see his family in Berkshire for new year, was eventually airlifted to safety.
He said: "When the flames are licking up around the boat and there is just no sign of help and they are talking about sending a boat that is going to be four hours away you feel somewhat helpless.
"Around five o'clock this morning I did send a couple of text messages out to people because I sort of had convinced myself that we were going a little bit the wrong way."
By the time he reached the deck, he said, "the flames were huge, and all the cars were on fire".
He added: "It was actually very scary, to be honest. There was only one place you could stand, which was in the rain.
"The fire was basically cooking everybody's feet and everyone was in a queue to get on a lifeboat. With the heat just being so enormous, people just panicked. I didn't even try and get on one.
"Regina and I were stood upstairs and just hoped for the best, really, that someone would come and help us."Susan Daltas, who lives in Corfu, said four members of her British family had been on the stricken boat. She told BBC Breakfast she believed that women and children had been taken off first.
Her son-in-law, Marcus, spoke to Mrs Daltas from the ferry on Sunday evening after her daughter, Mia, and her two granddaughters had been airlifted to safety.
Her youngest granddaughter had been taken to hospital in the city of Brindisi, in southern Italy, Mrs Dalta said.
"She was suffering from hypothermia because they didn't even manage to get a coat out of the cabin before they had to go on deck," she said.
The girl's mother and sister were on one of the rescue boats, said Mrs Daltas.
Her son-in-law was not able to tell her much about conditions on board, other than to say it was "cold and wet" and that he wanted to be reunited with his family.
British ambassador to Greece John Kittmer said the embassy was in "close contact" with the Greek authorities and was urgently seeking more information".
A Foreign Office spokeswoman said it remained "in close touch with the Italian authorities".
