Voice print analysis will be used to see if the voice of the man, who threatens Prime Minister David Cameron and says IS will one day occupy the UK, matches that of any individuals known to have traveled to Syria and Iraq.
The man then shoots one of the hostages in the head, and masked men appear to kill the other four.
Following the killings, a young boy who looks about four or five years old and is wearing a bandanna, is shown saying: "So go kill the kuffar [unbelievers] right over there."
Before they are apparently shot, the hostages identify themselves as Syrians from Raqqa, and say they were supplied with equipment such as concealed cameras "to monitor" IS activities and purportedly "confess" to having completed reconnaissance missions sanctioned by British authorities.
It's unclear if they're involved with activist groups that have faced reprisal killings for documenting conditions in IS-controlled territory.
Abu al Furat, a member of one Raqqa-based activist group opposed to IS told the BBC the victims were shopkeepers and local businessmen and he doubted they had been spying for Britain.
"I know personally two of them, who started demonstrating very early," he said. "They are the sons of the Syrian revolution in Raqqa."
Using the Arabic acronym for the militant group, al Furat said: "Daesh is collapsing at the moment, but they are trying by doing this to say that their security bodies are working very well and they are still strong."
A UK Foreign Office (FCO) spokesman said the government was examining the content of the video, which he called "Daesh propaganda."
An FCO source told the Guardian it was likely IS is trying to divert attention from "its recent military failings in Iraq and its inability to look after people in the areas that it controls".
Margaret Gilmore, senior associate fellow at security think tank the Royal United Services Institute, reiterated this position, telling the BBC that IS was "trying to detract from recent setbacks." There was "an air of desperation" about the video, she said.
It will likely take some time for UK authorities to identify the masked man, who is being called the "new Jihadi John" — referring to Mohammed Emwazi, the Briton who appeared in multiple IS execution videos before being killed in a drone strike in Syria in November.
Around 800 Britons are known to have traveled to Iraq and Syria to join IS, of whom around half have returned.
Follow VICE News on Twitter: @vicenews