ACFE ¦ Fraud Magazine Interviews AI and Deepfake Expert Hany Farid, Ph.D.

Deepfakes, identity theft and the end of seeing as believing

Images have been manipulated for as long as cameras have existed. Under Stalin, people were famously airbrushed out of official photographs, and digital cameras and photo-editing software later made alteration easier still. Convincing manipulation nevertheless remained the exception until recently. Generative AI has changed the ease, speed and cost of faking images, audio and video: anyone can now produce a convincing fake without skill or money, and social media can put it in front of millions of viewers within minutes. For fraud examiners, compliance officers and investigators, that pairing of cheap creation and instant amplification has become an operational threat to evidence, identity and trust.

A shared sense of reality under pressure

The deeper problem is the erosion of a common understanding of what is real. Institutions, markets and democracies can live with people disagreeing about how to respond to the facts. They struggle when people disagree about whether the facts exist. Once any recording can be dismissed as fabricated and any fabrication can pass as genuine, that common ground starts to give way.

The internet promised to democratise access to information. In practice, gatekeeping power moved from broadcasters who were at least publicly accountable to a handful of technology platforms whose incentives are built around attention and advertising. For some two decades, society and policymakers took a largely hands-off approach to the internet and the technology sector. That period may be ending, as many organisations, regulators and individuals have concluded that the current course harms children, institutions and public debate. Decision-makers who do not work out what future they want will find that the technology industry has decided it for them.

Bastian Schwind-Wagner _Follow

"Deepfakes have become a working tool for fraudsters: a few seconds of someone's voice and a single photograph are now enough to impersonate them on a call. The controls compliance and anti-fraud teams rely on most – a familiar face on a video call, an instruction from the top, a recording submitted as evidence – are the very ones this technology attacks.

No detector will ever be perfect, so the defence has to come from disciplined process. That means verifying who is on the call, keeping a clean chain of custody, treating every AI output as triage for a human to check, and rehearsing incident response for information that has been falsified on purpose."

Understanding the technology is the first control

Professionals whose work depends on establishing facts – fraud examiners, auditors, journalists, prosecutors and regulators – cannot afford to ignore AI. The technologies are here to stay and will be put to creative and harmful uses alike. The first defence is knowing what these tools are and what they are not.

This applies inside organisations as much as outside them. Employees already use AI systems, often without any oversight, and upload company documents, presentations and invoices to public platforms. Organisations need clear policies and procedures, an honest view of where AI helps and where it does damage, and structured training. Where an organisation offers no training, staff should press for it.

Choosing whom to listen to also takes care. Professional networks are full of self-declared AI experts who were not working on the subject a year ago. Tenure is a practical test: has the person worked on machine learning or AI for many years, often in academia, or did they join the conversation last month? Serious understanding takes time and will not come from a single article or a short video.

Pattern matching, not intelligence

Much of the confusion comes from terminology. Academics coined the term artificial intelligence in the 1950s, aiming to build computers that reasoned like the human brain by following encoded rules. For decades the field overpromised and underdelivered, passing through repeated cycles of hype and disappointment. In the 1980s, machine learning branched off with a different approach: ignore the rules, feed the system vast numbers of examples and let it learn from the data. Machine learning stalled as well, until about a decade ago, when graphics processors and above all the enormous volumes of data produced by two decades of social media made it work.

Almost everything called AI today is therefore machine learning, which is sophisticated pattern matching. These systems have no understanding of the physical world. Two broad categories matter. Predictive systems estimate outcomes, such as whether a loan applicant will default or whether someone applying for parole will reoffend, and they differ little from the statistical modelling that existed before anyone attached the AI label. Generative systems are the genuinely new part. They produce images, audio, video and text, from deepfakes of people saying and doing things they never said or did to large language models.

A large language model is a highly sophisticated autocomplete system. It is trained on vast amounts of text, much of it scraped from the internet, and refined through human feedback, with people rating answers and supplying better ones. The models are increasingly able to help train themselves. They work remarkably well, yet thinking of them as pattern matchers changes how they should be built into professional workflows and which questions they can be trusted to answer.

Detection will never be perfect

AI-generated text is getting harder to spot. Tools that claim to detect machine-written text are not very accurate, and a text produced by one model and then rewritten to sound more natural gets past many of them. The familiar stylistic tells will fade. A 200-page thesis written by a language model will probably still be caught; an 800-word opinion piece is much harder. Content is also turning hybrid, with photographs edited by AI, recordings cleaned up by AI and texts polished by AI, so questions of provenance will get more complicated over time.

For images, audio and video, perfect detection is out of reach. Watermarking, provenance standards and sophisticated forensic techniques all help, yet none of them will reach 100 per cent accuracy. Time is just as hard a constraint. By the time a viral video reaches a forensic analyst and has been examined, it may already have been viewed ten million times. What defenders can realistically achieve is to raise the cost and difficulty for attackers, much as a locked door makes burglary harder without making it impossible.

Practical rules for examining media

Fraud examiners who have to assess documents, images or videos should start from humility. Human eyes and ears are poor detectors of synthetic media, and guidance that was accurate six months ago may already be out of date. Genuine content can look strange because of low resolution and compression, and fakes can look entirely normal, so a single oddity should never be treated as decisive.

How much can be said about a file depends almost directly on its quality, resolution and length. A heavily compressed eight-second clip supports very limited conclusions, while a twenty-minute video in 4K supports many more. The more the signal has been degraded, the harder the analysis. Examiners should also assume that what AI cannot do today it will probably do tomorrow.

The best defence against an adversary is to think like one. Examiners should use image, voice and video generators themselves to learn what they can and cannot produce, because the limits of the tools are also the limits of the attacker.

A forensic workflow built on provenance

Every case differs. Images behave differently from audio, video with sound from silent footage, a public figure from a private individual, an outdoor scene from an indoor one. A typical workflow still follows a consistent sequence.

It begins with provenance: where the content came from, where it was first shared and who is sharing it. A known bad actor or a state-backed source changes the assessment considerably. Next comes the best available copy. Viral content usually circulates in many versions, and the earliest one in the chain usually holds the most evidence. Relatively simple checks follow, such as extracting metadata and looking for watermarks, which sometimes settle the matter and often do not.

The analysis then turns to how the content would have been faked. Was a person added or removed? Is the whole audio track synthetic, or were only a few words changed to alter the meaning? Each hypothesis points to specific traces to test, such as inconsistent shadows, lighting, reflections or vanishing points, or artefacts of synthetic faces and voices.

Visual intuition has become far less reliable than it was even a year or two ago, so the work depends more and more on specialised software and on several reviewers looking at the same material. A second pair of eyes often spots in seconds what the first missed after twenty minutes. Some cases take five minutes and others take days, and the stakes decide the depth of review: war footage gets different treatment from a harmless viral image.

The method has a built-in asymmetry. A flaw that can only be explained by manipulation supports the conclusion that the content is fake. Finding nothing after a long series of tests proves nothing about authenticity, and the honest finding is that no evidence of manipulation was found. That uncertainty is uncomfortable, particularly when the content records real suffering, and it comes with the territory of authentication.

Digital evidence in court

Courts demand the highest standards because mistakes there have severe consequences. Evidence is strongest when it comes from the source device itself rather than from a social media platform, with its metadata intact and without having been reopened and recompressed. A clear chain of custody is essential, and even opening and saving an image changes it, since the file is recompressed and may pick up new metadata.

AI-based image enhancement calls for particular caution. Studies of these tools show that they can distort or invent detail when sharpening a blurred image. Enhanced material should not be accepted as evidence on its own and, if it is used at all, must always be checked against the original.

A further problem is emerging inside the camera. Smartphones already run a processing pipeline between the press of the shutter and the stored file. As AI moves into that pipeline and closer to the sensor, images become AI-enhanced by default. At least one smartphone camera has been shown to generate lunar surface detail instead of recording the moon actually in front of it. For birthday photographs this hardly matters. For documenting crimes, human rights violations and other evidence, it raises serious questions that camera manufacturers, courts and investigators will have to answer.

Why AI systems hallucinate

Hallucinations follow directly from how these systems work. They are statistical inference engines that produce the most probable output for the input they receive. Enhance a blurred frame showing a dark shadow beside a person’s hand, and the system may turn the shadow into a gun or a phone, simply because that is a statistically plausible completion. Language models hallucinate for a second reason too: the internet they were trained on is often wrong, so some apparent hallucinations are uncorrected errors in the training data.

For investigators, AI output can never stand in for the underlying evidence. It is an inference drawn from that evidence.

The sectors that are not paying attention

Financial institutions are beginning to take the threat seriously. They are usually the first line of defence when new technology is abused, although their response is not yet fully effective. IT help desks are also waking up to AI-supported social engineering in password resets, credential recovery and phishing.

Human resources gets far less attention as a point of attack. Recruitment, onboarding and daily work have largely moved online, and the trust built on a video call is often taken for granted. Sophisticated hiring fraud is already happening, from North Korean IT workers infiltrating companies to bait-and-switch interviews where one person sits the interview and another takes the job. In one case, a single individual held five full-time positions at competing companies and secretly outsourced the work abroad. Apart from the payroll fraud, schemes like this expose confidential information and create serious sanctions and security risks.

Brand protection is another neglected area. A fabricated video or image of a chief executive saying or doing something offensive can spread fast and damage a company before it has time to react. The influencer economy is filling up with AI-generated personas and bots, often without the paying brands realising it. Third-party vendor management is exposed as well: large companies with thousands of suppliers face fake invoices and fake calls that exploit how hard it is to keep track of every counterparty. Cybercriminals adapt to new technology quickly, and no sector is immune.

Verifying who is on the call

Human resources has one advantage, which is that organisations control their own processes. Visitors to an office building usually show identification, pass security and receive a badge. The same organisations routinely hold sensitive calls with participants whose cameras are off, who appear only as a generic name or phone number, or who have renamed themselves. Technology now exists to verify who is actually on a call and whether they are using AI to disguise their identity. Recruitment, where candidates can reasonably expect scrutiny, is the logical place to start. The problem is hard but solvable.

Brand protection is harder because no organisation can control what eight billion people do. The best available response combines continuous monitoring, awareness of the threat and established relationships with the major platforms, so that a channel for rapid takedown or response exists before an attack escalates. These attacks happen almost daily, and AI has made them faster and more damaging.

What an AI training programme should contain

An internal programme should start with hands-on use, so that staff know what these tools are, how adversaries use them and how they behave in practice. Clear policy and clear ownership come next. In many large organisations nobody knows who is responsible for AI-related risk. Whether or not a chief AI officer is needed, someone has to own the issue, and everyone has to know who it is.

Procedures should also cover specific scenarios. If a deepfake appears during a call, staff should know whether to end it, isolate it, record it or escalate it. Only after education and policy should organisations decide where detection technology makes sense, with recruitment as an obvious first candidate. They should also find out where they are being attacked, on the assumption that they are. Penetration testing and red-teaming can expose weaknesses before adversaries find them.

Responsible use belongs in the same programme. Uploading confidential documents to public AI platforms can create significant legal liability and expose trade secrets. Some uses should be ruled out altogether. Algorithms should not decide questions of civil liberty such as bail or parole, even though some courts already use them for exactly that. All these systems are trained on human-generated content, which carries biases of gender, race, sexual orientation and age. Used predictively on historical data, they reproduce past injustice in a modern form that looks objective. Knowing that the technology learns patterns from history is the best protection against treating it as an oracle.

Incident response must assume adversarial information

Organisations should run tabletop exercises that include deepfake scenarios. In one exercise built around a classic ransomware attack, the company had a well-rehearsed playbook. The organisers then added a fake video call from the chief executive telling the team to pay the ransom, and the response fell apart because nobody had anticipated a fabricated instruction from the top.

Ownership matters here too. If a fabricated video of the chief executive behaving outrageously starts circulating, it has to be clear whether communications, human resources, the chief technology officer, IT or some other function leads the response. Such exercises often go badly for a simple reason: incident response assumes the available information is broadly accurate, and few organisations are prepared for information that has been falsified on purpose by an adversary. Impersonation fraud schemes exploit exactly this weakness, using synthetic voices and faces to get payments authorised, and their proceeds then enter the laundering chain like those of any other predicate offence.

Using AI to detect fraud – with a human in the loop

The technology has real benefits. Generative tools lower the barriers to film-making, music and art, and let small teams do work that once needed many more people. A language model used well can act as a tireless tutor that explains concepts, generates examples and tests understanding. Used badly, it lets students hand in work they could never have produced and leaves them unable to read or write properly. The responsibility lies with the people who set guardrails and teach others to use the tools well, and who also face the hard questions about training data taken without consent and creators being displaced.

Many fraud examiners already use AI to detect anomalies in transactions, which is a sensible use. Language models do poorly when asked to find manipulation in images and video, yet finding patterns in data that may point to fabrication plays to their strengths. Three safeguards apply. Every output should be treated as triage for a human to verify, because the model may invent results and its reasoning is opaque. Users should regularly test the tools with questions whose answers they already know, to learn what each model does well and badly. And they should keep in mind that these models are probabilistic and change constantly, so the same question can get different answers on different days or even on successive attempts. Running the same task on several models in parallel and looking for agreement between them makes the result more trustworthy.

Cost is turning into a practical constraint. Cheap or free access to large language models is unlikely to last, since providers are losing large sums and will raise prices. Open-source models are a reasonable alternative, but running or fine-tuning them in-house takes expertise, staff and expensive hardware. Smaller organisations face hard choices either way, much as they did when cloud computing bills first had to be brought under control.

Identity theft is the trend that matters most

The most worrying development is how easy it has become to steal someone’s identity. A few years ago, cloning a face and voice took hours of recorded material. Today ten to fifteen seconds of audio and a single photograph are enough, and almost anyone’s voice and image can be found online or simply recorded in public. A threat that once mainly concerned world leaders and celebrities now reaches people with modest digital footprints, and joining a call while posing as someone else has become easy.

Personal protection starts with awareness of the threat. Spending less time on social media helps as well, since the platforms are designed to capture attention for advertising and they expose users to manipulation, deception and fraud. Deleting the apps from a phone while keeping the accounts is a workable compromise. The opposite mistake is just as unhelpful: assuming everything is fake leaves people no better informed than believing everything. Humility, a healthy dose of scepticism and the patience to read past headlines and short clips serve far better.

Liability may move faster than regulation

Good, sensible regulation is needed, but technology moves faster than legislation, and rules are often out of date by the time they take effect. Liability may work faster. Cars, aviation, food and pharmaceuticals became safer because manufacturers learned they would face serious legal consequences if they knew, or should have known, that their products would cause harm. The technology sector has largely been shielded from comparable liability, and that shield is starting to wear thin. A few successful lawsuits could change corporate behaviour much sooner than a new regulatory framework.

No other safety-critical industry would be allowed to work under the motto “move fast and break things”, and the harms caused by technology reach people’s lives in the physical world. On regulation, the European Union has produced carefully considered rules, the United Kingdom has taken useful steps and Australia has thought hard about online safety. That international movement may yet influence jurisdictions that have so far held back.

Deepfakes and synthetic identities attack the trust on which onboarding, payment authorisation, recruitment and evidence all rely. The anti-fraud and AML functions that cope best will understand the technology, verify identities deliberately, rehearse their response to falsified information and keep a human responsible for every decision that matters.