Deepfake democracy: Behind the AI trickery shaping India’s 2024 election

New Delhi, India — As voters queued up early morning on November 30 last year to vote in legislative elections to choose the next government of the southern Indian state of Telangana, a seven-second clip started going viral on social media.

Posted on X by the Congress party, which is in opposition nationally, and was in the state at the time, it showed KT Rama Rao, a leader of the Bharat Rashtra Samiti that was ruling the state, calling on people to vote in favour of the Congress.

The Congress shared it widely on a range of WhatsApp groups “operated unofficially” by the party, according to a senior leader who requested anonymity. It eventually ended up on the official X account of the party, viewed more than 500,000 times.

It was fake.

“Of course, it was AI-generated though it looks completely real,” the Congress party leader told Al Jazeera. “But a normal voter would not be able to distinguish; voting had started [when the video was posted] and there was no time for [the opposition campaign] to control the damage.”

The astutely timed deepfake was a marker of the flood of AI-generated, or manipulated, media that marred a series of elections in India’s states in recent months, and that’s now threatening to fundamentally shape the country’s coming general elections.

Between March and May, India’s nearly one billion voters will pick their next national government in the world’s, and history’s, biggest elections. The threats posed by deceptive AI-generated media caught the world’s attention when faked sexually explicit images of the artist Taylor Swift appeared on social media platforms in January. In November, Ashwini Vaishnaw, India’s information technology minister, called deepfakes a “threat to democracy” and Prime Minister Narendra Modi has echoed those concerns.

But with the increased availability of handy artificial intelligence tools, teams across India’s political parties, including Modi’s Bharatiya Janata Party and the Congress, are deploying deepfakes to influence voters, managers of nearly 40 recent campaigns told Al Jazeera. While several AI tools used to generate deepfakes are free, others are available on subscription for as little as 10 cents per video.

‘Creating perception’

The BJP, arguably India’s most technologically sophisticated party, has been at the forefront of using illusions for campaigning. As far back as 2012, the party used 3D hologram projections of Modi so that he could simultaneously “campaign” in dozens of places at the same time. The strategy was deployed widely during the 2014 general elections that brought Modi to power.

There was little deception involved there, but in February 2020, Manoj Tiwari, a BJP member of parliament, became among the world’s first to use deepfakes for campaigning. In three videos, Tiwari addressed voters in Delhi ahead of the capital’s legislative assembly elections in Hindi, Haryanvi and English – reaching three distinct audiences in the multicultural city. Only the Hindi video was authentic: The other two were deepfakes, where AI was used to generate his voice and words and alter his expressions and lip movement to make it almost impossible to detect, just on viewing, that they were not genuine.

In recent months, the Dravida Munnetra Kazhagam (DMK), which rules the southern state of Tamil Nadu, has used AI to resurrect its iconic leader M Karunanidhi from the dead, using lifelike videos of the former movie writer and veteran politician at campaign events.

Now, consultants and campaign managers say the 2024 elections could turbocharge the use of deepfakes even further.

“Politics is about creating perception; with AI tools [of voice and video modulation] and a click, you can turn the perception on its head in a minute,” said Arun Reddy, the national coordinator for social media at the Congress, who oversaw the party’s tech-savvy Telangana election. He added that the team was bursting with ideas to incorporate AI in campaigning, but that they didn’t have enough “trained people” to execute them all.

Reddy is strengthening his team – as are other parties.

“AI will have a resounding effect in creating the narrative,” Reddy told Al Jazeera. “The political AI-manipulated content will increase multifold, much more than what it ever was.”

‘Campaigns are getting weirder’

From the desert town of Pushkar in western India, 30-year-old Divyendra Singh Jadoun runs an AI startup, The Indian Deepfaker. Launched in October 2020, his company cloned the voice of Rajasthan state’s Congress chief ministerial candidate Ashok Gehlot for his team to send personalised messages on WhatsApp, addressing each voter by their name, during November assembly elections. The Indian Deepfaker is currently working with the team of Sikkim’s Chief Minister Prem Singh Tamang for holograms during upcoming campaigns. Sikkim is one of India’s smallest states in the northeast, perched on the Himalayas between India, Bhutan and China.

That’s the clean, official work, he said. But in recent months, he has been swamped by what he describes as “unethical requests” from political campaigns. “The political parties reach out indirectly via international numbers on WhatsApp, burner handles on Instagram, or connect on Telegram,” Jadoun told Al Jazeera in a phone interview.

In the November election, his company denied more than 50 such requests, he said, where potential clients wanted videos and audio altered to target political opponents, including with pornography. As a startup, Jadoun said his company is particularly careful to avoid any legal trouble. “And it is a very unethical use of AI,” he added. “But I know many people who are doing it for very low prices and are readily available now.”

During the election campaigns for the state legislatures of Madhya Pradesh in central India and Rajasthan in the west last November, police registered multiple cases for deepfake videos targeting senior politicians including Modi, Shivraj Singh Chauhan, Kailash Vijayvargia (all BJP) and Kamal Nath (Congress). The deepfake content production is often outsourced to private consulting firms, which rely on social media networks for distribution, spearheaded by WhatsApp.

A political consultant who requested anonymity told Al Jazeera that numbers of ordinary citizens with no public profile are registered on WhatsApp and used for the campaigns to make it harder for anyone to directly trace them back to parties, candidates, consultants and AI firms.

This consultant ran six campaigns in assembly elections last year for both the BJP and Congress. “In Rajasthan, we were using phone numbers of construction labourers to run our network on WhatsApp,” they said, “where deepfakes were primarily circulated.”

Meanwhile, AI-manipulated audios are particularly valuable tools in smaller constituencies, “targeting candidates with forged call recordings about arranging ‘black money’ for elections or threatening someone to buy votes,” the consultant said, whose own candidate was targeted with one such recording. The recordings are generally masked with candidates’ voices to cast them as evidence of corruption.

“Manipulating voters by AI is not being considered a sin by any party,” they added. “It is just a part of the campaign strategy.”