A patent application from Oregon-based software company Digimarc describes the use of digital watermarks and blockchain to combat deepfakes, which are being exploited for the purposes of misinformation and privacy violation.
Deepfakes make use of a form of artificial intelligence called deep learning to produce fake media, hence the name. Creating them requires thousands of face images of two people, which go through an AI algorithm, called an encoder. The encoder produces compressed images of the two faces with only their shared common features. Another AI algorithm, called a decoder, is employed for each person being swapped, recovering their faces from the respective compressed images. Feeding one person’s encoded images into another’s decoder results in convincing facial reconstructions accounting for expressions and facial orientation.
The technology can and has been applied to defame or defraud unwilling parties, and not only through visual fakery. In 2019, the deepfaked voice of a CEO allegedly netted fraudsters about $243,000 from a German energy company. In another worrying aspect, a report published by The Brookings Institute predicts that deepfakes will play a heavy role in distorting democratic discourse, manipulating elections, eroding trust in institutions, weakening journalism, exacerbating social divisions, undermining public safety, and inflicting hard-to-repair damage on the reputation of prominent individuals. Similarly, a Forbes article calls for urgent action against the political and social destruction deepfakes are bound to unleash.
The method described in Digimarc’s filing involves embedding watermark payloads into the audio and video tracks of files from trusted content producers. The watermark payloads comprise segment identifiers which are numerical sequence indicators for media content segments and a hash of said content segments. Hashing is when a string of characters is transformed into one of a shorter fixed-length value. In this case, data on a person’s face would be assigned a unique value. This feature serves to counter attempts to copy the watermark, since the hash for segments with a person’s facial features will not match the hash embedded in the watermark.
This process is done at the time a video is captured or before its distribution. These watermarks are expected to appear everywhere in a deepfake video’s frames except where the faces had been swapped. The watermarks also allow video source tracking, integrity verification, and alteration localization.
A diagram of a process of authenticating and localizing alteration using watermarks.
Digimarc suggests creating watermark detectors as standalone software applications, or integrated into other platforms such as in social media. The watermark detectors perform three main tasks: alert users when they encounter deepfake content, prevent said content from spreading, and enable forensic analysis to help track and remove deepfake postings.
Digimarc’s approach is unique from other deepfake detectors because of their use of blockchain technology. Blockchain is employed by various industries to handle large amounts of data, from financial transactions to supply chains to user credentials. A blockchain is an immutable ledger of information, where any and all entries are immediate, shared, and visible to network members. Every transaction is recorded as a “block” of data, and since every block is connected one after the other, the ledger becomes harder to tamper with as it grows longer.
Digimarc says embedded watermarks would contain an identifier corresponding to one block in the blockchain, created before a piece of content is distributed. The block includes the original video’s metadata, with extracted facial features such as the location of a person’s eye, nose, or mouth in an image. This unmodifiable record of features would match those found in a distributed video that has not been altered by AI.
The technology described in Digimarc’s patent application could serve as a safeguard against a new and powerful form of content. Efforts to protect genuine information can save many lives from ruin, and ensure deepfakes remain nothing more than an advanced internet oddity.
The featured patent application, “System For Mitigating The Problem Of Deepfake Media Content Using Watermarking”, was filed with the USPTO on January 15, 2021 and published thereafter on July 29, 2021. The listed applicant is Digimarc Corporation. The listed inventors are Adnan M. Alattar, Ravi K. Sharma, and John A. Scriven.
