An attacker who replaces a file can also replace its hash, since hashing requires no secret information. A check that depends on a secret key gives the receiver a way to distinguish an authentic file from a substitute. The sender and receiver can calculate the check, while an outsider must guess it or find a weakness in the algorithm.
Suppose a software publisher sends updates to a partner company, and the two already share a randomly generated key. The publisher calculates a tag from the file and the key, then sends the tag with the file. The partner repeats the calculation on the copy it receives. If the results agree, the partner has evidence that someone holding the shared key authenticated those bytes.
Message Authentication Codes
A message authentication code (MAC) is a value calculated from a message and a secret key. Also known as an authentication tag, it binds the message to the shared secret. The calculation is written as:
\[t = \operatorname{MAC}_K(m)\]
Here, \(m\) is the message, \(K\) is the shared key, and \(t\) is the tag. A receiver accepts the message only if the tag matches the result calculated with its own copy of the key. The term MAC names both the algorithm and the value it produces, so “the MAC of the file” and “the tag on the file” refer to the same thing.
A matching tag establishes two things at once: (1) the message was not altered, and (2) it was produced by someone holding the key. A hash function on its own establishes neither, since whoever changed the message can compute a new digest for it. A MAC provides integrity and authenticity together, and it does not need a separate trusted channel for the value it produces.
The tag can travel in the open. Even after observing many messages and their tags, an outsider should be unable to generate a valid tag for a new message. For an ideal 128-bit tag, a single blind guess succeeds with probability \(1/2^{128}\). The message itself stays readable throughout, so an application that also needs confidentiality must encrypt it.
Block ciphers provided an early foundation for MACs, and banking was the first commercial use. A wire transfer instruction is not especially secret, and both banks already know the kind of traffic that passes between them, but the amount, routing number, and the account number have to arrive exactly as they were sent. By 1985, the U.S. government had standardized data authentication using DES, the Data Encryption Standard, for that purpose. The current block cipher construction is CMAC, which turns up where a cipher is already available in hardware, as in Wi-Fi security.
A cipher produces as much output as it consumes, so a short tag takes an extra step. These constructions borrow the structure of cipher block chaining: each block of the message is exclusive-ored with the encrypted output of the block before it and then encrypted in turn. The final block of output therefore depends on every block of the message and on the key. That block is kept as the tag and the rest of the output is discarded. CMAC adds one adjustment, derived from the key, to the last block, which is what keeps the construction secure when messages differ in length.
Hash functions offered another way to produce these checks efficiently, but incorporating a secret key required care.
HMAC: Authentication Built from a Hash
A cryptographic hash already produces a compact fingerprint of a message. Putting a secret key before the message and hashing the combination seems like a natural way to make that fingerprint depend on a secret. With SHA-256 and some other hashes, however, this construction permits an attack called length extension.
SHA-256 processes its input in blocks, carrying a calculation forward from one block to the next. Its final digest contains enough information to continue that calculation. An attacker who knows the tag and the length of the key and message can calculate a tag for a longer message without learning the key. The added bytes must follow the original message and the padding SHA-256 used to complete its blocks. Although that requirement limits the changes an attacker can make, the proposed MAC has failed to prevent forgery.
The second preimage resistance property does not prevent this, because no second preimage is involved. The attacker is not looking for a different message that produces the digest already in hand. The extended message is a new message with a new digest, and the flaw is that the digest can be computed from the old one instead of from the key and the message. The hash function is doing what it was designed to do. The mistake is treating a digest as a value only a key holder could produce.
Flickr, a photo sharing service, authenticated its programming interface this way. In September 2009, two security researchers, Thai Duong and Juliano Rizzo, showed that an attacker could append parameters to a captured request and produce a valid signature for the longer version, without knowing the secret. Several other services had the same flaw.
HMAC is a hash-based message authentication code that avoids the length-extension problem by using two nested hashes. Hugo Krawczyk (IBM), Mihir Bellare (UCSD), and Ran Canetti (IBM) documented the construction in RFC 2104 in 1997. HMAC computes an authentication tag in three steps:
-
Prepare the key. If the key exceeds the hash function’s block size, hash it first. Then append zeros as needed to reach that block size.
-
Compute the inner hash. XOR the prepared key with a fixed, public pattern called the inner pad, append the message, and hash the result.
-
Compute the outer hash. XOR the prepared key with a different fixed, public pattern called the outer pad, append the inner digest, and hash the result. This final digest is the authentication tag.
The receiver repeats the calculation using the shared key and checks whether the tags match. The outer hash always processes a key-derived block followed by a fixed-length inner digest. Continuing the hash calculation from the final tag would extend this outer input, producing a different structure from the one the receiver hashes. The result would therefore not be a valid HMAC for an extended message.
HMAC-SHA-256 uses SHA-256 for both hashes and produces a tag of 256 bits. A library handles the key preparation and the hashing. The application supplies a key and the message bytes, then uses the library’s verification operation to check received tags. HMAC appears in TLS (the protocol behind secure web connections), IPsec (network-layer protection, as used by a VPN), and SSH (remote login), in signed web tokens (the session record a site issues after a login and checks on later requests), and in the request signatures used by most cloud provider interfaces. We will cover the first three later. The use of HMAC in TLS has narrowed in the latest version: data is protected by a different mechanism (AEAD, described below), while HMAC is used to derive the session keys for communication and to authenticate the end of the setup exchange.
What the Tag Protects
An authentication tag protects only the bytes included in its calculation. Suppose an update arrives with a file, a version number, and an HMAC covering the file. An attacker can change the version number without invalidating the tag. The receiver may then accept an intact file under a false description.
A manifest collects the information needed to identify a release, including its product name, version, and file digest. Authenticating the manifest binds those fields together. The receiver checks its tag and then compares the downloaded file with the authenticated digest. Both sides must interpret the record in the same way, which is why the format needs unambiguous field boundaries.
Combining Authentication and Encryption
Many applications also need to hide the messages they authenticate. Authenticated encryption with associated data (AEAD) provides encryption and an authentication tag in one operation. It can also authenticate information that remains readable, such as a packet header that routers have to read and nobody should be able to alter. That readable information is the associated data.
AES-GCM and ChaCha20-Poly1305 provide this combined protection for AES and ChaCha20 encryption, respectively, using their own authentication mechanisms, so they need no separate HMAC. TLS breaks the data it carries into chunks called records, and version 1.3 requires AEAD for every record sent once a connection is established, so each one arrives with its own tag.
New designs use AEAD rather than assembling their own combination. Systems that compose a cipher with a separate MAC have to choose which operation comes first. The safe order is encrypt-then-MAC: encrypt the plaintext, then compute the tag over the resulting ciphertext, so the receiver rejects a forged message before decrypting any of it. The older TLS cipher suites that paired AES in CBC mode with HMAC computed the tag over the plaintext instead, an ordering that produced a series of attacks and was eventually corrected by an extension.
These tags are what makes a stream cipher’s malleability harmless in practice. An attacker can still flip a ciphertext bit, and the receiver will still decrypt a flipped plaintext bit, but the tag will not verify and the message will be discarded before anything uses it.
The Limit of a Shared Secret
HMAC works for the publisher and its partner because both hold a secret that outsiders lack. This arrangement also gives the partner the ability to create an authentication tag for a file it wrote itself. A third party examining the file and its tag cannot determine which of the two key holders produced them. A MAC therefore provides no non-repudiation: a sender can deny having generated the tag on a message, and nothing about the tag contradicts the denial.
Shared keys also do not scale. Giving every pair of parties in a group of \(n\) its own key requires \(n(n-1)/2\) keys, and it says nothing about how two parties who have never met obtain a key securely.
Public software distribution makes the limitation even harder to accommodate. Putting the same MAC key in every installation means that anyone who extracts it from a single copy can produce forged updates that every other installation accepts. Giving each customer a different key requires the publisher to distribute and manage all those secrets, then generate separate tags for each customer.
Two solutions exist.
One is to introduce a party that everyone already shares a key with, which can then hand out keys to any pair that needs one. We will cover those protocols in detail later.
The other is to make verification public while keeping the ability to authenticate a release private, using a key that splits in two: one part kept secret and used to produce the value, and another part published freely and used only to check it.
The second answer is what public software distribution needs, because it puts no secret at all on the machines doing the verifying.