Applying Cryptography: TLS, IPsec, Email Security, and Crypto in the Wild

Hook / Why This Matters

CISSP Lens: Pick answers that align business risk, governance intent, and practical control execution.

Knowing AES and RSA is necessary but not sufficient. The real questions are: how does TLS negotiate a session? What is the difference between IPsec transport and tunnel mode? Why does S/MIME use hybrid encryption? Domain 3 tests your understanding of cryptography in action, not just in theory.

Core Concept Explained Simply

Cryptographic protocols assemble primitives (symmetric encryption, asymmetric encryption, hashing, digital signatures) into working systems that protect real-world communications. Understanding how these protocols work, and where they can be misconfigured, is essential for both the CISSP exam and practical security architecture.

TLS (Transport Layer Security)

TLS is the protocol that secures HTTPS, email transport, VPN connections, and most modern encrypted communications. The TLS handshake establishes a secure session through several steps:

1. Client Hello: The client sends supported cipher suites and a random value.
2. Server Hello: The server selects a cipher suite and sends its certificate.
3. Key Exchange: The client and server establish a shared secret using asymmetric cryptography (typically ECDHE).
4. Session Keys: Both sides derive symmetric session keys from the shared secret.
5. Encrypted Communication: All subsequent data is encrypted with the symmetric session keys.

TLS 1.3 improved on 1.2 significantly:

• Reduced the handshake from two round trips to one (improving latency)
• Removed support for legacy cipher suites (RC4, 3DES, static RSA key exchange)
• Required Perfect Forward Secrecy (PFS), ensuring that compromising the server's private key does not decrypt past sessions

IPsec (Internet Protocol Security)

IPsec secures network-layer communications and is the foundation of most site-to-site VPNs. It has two core protocols:

• AH (Authentication Header): Provides integrity and authentication but no encryption. It verifies that packets have not been modified and confirms the sender's identity. AH does not provide confidentiality.
• ESP (Encapsulating Security Payload): Provides confidentiality, integrity, and authentication. ESP encrypts the payload and can optionally authenticate the packet. For most use cases, ESP is the correct choice.

IPsec operates in two modes:

• Transport mode: Encrypts only the payload of the IP packet. The original IP header remains intact. Used for end-to-end communication between two hosts.
• Tunnel mode: Encrypts the entire original IP packet and wraps it in a new IP header. Used for site-to-site VPNs where entire network-to-network traffic is protected.

IKE (Internet Key Exchange) is the protocol that negotiates IPsec security associations and establishes keys before the encrypted session begins.

Email Security

Two primary standards protect email:

• S/MIME (Secure/Multipurpose Internet Mail Extensions): Uses X.509 certificates from a PKI for encryption and signing. The trust model is hierarchical, based on certificate authorities.
• PGP (Pretty Good Privacy) / OpenPGP: Uses a decentralized web of trust model where users sign each other's keys. No central authority is required.

Both use hybrid encryption: asymmetric cryptography protects the session key, and symmetric encryption protects the message body. The key difference is the trust model, not the cryptographic mechanisms.

SSH (Secure Shell)

SSH provides encrypted remote access and file transfer. It authenticates using passwords, public keys, or certificates. SSH key management (distributing, rotating, and revoking keys) is a common operational challenge.

VPN Architectures

• Site-to-site VPN: Connects two networks, typically using IPsec tunnel mode. Traffic between the sites is encrypted transparently.
• Remote access VPN: Connects individual users to a network, using IPsec, TLS-based VPN (like OpenVPN or SSL VPN), or WireGuard.
• Split tunneling: Only traffic destined for the corporate network goes through the VPN. Other traffic goes directly to the internet. This saves bandwidth but creates a risk: the user's device is simultaneously connected to the corporate network and the untrusted internet.
• Full tunneling: All traffic goes through the VPN. More secure but higher bandwidth cost and latency.

Blockchain Basics

Blockchain uses cryptographic hashing and consensus mechanisms to create immutable, distributed ledgers. Each block contains a hash of the previous block, making tampering evident. While primarily associated with cryptocurrency, the underlying concepts of hash chaining and distributed consensus are relevant to data integrity applications.

CISSP Lens

The exam tests IPsec components heavily. Know that AH provides integrity only (no encryption) and ESP provides both confidentiality and integrity. Know that tunnel mode is for site-to-site VPNs and transport mode is for host-to-host communication.

TLS questions test your understanding of the handshake process and why specific components matter (cipher suite negotiation, certificate verification, key exchange).

For email security, know that S/MIME uses PKI (certificates) while PGP uses a web of trust. The exam may describe a scenario and ask which trust model is in use.

Real-World Scenario

An organization deployed a remote access VPN with split tunneling enabled to reduce bandwidth costs. Remote workers connected to the corporate network for internal applications while accessing the internet directly for everything else.

An attacker compromised a remote worker's home network through an insecure IoT device. Because the worker's laptop was connected to both the corporate VPN and the compromised home network simultaneously, the attacker used the laptop as a bridge. They routed through the split tunnel connection into the corporate network, bypassing the perimeter firewall entirely.

Post-incident, the organization evaluated two options: switching to full tunneling (which would increase bandwidth costs) or keeping split tunneling with additional controls (endpoint security agents, network access control that verified device posture before VPN connection, and DNS-based filtering for non-VPN traffic). They chose enhanced split tunneling with mandatory endpoint protection, accepting the residual risk with documented compensating controls.

Common Mistakes and Misconceptions

• Confusing IPsec AH with ESP. AH provides integrity only, no encryption. If confidentiality is needed, ESP is required.
• Allowing TLS 1.0 or 1.1 for backward compatibility. Both versions have known vulnerabilities. TLS 1.2 should be the minimum, with TLS 1.3 preferred.
• Enabling split tunneling without additional endpoint controls. Split tunneling creates a bridge between the trusted and untrusted networks through the user's device.
• Assuming PGP and S/MIME are interchangeable. They use different trust models. S/MIME relies on certificate authorities; PGP relies on a web of trust.
• Disabling certificate verification in API integrations. "Ignore certificate errors" in code eliminates the authentication that TLS provides.

Actionable Checklist

• Audit TLS configurations for deprecated protocol versions and weak cipher suites
• Verify IPsec configurations use ESP (not just AH) where confidentiality is needed
• Evaluate VPN split tunneling policies and document the risk acceptance if split tunneling is used
• Confirm email encryption (S/MIME or PGP) is available for sensitive communications
• Test TLS implementations with tools like SSL Labs or testssl.sh
• Disable backward-compatible fallback to insecure protocol versions

Key Takeaways

• TLS, IPsec, and S/MIME are different assemblies of the same cryptographic primitives
• IPsec ESP provides confidentiality and integrity; AH provides integrity only
• TLS 1.3 removed insecure options; enforce it where possible
• Split tunneling trades security for performance; know the trade-off and compensate
• Protocol configuration matters as much as protocol choice

Exam-Style Reflection Question

An organization needs to encrypt all traffic between two office locations across the internet. Which IPsec mode is most appropriate?

Answer: Tunnel mode. It encrypts the entire original IP packet and encapsulates it in a new IP header, making it suitable for site-to-site VPN connections. Transport mode encrypts only the payload and is used for end-to-end communication between two hosts, not between networks.