MODULE m7 · 8.0 HOURS

Hybrid TLS Handshake Analysis

0% complete

LEARNING OBJECTIVES

Module objectives

  1. Explain the vulnerabilities of TLS 1.3 in a quantum threat environment.
  2. Understand the ML-KEM-based hybrid TLS handshake structure.
  3. Measure the performance of hybrid mode in a local OQS (Open Quantum Safe)-based environment.
  4. Analyze the handshake process where classical algorithms and PQ algorithms are combined through packet capture.

Hybrid TLS Handshake Theory

Current TLS 1.3 relies on classical public-key cryptography such as ECDH (Elliptic Curve Diffie-Hellman), which is exposed to attack threats upon the emergence of sufficiently powerful quantum computers[S2]. In response, active research is underway to integrate Post-Quantum Cryptography (PQC) such as the NIST standard ML-KEM (Module-Lattice-Based Key-Encapsulation Mechanism)[S2].

Security Assumptions of Hybrid Mode

Before full PQC migration, ‘Hybrid mode’ combining classical algorithms and PQ algorithms is used for operational stability. This method combines the shared secrets of the two algorithms and is designed to guarantee the secrecy of communications even if only one of the two algorithms remains secure.

Performance Considerations

TLS 1.3 handshake performance is directly dependent on the computational cost of KEM[S2]. In particular, operation efficiency in the Key Generation, Encapsulation, and Decapsulation stages determines total handshake speed. Recently, techniques to increase handshake performance by accelerating polynomial multiplication and modular reduction processes through instruction set optimizations like AVX-512 are being applied[S2].

WORKED EXAMPLES

Worked examples

  1. Example 1: Hybrid key exchange process - The client transmits the classical shared secret value and the PQC encapsulation result value to the server. After decapsulation, the server combines the two shared secrets in the KDF (Key Derivation Function) to generate a session key.
  2. Example 2: Handshake latency analysis - Comparative analysis is performed on the phenomenon where network transmission time increases as key size increases when applying PQ algorithms. The difference in Handshakes per second before and after applying performance optimizations (e.g., using AVX-512) is quantified[S2].

LAB PROTOCOL

OQS-based Hybrid TLS Handshake Reproduction Experiment

  1. 1

    Build an OpenSSL 3 build environment with integrated liboqs and oqs-provider within an isolated Docker environment.

  2. 2

    Configure existing ECDH mode and ML-KEM-based hybrid mode (e.g., secp256r1_mlkem768).

  3. 3

    Execute TLS 1.3 handshake between the two modes using a local loopback interface.

  4. 4

    Capture key exchange data fields in ClientHello and ServerHello messages using Wireshark.

  5. 5

    Measure the average time of 100 or more handshakes using an execution time measurement script and record the overhead of hybrid mode.

Safety check
  • This experiment is performed only on a local isolated loopback interface and generates no external network traffic.
  • Never use operational security assets such as actual certificates, private keys, or user data.
  • Note that learning-purpose OQS libraries are experimental tools and not to be used for actual operational security services.
  • Delete all generated temporary keys and local measurement data upon experiment conclusion.

Lab deliverables

  • Configuration and execution log files.
  • Wireshark packet capture file (.pcapng) and key exchange field analysis notes.
  • Handshake performance comparison measurement CSV data.

ASSIGNMENT

TLS 1.3 Hybrid Migration Performance Analysis Report

Deliverables

Rubric

  • Reproducibility of the experiment (whether environment setup and scripts are specified).
  • Accuracy of packet analysis (identification of key exchange fields).
  • Logical interpretation of performance measurement data.
  • Connectivity between theoretical security assumptions and experimental results.

KNOWLEDGE CHECK

Knowledge check

1What is the primary security advantage of a hybrid TLS handshake?
2What is the PQ algorithm element that has the greatest impact on TLS 1.3 handshake performance?

FIELD CHECK

Completion criteria

  • Successful reproduction of liboqs-based hybrid TLS handshake in a local environment.
  • Verification of hybrid key exchange messages through Wireshark analysis.
  • Submission of a result analysis report based on performance measurement data.
  • Confirmation of adherence to safety constraints (data isolation, no use of production keys).

MODULE SOURCES

Module sources

  1. Faster Post-Quantum TLS 1.3 Based on ML-KEM: Implementation ...arxiv.org · paper