Understanding Large Language Model Parameters and Memory Requirements: A Deep Dive

Calculating the Variety of Parameters

Pretrained Fashions For Environment friendly Transformer Coaching

Calculating Parameters in Transformer-based LLMs

Let’s break down the parameter calculation for every element of a Transformer-based LLM. We’ll use the notation from the unique paper, the place d_model represents the dimension of the mannequin’s hidden states.

1. Embedding Layer:
   • Parameters = vocab_size * d_model
2. Multi-Head Consideration:
   • For h heads, with d_k = d_v = d_model / h:
   • Parameters = 4 * d_model^2 (for Q, Ok, V, and output projections)
3. Feed-Ahead Community:
   • Parameters = 2 * d_model * d_ff + d_model + d_ff
   • The place d_ff is often 4 * d_model
4. Layer Normalization:
   • Parameters = 2 * d_model (for scale and bias)

Complete parameters for one Transformer layer:

• Parameters_layer = Parameters_attention + Parameters_ffn + 2 * Parameters_layernorm

For a mannequin with N layers:

• Complete Parameters = N * Parameters_layer + Parameters_embedding + Parameters_output

Instance Calculation

Let’s think about a mannequin with the next specs:

• d_model = 768
• h (variety of consideration heads) = 12
• N (variety of layers) = 12
• vocab_size = 50,000

1. Embedding Layer:
   • 50,000 * 768 = 38,400,000
2. Multi-Head Consideration:
3. Feed-Ahead Community:
   • 2 * 768 * (4 * 768) + 768 + (4 * 768) = 4,719,616
4. Layer Normalization:

Complete parameters per layer:

• 2,359,296 + 4,719,616 + (2 * 1,536) = 7,081,984

Complete parameters for 12 layers:

• 12 * 7,081,984 = 84,983,808

Complete mannequin parameters:

• 84,983,808 + 38,400,000 = 123,383,808

This mannequin would have roughly 123 million parameters.

Sorts of Reminiscence Utilization

When working with LLMs, we have to think about two most important forms of reminiscence utilization:

1. Mannequin Reminiscence: The reminiscence required to retailer the mannequin parameters.
2. Working Reminiscence: The reminiscence wanted throughout inference or coaching to retailer intermediate activations, gradients, and optimizer states.

Calculating Mannequin Reminiscence

The mannequin reminiscence is instantly associated to the variety of parameters. Every parameter is often saved as a 32-bit floating-point quantity, though some fashions use mixed-precision coaching with 16-bit floats.

Mannequin Reminiscence (bytes) = Variety of parameters * Bytes per parameter

For our instance mannequin with 123 million parameters:

• Mannequin Reminiscence (32-bit) = 123,383,808 * 4 bytes = 493,535,232 bytes ≈ 494 MB
• Mannequin Reminiscence (16-bit) = 123,383,808 * 2 bytes = 246,767,616 bytes ≈ 247 MB

Estimating Working Reminiscence

Working reminiscence necessities can differ considerably primarily based on the precise process, batch measurement, and sequence size. A tough estimate for working reminiscence throughout inference is:

Working Reminiscence ≈ 2 * Mannequin Reminiscence

This accounts for storing each the mannequin parameters and the intermediate activations. Throughout coaching, the reminiscence necessities will be even increased because of the have to retailer gradients and optimizer states:

Coaching Reminiscence ≈ 4 * Mannequin Reminiscence

For our instance mannequin:

• Inference Working Reminiscence ≈ 2 * 494 MB = 988 MB ≈ 1 GB
• Coaching Reminiscence ≈ 4 * 494 MB = 1,976 MB ≈ 2 GB

Regular-State Reminiscence Utilization and Peak Reminiscence Utilization

When coaching massive language fashions primarily based on the Transformer structure, understanding reminiscence utilization is essential for environment friendly useful resource allocation. Let’s break down the reminiscence necessities into two most important classes: steady-state reminiscence utilization and peak reminiscence utilization.

Regular-State Reminiscence Utilization

The steady-state reminiscence utilization includes the next parts:

1. Mannequin Weights: FP32 copies of the mannequin parameters, requiring 4N bytes, the place N is the variety of parameters.
2. Optimizer States: For the Adam optimizer, this requires 8N bytes (2 states per parameter).
3. Gradients: FP32 copies of the gradients, requiring 4N bytes.
4. Enter Information: Assuming int64 inputs, this requires 8BD bytes, the place B is the batch measurement and D is the enter dimension.

The full steady-state reminiscence utilization will be approximated by:

• M_steady = 16N + 8BD bytes

Peak Reminiscence Utilization

Peak reminiscence utilization happens in the course of the backward cross when activations are saved for gradient computation. The principle contributors to peak reminiscence are:

1. Layer Normalization: Requires 4E bytes per layer norm, the place E = BSH (B: batch measurement, S: sequence size, H: hidden measurement).
2. Consideration Block:
   • QKV computation: 2E bytes
   • Consideration matrix: 4BSS bytes (S: sequence size)
   • Consideration output: 2E bytes
3. Feed-Ahead Block:
   • First linear layer: 2E bytes
   • GELU activation: 8E bytes
   • Second linear layer: 2E bytes
4. Cross-Entropy Loss:
   • Logits: 6BSV bytes (V: vocabulary measurement)

The full activation reminiscence will be estimated as:

• M_act = L * (14E + 4BSS) + 6BSV bytes

The place L is the variety of transformer layers.

Complete Peak Reminiscence Utilization

The height reminiscence utilization throughout coaching will be approximated by combining the steady-state reminiscence and activation reminiscence:

• M_peak = M_steady + M_act + 4BSV bytes

The extra 4BSV time period accounts for an additional allocation at the beginning of the backward cross.

By understanding these parts, we are able to optimize reminiscence utilization throughout coaching and inference, making certain environment friendly useful resource allocation and improved efficiency of enormous language fashions.