A Visual Guide to Quantization - Demystifying the Compression of LLMs

A Visual Guide to Quantization Demystifying the Compression of Large Language Models Translations - Korean - Chinese - French As their name suggests, Large Language Models (LLMs) are often too large to run on consumer hardware. These models may exceed billions of parameters and generally need GPUs with large amounts of VRAM to speed up inference. As such, more and more research has been focused on making these models smaller through improved training, adapters, etc. One major technique in this field is called quantization. In this post, I will introduce the field of quantization in the context of language modeling and explore concepts one by one to develop an intuition about the field. We will explore various methodologies, use cases, and the principles behind quantization. In this visual guide, there are more than 50 custom visuals to help you develop an intuition about quantization! To see more visualizations related to LLMs and to support this newsletter, check out the book I wrote on Large Language Models! P.S. If you read the book, a quick review would mean the world-it really helps us authors! Part 1: The “Problem“ with LLMs LLMs get their name due to the number of parameters they contain. Nowadays, these models typically have billions of parameters (mostly weights) which can be quite expensive to store. During inference, activations are created as a product of the input and the weights, which similarly can be quite large. As a result, we would like to represent billions of values as efficiently as possible, minimizing the amount of space we need to store a given value. Let’s start from the beginning and explore how numerical values are represented in the first place before optimizing them. How to Represent Numerical Values A given value is often represented as a floating point number (or floats in computer science): a positive or negative number with a decimal point. These values are represented by “bits”, or binary digits. The IEEE-754 standard describes how bits can represent one of three functions to represent the value: the sign, exponent, or fraction (or mantissa). Together, these three aspects can be used to calculate a value given a certain set of bit values: The more bits we use to represent a value, the more precise it generally is: Memory Constraints The more bits we have available, the larger the range of values that can be represented. The interval of representable numbers a given representation can take is called the dynamic range whereas the distance between two neighboring values is called precision. A nifty feature of these bits is that we can calculate how much memory your device needs to store a given value. Since there are 8 bits in a byte of memory, we can create a basic formula for most forms of floating point representation. Now let’s assume that we have a model with 70 billion parameters. Most models are natively represented with float 32-bit (often called full-precision), which would require 280GB of memory just to load the model. As such, it is very compelling to minimize the number of bits to represent the parameters of your model (as well as during training!). However, as the precision decreases the accuracy of the models generally does as well. We want to reduce the number of bits representing values while maintaining accuracy… This is where quantization comes in! Part 2: Introduction to Quantization Quantization aims to reduce the precision of a model’s parameter from higher bit-widths (like 32-bit floating point) to lower bit-widths (like 8-bit integers). There is often some loss of precision (granularity) when reducing the number of bits to represent the original parameters. To illustrate this effect, we can take any image and use only 8 colors to represent it: Notice how the zoomed-in part seems more “grainy” than the original since we can use fewer colors to represent it. The main goal of quantization is to reduce the number of bits (colors) needed to represent the original parameters while preserving the precision of the original parameters as best as possible. Common Data Types First, let’s look at common data types and the impact of using them rather than 32-bit (called full-precision or FP32) representations. FP16 Let’s look at an example of going from 32-bit to 16-bit (called half precision or FP16) floating point: Notice how the range of values FP16 can take is quite a bit smaller than FP32. BF16 To get a similar range of values as the original FP32, bfloat 16 was introduced as a type of “truncated FP32”: BF16 uses the same amount of bits as FP16 but can take a wider range of values and is often used in deep learning applications. INT8 When we reduce the number of bits even further, we approach the realm of integer-based representations rather than floating-point representations. To illustrate, going FP32 to INT8, which has only 8 bits, results in a fourth of the original number of bits: For each reduction in bits, a mapping is performed to “squeeze” the initial FP32 representations into lower bits. In practice, we do not need to map the entire FP32 range [-3.4e38, 3.4e38] into INT8. We merely need to find a way to map the range of our data (the model’s parameters) into INT8. Common squeezing/mapping methods are symmetric and asymmetric quantization and are forms of linear mapping. Let’s explore these methods to quantize from FP32 to INT8. Symmetric Quantization In symmetric quantization, the range of the original floating-point values is mapped to a symmetric range around zero in the quantized space. In the previous examples, notice how the ranges before and after quantization remain centered around zero. This means that the quantized value for zero in the floating-point space is exactly zero in the quantized space. A nice example of a form of symmetric quantization is called absolute maximum (absmax) quantization. Given a list of values, we take the highest absolute value (α) as the range to perform the linear mapping. Since it is a linear mapping centered around zero, the formula is straightforward. We first calculate a scale factor (s) using: b is the number of bytes that we want to quantize to (8), α is the highest absolute value, Then, we use the s to quantize the input x: Filling in the values would then give us the following: To retrieve the original FP32 values, we can use the previously calculated scaling factor (s) to dequantize the quantized values. Applying the quantization and then dequantization process to retrieve the original looks as follows: You can see certain values, such as 3.08 and 3.02 being assigned to the INT8, namely 36. When you dequantize the values to return to FP32, they lose some precision and are not distinguishable anymore. This is often referred to as the quantization error which we can calculate by finding the difference between the original and dequantized values. Generally, the lower the number of bits, the more quantization error we tend to have. Asymmetric Quantization Asymmetric quantization, in contrast, is not symmetric around zero. Instead, it maps the minimum (β) and maximum (α) values from the float range to the minimum and maximum values of the quantized range. The method we are going to explore is called zero-point quantization. Notice how the 0 has shifted positions? That’s why it’s called asymmetric quantization. The min/max values have different distances to 0 in the range [-7.59, 10.8]. Due to its shifted position, we have to calculate the zero-point for the INT8 range to perform the linear mapping. As before, we also have to calculate a scale factor (s) but use the difference of INT8’s range instead [-128, 127] Notice how this is a bit more involved due to the need to calculate the zeropoint (z) in the INT8 range to shift the weights. As before, let’s fill in the formula: To dequantize the quantized from INT8 back to FP32, we will need to use the previously calculated scale factor (s) and zeropoint (z). Other than that, dequantization is straightforward: When we put symmetric and asymmetric quantization side-by-side, we can quickly see the difference between methods: Note the zero-centered nature of symmetric quantization versus the offset of asymmetric quantization. Range Mapping and Clipping In our previous examples, we explored how the range of values in a given vector could be mapped to a lower-bit representation. Although this allows for the full range of vector values to be mapped, it comes with a major downside, namely outliers. Imagine that you have a vector with the following values: Note how one value is much larger than all others and could be considered an outlier. If we were to map the full range of this vector, all small values would get mapped to the same lower-bit representation and lose their differentiating factor: Instead, we can choose to clip certain values. Clipping involves setting a different dynamic range of the original values such that all outliers get the same value. In the example below, if we were to manually set the dynamic range to [-5, 5] all values outside that will either be mapped to -127 or to 127 regardless of their value: The major advantage is that the quantization error of the non-outliers is reduced significantly. However, the quantization error of outliers increases. Calibration In the example, I showed a naive method of choosing an arbitrary range of [-5, 5]. The process of selecting this range is known as calibration which aims to find a range that includes as many values as possible while minimizing the quantization error. Performing this calibration step is not equal for all types of parameters. Weights (and Biases) We can view the weights and biases of an LLM as static values since they are known before running the model. For instance, the ~20GB file of Llama 3 consists mostly of its weight and biases. Since there are significantly fewer biases (millions) than weights (billions), the biases are often kept in higher precision (such as INT16), and the main effort of