- What is Compression?
- What assets should I compress?
- What is Gzip/deflate compression
- What is Brotli compression
- Should I support deflate and gzip in nginx if I use brotli ?
- Install and enable nginx brotli and gzip modules
- Do not re-compress already compressed brotli and gzip files
- Nginx – configure dynamic brotli content compression by content type (with gzip fallback)
- Testing nginx brotli and gzip compression with curl
What is Compression?
Compression is a technique used to reduce the data size as reducing the data size means that data is smaller and smaller data will travel over the internet faster improving site response times. Most of the websites/web-servers these days should use compression – and it is truly mind-boggling that according to the WebAlmanac 2021 study – 57% of websites didn’t use it as of 2021.
Don’t be part of those 57% of websites – compress your responses. Your website would likely suffer otherwise to some extent in search engine rankings given how the CWV rankings are becoming more and more important for SEO.
Browsers these days typically support these 3 compression mechanisms (technically there are more but practically it’s only these):
- Brotli
- Gzip
- Deflate
Most browsers support all three of these however and typically when your browser sends a request to the website – it needs to indicate which format this browser supports using Accept-Encoding request header.
For example, see this curl command:
curl -I -H 'Accept-Encoding: br,gzip,deflate' https://bytepursuits.com
HTTP/2 200
date: Wed, 19 Oct 2022 09:31:09 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-ua-compatible: IE=edge
link: <https://bytepursuits.com/wp-json/>; rel="https://api.w.org/"
last-modified: Wed, 19 Oct 2022 09:31:09 GMT
pragma: public
cache-control: max-age=86400, public
etag: W/"01ece1313e374b882395f4d896c95b4c"
referrer-policy: no-referrer-when-downgrade
content-encoding: br
x-varnish: 3278630 1378308
age: 2122
x-vcache: HIT
accept-ranges: bytes
content-length: 13536
^ Note how the webserver sent back the “content-encoding: br” header to indicate which compression format server have chosen to respond with, in this case (br)otli.
Here in above curl example we are indicating to the server that we (user agent like “curl cli command” or real browser like firefox) do support all three mechanisms: br,gzip,deflate.
Accept-Encoding header also accepts optional quality values that can range from 0 to 1, for example:
// Multiple algorithms, weighted with the quality value syntax:
Accept-Encoding: br;q=1.0,deflate, gzip;q=0.8, *;q=0.5
^ above means that brotli is preferred more then gzip and gzip is preferred over deflate.
Just like your browser – a website/server responding to a request containing an Accept-Encoding header may or may not support all of the encodings the browser supports (listed in request Accept-Encoding header) or may not even support compression at all.
In a case where webserver/website does not support compression – webserver/website can ignore compression request and return uncompressed data. Most webservers and browsers do support compression these days and not compressing assets should be a very rare occurrence (but again – in real life we are seeing 57% of the webservers not using compression according to Web Almanac study).
What assets should I compress?
Typically it is recommended to only compress text-based formats, such as html, css, js and similar.
As a point of comparison – I like to refer people to aws cloudfront compression list that offers some insights into what formats AWS considers well-compressible (hint: binary formats like png, jpeg, gif, mp3, ogg are not on it).
You are very unlikely to benefit from compressing already compressed formats such as png, jpeg, gif, mp3 etc – that are already compressed using techniques that are best for those formats .
What is Gzip/deflate compression
Gzip is an older file format, written by Jean-loup Gailly and Mark Adler as far back as 1992.
Gzip is based on the DEFLATE algorithm, which is a combination of LZ77 and Huffman coding.
The decompression of the gzip format can be implemented as a streaming algorithm, an important feature for various Web protocols and integration with various servers like nginx.
HTTP Deflate – or Zlib (RFC 1950 – ZLIB Compressed Data Format Specification) and Gzip (RFC 1952 – GZIP file format specification) are both just wrappers for the Deflate (RFC 1951 – DEFLATE Compressed Data Format Specification) compression format. There was some naming confusion back in the olden days of the internet which led to incompatible implementation – which is why we have both of these now. However gzip naming was less confusing – which is probably why it became more prevalent on the internet.
What is Brotli compression
Brotli is a newer generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling.
Brotli is used by web servers/ web applications and CDN networks to compress HTTP content, making internet websites load faster. It is seen as a successor to gzip and it is supported by all major web browsers and has became increasingly popular, as it provides better compression than gzip.
There are caveats however – per the Squash Benchmark tests show that Brotli has a better compression ratio, but gzip does beat brotli in speed most of the time.
For our purposes we just should know that it can potentially offer significantly better compression ratios and thus smaller file-sizes. Smaller file-sizes then means faster websites.
According to caniuse most modern browsers do support brotli these days:
Should I support deflate and gzip in nginx if I use brotli ?
deflate is dead in my view. There is this November 2021 study from Web Almanac where a staggering 0.0% of servers returned a deflate-compressed response, it is a change from Web Almanac 2020 study where 0.02% of servers were supporting it. I would not bother supporting it on the server side.
What is even more staggering from the same study Web Almanac study – 57% of surveyed websites didn’t use compression at all, this is very wrong.
It should be noted that if server does not support compression – it is free to return back the uncompressed result.
Should I still use Gzip fallback in nginx if I use brotli? This is a little bit more nuanced – Brotli could have a better compression ratio compared to competing algorithms but could fall behind in compression and decompression times, so YMMV. Brotli adoption rates are trending up from 7% in 2019 to 14% in 2021 according to the same 2021 Web Almanac study, however, gzip adoption is largely staying flat and is trending down only very very slowly (from 29.5% in 2019 -> 30.8% in 2020 -> 28.2% in 2022). I am very curious about what stats we are going to see after 2022 is over.
Install and enable nginx brotli and gzip modules
Here’s a thing – gzip is a core module in nginx, so it will be already installed.
Brotli on the other hand is a third party module to nginx and would have to be either installed from paid “nginx plus” repository or compiled manually. Depending on how you install nginx – there are many tutorials online for installing brotli module and Im not going to go into installation weeds here.
Personally, I tend to use nginx containerized – you can see instructions for adding brotli into containerized nginx in nginxinc /docker-nginx github repo.
# Load Brotli module
load_module "modules/ngx_http_brotli_filter_module.so";
load_module "modules/ngx_http_brotli_static_module.so";
^ Once the module is installed you have to enable it, and add these lines into the server block.
Do not re-compress already compressed brotli and gzip files
It is a very good idea to pre-compress all of the static files that you intend to serve in brotli format anyways, like .css, .js, .svg etc. Some pretty good reasons are :
- There is no need to waste cpu cycles and processing time on re-compressing the same file for every single request over and over and over. You can immediately grab and serve the already present compressed file without spending any time on compression.
- brotli and gzip compression levels for the content being compressed by nginx dynamically per request are typically not maxed out (and should not be!) – to reduce overall response times. However, if you precompress your css ahead of time you can easily set the compression level to the maximum you want (11/11 instead of 6/11). However – don’t max out the brotli compression levels even for statics if this is for a typical website use case (but u could still increase from 4-6/11 to 7-8/11) – it has been shown in many benchmarks that decompression time might outweigh the benefit of smaller file size, see this and this. (thank you for reminding Reddit!)
As a developer – you can easily configure automatic brotli and gzip compression as part of the build – using webpack or gulp or similar build tools.
Here’s how you would configure nginx to use pre-compressed brotli .br files if found:
# Static Brotli:
# Browser accepts brotli, and matching pre-compressed file exists => rewrite to .br file
# For each file format set the correct mime type (otherwise brotli mime type is returned) and prevent Nginx for recompressing the files
set $extension "";
if ($http_accept_encoding ~ br) {
set $extension .br;
}
if (-f $request_filename$extension) {
rewrite (.*) $1$extension break;
}
location ~ /*.html.br$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type text/html;
add_header Content-Encoding br;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.css.br$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type text/css;
add_header Content-Encoding br;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.js.br$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type application/javascript;
add_header Content-Encoding br;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.svg.br$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type image/svg+xml;
add_header Content-Encoding br;
add_header Vary "Accept-Encoding";
expires max;
}
These are basically 4 repeating blocks for js, css, html, svg that do the same thing, to clarify further – in each block we are:
- turning off an additional gzip and brotli processing (gzip off; brotli off;) as we don’t want to re-compress already compressed files.
- resetting type to a specific format, ex: application/javascript
- adding Content-Encoding header to indicate to a browser an expected compression format
- adding Vary response header – this header describes the parts of the request message aside from the method and URL that influenced the content of the response it occurs in. This header is extremely important for caching proxies – don’t forget to add it.
- adding expires header – to indicate that this is a long lasting content and that caches could keep it for a long time. Adjust this for your use case.
After this you should add the same for gzip static pre-compressed content:
# Static gzip:
# Browser accepts gzip, and matching pre-compressed file exists => rewrite to .gzip file
# For each file format set the correct mime type (otherwise gzip mime type is returned) and prevent Nginx for recompressing the files
set $extension "";
if ($http_accept_encoding ~ gzip) {
set $extension .gzip;
}
if (-f $request_filename$extension) {
rewrite (.*) $1$extension break;
}
location ~ /*.html.gzip$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type text/html;
add_header Content-Encoding gzip;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.css.gzip$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type text/css;
add_header Content-Encoding gzip;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.js.gzip$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type application/javascript;
add_header Content-Encoding gzip;
add_header Vary "Accept-Encoding";
expires max;
}
location ~ /*.svg.gzip$ {
gzip off;
brotli off;
#Clearing types
types {}
default_type image/svg+xml;
add_header Content-Encoding gzip;
add_header Vary "Accept-Encoding";
expires max;
}
As an alternative to above nginx configs, there are also these nginx instructions you could try instead:
brotli_static on;
gzip_static on;
Not all data could be pre-compressed as static files. Sure – if you have some SSG site (static site) setup – it is quite possible to pre-compress near everything including html.
But what if the data your site is serving is highly dynamic, like what if your site is some database (mongo/redis/elasticsearch/postgres) driven site where site’s html is being updated with every request – but you still want to benefit from those quick compressed transfer times. This is where you need to configure dynamic brotli content compression by content type. See below.
Nginx – configure dynamic brotli content compression by content type (with gzip fallback)
After static pre-compressed content is dealt with, we should address dynamic content and anything else that is not pre-compressed but could be compressed effectively.
This nginx snippet should be added right after brotli and gzip snippets dealing with static content.
Here – we are only compressing on the fly the most typical content types: css, js, html, svg (svg is text/xml based format so is compressible) that was not already compressed -> your application might have more content types so adjust list as needed.
In my tests – ordering of below brotli && gzip blocks didn’t matter (in nginx/1.22.1 I was using) – brotli would still be preferred compression (as it should).
# Dynamic gzip:
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
#text/html is always included by default, no need to include explicitely
gzip_types text/plain text/xml text/css
application/x-javascript application/javascript application/ecmascript text/javascript application/json
application/rss+xml
application/xml
image/svg+xml
application/x-font-ttf application/vnd.ms-fontobject image/x-icon;
# Dynamic brotli:
brotli on;
brotli_comp_level 6;
#text/html is always included by default, no need to include explicitely
brotli_types text/plain text/xml text/css
application/x-javascript application/javascript application/ecmascript text/javascript application/json
application/rss+xml
application/xml
image/svg+xml
application/x-font-ttf application/vnd.ms-fontobject image/x-icon;
Testing nginx brotli and gzip compression with curl
Now that we have nginx configured to serve assets brotli compressed with gzip fallback – let us see if we can test it really works. You can easily test it with curl command and passing of Accept-Encoding header:
a. Requesting br,gzip,deflate -> getting back brotli as expected:
[linuxdev@hs-dev1 ~]$ curl -I -H 'Accept-Encoding: br,gzip,deflate' 'https://bytepursuits.com'
HTTP/2 200
date: Thu, 20 Oct 2022 10:13:30 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-ua-compatible: IE=edge
link: <https://bytepursuits.com/wp-json/>; rel="https://api.w.org/"
last-modified: Thu, 20 Oct 2022 10:13:30 GMT
pragma: public
cache-control: max-age=86400, public
etag: W/"48eb8abf348c48e379bb118d108ab143"
referrer-policy: no-referrer-when-downgrade
content-encoding: br
x-varnish: 3384916 3384736
age: 219
x-vcache: HIT
accept-ranges: bytes
content-length: 13545
b. Requesting only gzip and deflate – gzip compressed content is returned as expected:
[linuxdev@hs-dev1 ~]$ curl -I -H 'Accept-Encoding: gzip,deflate' 'https://bytepursuits.com'
HTTP/2 200
date: Wed, 19 Oct 2022 12:04:37 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-ua-compatible: IE=edge
link: https://bytepursuits.com/wp-json/; rel="https://api.w.org/"
last-modified: Wed, 19 Oct 2022 12:04:37 GMT
pragma: public
cache-control: max-age=86400, public
etag: W/"0179197a39ebae7d153001d19edc45a9"
referrer-policy: no-referrer-when-downgrade
content-encoding: gzip
x-varnish: 3385729 3182152
age: 80948
x-vcache: HIT
accept-ranges: bytes
content-length: 14746
c. Requesting just deflate which we don’t have enabled – uncompressed content is returned as expected, not the lack of content-encoding header:
[linuxdev@hs-dev1 ~]$ curl -I -H 'Accept-Encoding: deflate' 'https://bytepursuits.com'
HTTP/2 200
date: Wed, 19 Oct 2022 11:59:54 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-ua-compatible: IE=edge
link: <https://bytepursuits.com/wp-json/>; rel="https://api.w.org/"
last-modified: Wed, 19 Oct 2022 11:59:54 GMT
pragma: public
cache-control: max-age=86400, public
etag: "e137b25fbce242d1aa806371c71b80db"
referrer-policy: no-referrer-when-downgrade
x-varnish: 3385731 2003396
age: 81234
x-vcache: HIT
accept-ranges: bytes
content-length: 81056
Our nginx config is working as expected compressing with brotli and gzip.
Thank you for reading!
1 thought on “Nginx – enabling brotli compression with gzip fallback”