Quite often we need to launch a small app on a simple VPS server. In those cases my approach for quick delivery may be a docker stack with one or multiple proxy containers.
Now the common issue with logging in above scenario is that:
- logs dont contain real ip addresses and would be filled with junk internal addresses instead
- it’s not easy to see if requests came from or some internal tooling (like haproxy health checks, load-testing script etc)
- wordpress installation may not automatically pick up correct ip address
For example for wordpress a typical stack may include:
- mysql 8.0 container (typical wordpess backend db)
- redis 5+ container (for db/object caching)
- apache/php/wordpress container
- haproxy (SSL/TLS termination)
- varnish (caching, on the fly minification and more)
So requests would flow as follow: browser -> haproxy-> varnish -> wordpress -> redis + mysql
With simplistic off the shelf setup apache logs would contain IP addresses of nearest proxy (varnish in this case), which makes logs, well just a lot “less useful”.
I will show you how to make sure your apache logs do indeed contain real ip addresses not disguised by proxies.
Here’s an example of haproxy configuration for above scenario:
global
# Settings under global define process-wide security and performance tunings that affect HAProxy at a low level.
# Max number of connections haproxy will accept
maxconn 1024
# Logging to stdout preferred when running as a container.
log stdout format raw local0
# Only TLS version 1.2 and newer is allowed:
ssl-default-bind-options ssl-min-ver TLSv1.2
defaults
# Defaults here
# As your configuration grows, using a defaults section will help reduce duplication.
# Its settings apply to all of the frontend and backend sections that come after it.
# You’re still free to override those settings within the sections that follow.
# this updates different proxies (frontend, backend, and listen sections) to send messages
# to the loggign mechanism/server(s) configured in the global section
log global
# Will enable more verbose HTTP logging
# Enable http logging format to incldue more details logs
option httplog
# Enable HTTP connection closing on the server side but support keep-alive with clients
# (This provides the lowest latency on the client side (slow network) and the fastest session reuse on the server side)
option http-server-close
# option httpclose
# Don't use httpclose and http-server-close, httpclose will disable keepalive on the client side
# Expect HTTP layer 7, rather than load-balance at layer 4
mode http
# A connection on which no data has been transferred will not be logged (such as monitor probes)
option dontlognull
# Various response timeouts
timeout connect 5s
timeout client 20s
timeout server 45s
frontend fe_wp_http
bind *:80
mode http
redirect scheme https code 301
frontend fe_wp_https
bind *:443 ssl crt /certs/bytepursuits.com/fullkeychain.pem alpn h2,http/1.1
mode http
default_backend be_wp
backend be_wp
mode http
balance roundrobin
# Enable insertion of the X-Forwarded-For header to requests sent to servers
option forwardfor
# Send these request to check health
option httpchk
http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy.local
# remove the Server header from responses
http-response del-header Server
server wp-backend1 bytepursuits-varnish:80 check
http-request set-header x-client-ip %[src]
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
The most important bit for our ip logging is this tidbit:
http-request set-header x-client-ip %[src]
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
Here we are adding several headers: x-client-ip, X-Forwarded-Port, X-Forwarded-Proto that we can then use in any subsequent proxies.
Our Next step would be updating our apache vhost file to let apache know to use this header:
#==========================================================================
# Logging behind the reverse proxy - start
#==========================================================================
#Determine if request came from web or from internal area based on a presence of the header.
SetEnvIfNoCase X-Client-Ip "(.+)" RQSRC=WEB
SetEnvIfNoCase X-Client-Ip "^$" RQSRC=INTERNAL
#Setting remote ip (RIP) value to real client web ip, or
#internal proxy addr if present.
SetEnvIfNoCase REMOTE_ADDR "(.+)" RIP=$1
SetEnvIfNoCase X-Client-Ip "(.+)" RIP=$1
# Defining custom acceess log format (error log can just stay same)
# We do it to make sure real ip is captured.
LogFormat "%{RQSRC}e %{RIP}e %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined2
LogLevel info
ErrorLog /dev/stderr
CustomLog /dev/stdout combined2
#==========================================================================
# Logging behind the reverse proxy - end
#==========================================================================
The end result would be a nice looking Apache log output that actually includes real ip addresses (note – in example below I’ve substituted real ip with 1.2.3.4 ). Note that all internal request are prefixed with INTERNAL and those that come from the web are prefixed with WEB. Internal requests can be various health checks from your proxies or results of any internal load testing etc.
bytepursuits-wp | INTERNAL 172.18.0.5 - - [25/Jan/2021:08:22:07 -0500] "GET / HTTP/1.1" 200 8602 "-" "-"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:22:47 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 14174 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:23:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:23:28 -0500] "POST /wp-json/wp/v2/posts/25?_locale=user HTTP/1.1" 200 16790 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:23:29 -0500] "POST /wp/wp-admin/post.php?post=25&action=edit&meta-box-loader=1&meta-box-loader-nonce=0d762b2f4e&_locale=user HTTP/1.1" 302 419 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:23:29 -0500] "GET /wp/wp-admin/post.php?post=25&action=edit&message=4 HTTP/1.1" 200 55797 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:23:47 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 14250 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:24:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | INTERNAL 172.18.0.5 - - [25/Jan/2021:08:24:07 -0500] "GET / HTTP/1.1" 200 8603 "-" "-"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:24:47 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 14816 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:25:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:25:48 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 15148 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:26:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | INTERNAL 172.18.0.5 - - [25/Jan/2021:08:26:07 -0500] "GET / HTTP/1.1" 200 8603 "-" "-"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:26:47 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 15396 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:27:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:27:48 -0500] "POST /wp-json/wp/v2/posts/25/autosaves?_locale=user HTTP/1.1" 200 18176 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:28:05 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
bytepursuits-wp | INTERNAL 172.18.0.5 - - [25/Jan/2021:08:28:08 -0500] "GET / HTTP/1.1" 200 8604 "-" "-"
bytepursuits-wp | WEB 1.2.3.4 - - [25/Jan/2021:08:29:16 -0500] "POST /wp/wp-admin/admin-ajax.php HTTP/1.1" 200 560 "https://bytepursuits.com/wp/wp-admin/post.php?post=25&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"