Introduction

We’re finally ready to set up centralized logging for our production application setup. Centralized logging is a great way to gather and visualize the logs of your servers. Generally, setting up an elaborate logging system is not as important as having solid backups and monitoring set up, but it can be very useful when trying to identify trends or problems with your application.

In this tutorial, we will set up an ELK stack (Elasticsearch, Logstash, and Kibana), and configure the servers that comprise our application to send their relevant logs to the logging server. We will also set up Logstash filters that will parse and structure our logs which will allow us to easily search and filter them, and use them in Kibana visualizations.

Prerequisites

If you want to access your logging dashboard via a domain name, create an A Record under your domain, like “logging.example.com”, that points to your logging server’s public IP address. Alternatively, you can access the monitoring dashboard via the public IP address. It is advisable that you set up the logging web server to use HTTPS, and limit access to it by placing it behind a VPN.

Install ELK on Logging Server

Set up ELK on your logging server by following this tutorial: How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04.

If you are using a private DNS for name resolution, be sure to follow Option 2 in the Generate SSL Certificates section.

Stop when you reach the Set Up Logstash Forwarder section.

Set Up Logstash Forwarder on Clients

Set up Logstash Forwarder, a log shipper, on your client servers, i.e. db1, app1, app2, and lb1, by following the Set Up Logstash Forwarder section, of the ELK tutorial.

When you are finished, you should be able to log into Kibana via the logging server’s public network address, and view the syslogs of each of your servers. Continue reading Building for Production: Web Applications — Centralized Logging →