Tag Archives: Logging

Linux

Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

Leave a comment

How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

Introduction

IP Geolocation, the process used to determine the physical location of an IP address, can be leveraged for a variety of purposes, such as content personalization and traffic analysis. Traffic analysis by geolocation can provide invaluable insight into your user base as it allows you to easily see where they users are coming from, which can help you make informed decisions about the ideal geographical location(s) of your application servers and who your current audience is. In this tutorial, we will show you how to create a visual geo-mapping of the IP addresses of your application’s users, by using a GeoIP database with Elasticsearch, Logstash, and Kibana.

Here’s a short explanation of how it all works. Logstash uses a GeoIP database to convert IP addresses into latitude and longitude coordinate pair, i.e. the approximate physical location of an IP address. The coordinate data is stored in Elasticsearch in geo_point fields, and also converted into a geohash string. Kibana can then read the Geohash strings and draw them as points on a map of earth, known in Kibana 4 as a Tile Map visualization.

Let’s take a look at the prerequisites now.

Prerequisites

To follow this tutorial, you must have a working ELK stack. Additionally, you must have logs that contain IP addresses that can be filtered into a field, like web server access logs. If you don’t already have these two things, you can follow the first two tutorials in this series. The first tutorial will set up an ELK stack, and second one will show you how to gather and filter Nginx or Apache access logs:

Download Latest GeoIP Database

MaxMind provides free and paid GeoIP databases—the paid versions are more accurate. Logstash also ships with a copy of the free GeoIP City database, GeoLite City. In this tutorial, we will download the latest GeoLite City database, but feel free to use a different GeoIP database if you wish. Continue reading Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana) →

Linux

How To Use Kibana Dashboards and Visualizations

Leave a comment

How To Use Kibana Dashboards and Visualizations

Introduction

Kibana 4 is an analytics and visualization platform that builds on Elasticsearch to give you a better understanding of your data. In this tutorial, we will get you started with Kibana, by showing you how to use its interface to filter and visualize log messages gathered by an Elasticsearch ELK stack. We will cover the main interface components, and demonstrate how to create searches, visualizations, and dashboards.

Prerequisites

This tutorial is the third part in the Centralized Logging with Logstash and Kibana series.

It assumes that you have a working ELK setup. The examples assume that you are gathering syslog and Nginx access logs. If you are not gathering these types of logs, you should be able to modify the demonstrations to work with your own log messages.

If you want to follow this tutorial exactly as presented, you should have the following setup, by following the first two tutorials in this series:

When you are ready to move on, let’s look at an overview of the Kibana interface.

Kibana Interface Overview

The Kibana interface is divided into four main sections:

  • Discover
  • Visualize
  • Dashboard
  • Settings

We will go over the basics of each section, in the listed order, and demonstrate how each piece of the interface can be used. Continue reading How To Use Kibana Dashboards and Visualizations →

Linux

Adding Logstash Filters To Improve Centralized Logging

Leave a comment

Adding Logstash Filters To Improve Centralized Logging

Introduction

Logstash is a powerful tool for centralizing and analyzing logs, which can help to provide and overview of your environment, and to identify issues with your servers. One way to increase the effectiveness of your Logstash setup is to collect important application logs and structure the log data by employing filters, so the data can be readily analyzed and query-able. We will build our filters around “grok” patterns, that will parse the data in the logs into useful bits of information.

This guide is a sequel to the How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04 tutorial, and focuses primarily on adding filters for various common application logs.

Prerequisites

To follow this tutorial, you must have a working Logstash server, and a way to ship your logs to Logstash. If you do not have Logstash set up, here is another tutorial that will get you started: How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04.

Logstash Server Assumptions:

  • Logstash is installed in /opt/logstash
  • You are receiving logs from Logstash Forwarder on port 5000
  • Your Logstash configuration files are located in /etc/logstash/conf.d
  • You have an input file named 01-lumberjack-input.conf
  • You have an output file named 30-lumberjack-output.conf

Continue reading Adding Logstash Filters To Improve Centralized Logging →

Linux

Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04

Leave a comment

How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04

Introduction

In this tutorial, we will go over the installation of the Elasticsearch ELK Stack on Ubuntu 14.04—that is, Elasticsearch 1.4.4, Logstash 1.5.0, and Kibana 4. We will also show you how to configure it to gather and visualize the syslogs of your systems in a centralized location. Logstash is an open source tool for collecting, parsing, and storing logs for future use. Kibana 4 is a web interface that can be used to search and view the logs that Logstash has indexed. Both of these tools are based on Elasticsearch.

Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It is also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

It is possible to use Logstash to gather logs of all types, but we will limit the scope of this tutorial to syslog gathering.

Our Goal

The goal of the tutorial is to set up Logstash to gather syslogs of multiple servers, and set up Kibana to visualize the gathered logs.
Continue reading Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04 →

Linux

Building for Production: Web Applications — Recovery Planning

Leave a comment

Building for Production: Web Applications — Recovery Planning

Introduction

Now that we have our example application setup, we should devise a recovery plan. A recovery plan is a set of documented procedures to recover from potential failures or administration errors within your server setup. Creating a recovery plan will also help you identify the essential components and data of your application server setup.

A very basic recovery plan for a server failure could consist of the list of steps that you took to perform your initial server deployment, with extra procedures for restoring application data from backups. A better recovery plan might, in addition to good documentation, leverage deployment scripts and configuration management tools, such as Ansible, Chef, or Puppet, to help automate and quicken the recovery process.

Recovery Plan Diagram

In this part of the tutorial, we will demonstrate how to create a basic recovery plan for the example WordPress application that we set up. Your needs will probably differ, but this should help you get started with devising a recovery plan of your own.

Recovery Plan Requirements

Our basic requirements are that we are able to recover from the loss of any server in the setup, and restore the application functionality and data (up to a reasonable point in time). To fulfill this goal, we will create an inventory of each server, determine which data needs to be backed up, and write a recovery plan based on our available assets. Of course, if any of these recovery plans are executed, the application should be tested to verify that it was restored properly. Continue reading Building for Production: Web Applications — Recovery Planning →

Linux

Building for Production: Web Applications — Deploying

Leave a comment

Building for Production: Web Applications — Deploying
Introduction

In this part of the tutorial, we will deploy our example PHP application, WordPress, and a private DNS:

DNS + Application Diagram

Your users will access your application over HTTPS via a domain name, e.g. “https://www.example.com”, that points to the load balancer. The load balancer will act as a reverse proxy to the application servers, which will connect to the database server. The private DNS will enable us to use names to refer to the private network addresses of our servers which ease the process of configuration of our servers.

We will set up the components that we just discussed on six servers, in this order:

  • Private DNS (ns1 and ns2)
  • Database Server (db1)
  • Application Servers (app1 and app2)
  • Load Balancer (lb1)

Let’s get started with the DNS setup.

Private DNS Servers

Using names for addresses helps with identifying the servers you are working with and becomes essential for the maintenance of a larger server setup, as you can replace a server by simply updating your DNS records (in a single place) instead of updating countless configuration files with IP addresses. In our setup, we will set up our DNS so we can reference the private network addresses of our servers by name instead of IP address.

We will refer to the private network address of each server by a hostname under the “nyc3.example.com” subdomain. For example, the database server’s private network address would be “db1.nyc3.example.com”, which resolves to it’s private IP address. Note that the example subdomain is almost completely arbitrary, and is usually chosen based on logical organization purposes; in our case, we “nyc3” indicates that the servers are in the NYC3 datacenter, and “example.com” is our application’s domain name.

Set this up by following this tutorial, and adding DNS records for each server in your setup:

After completing the DNS tutorial, you should have two BIND servers: ns1 and ns2. If you already know the private IP addresses of all of the servers in your setup, add them to your DNS now; otherwise, add the appropriate DNS records as you create your servers.

Now we’re ready to set up our database server.

Set Up Database Server

Because we want to load balance the our application servers, i.e. the ones running Apache and PHP, we need to decouple the database from the application servers by setting it up on a separate server. Decoupling the database from the application is an essential step before horizontally scaling many types of applications, as explained in this blog post: Horizontally Scaling PHP Applications: A Practical Overview.
Continue reading Building for Production: Web Applications — Deploying →

Linux

Building for Production: Web Applications — Overview

Leave a comment

Building for Production: Web Applications — Overview

Introduction

This 6-part tutorial will show you how to build out a multi-server production application setup from scratch. The final setup will be supported by backups, monitoring, and centralized logging systems, which will help you ensure that you will be able to detect problems and recover from them. The ultimate goal of this series is to build on standalone system administration concepts, and introduce you to some of the practical considerations of creating a production server setup.

If you are interested in reviewing some of the concepts that will be covered in this series, read these tutorials:

While the linked articles provide general guidelines of a production application setup, this series will demonstrate how to plan and set up a sample application from start to finish. Hopefully, this will help you plan and implement your own production server environment, even if you are running a different application on a completely different technology stack. Because this tutorial covers many different system administration topics, it will often defer the detailed explanation to external supporting articles that provide supplemental information.

Our Goal

By the end of this set of tutorials, we will have a production server setup for a PHP application, WordPress for demonstration purposes, that is accessible via https://www.example.com/. We will also include servers that will support the production application servers. The final setup will look something like this (private DNS and remote backups not pictured):

Production Setup

In this setup, the servers in the Application box are considered to be essential for the application run properly. Aside from the recovery plan and the remote backup server, the remaining components—backups, monitoring, and logging—will be added to support the production application setup. Each component will be installed on a separate Ubuntu 14.04 server within the same DigitalOcean region, NYC3 in our example, with Private Networking enabled.

Continue reading Building for Production: Web Applications — Overview →