Tag Archives: Monitoring

Linux

Install Graylog2 And Centralize Logs On Ubuntu 14.04

Leave a comment

Introduction

In this tutorial, we will cover the installation of Graylog2 (v0.20.2), and configure it to gather the syslogs of our systems in a centralized location. Graylog2 is a powerful log management and analysis tool that has many use cases, from monitoring SSH logins and unusual activity to debugging applications. It is based on Elasticsearch, Java, MongoDB, and Scala.

It is possible to use Graylog2 to gather and monitor a large variety of logs, but we will limit the scope of this tutorial to syslog gathering. Also, because we are demonstrating the basics of Graylog2, we will be installing all of the components on a single server.

About Graylog2 Components

Graylog2 has four main components:

  • Graylog2 Server nodes: Serves as a worker that receives and processes messages, and communicates with all other non-server components. Its performance is CPU dependent
  • Elasticsearch nodes: Stores all of the logs/messages. Its performance is RAM and disk I/O dependent
  • MongoDB: Stores metadata and does not experience much load
  • Web Interface: The user interface

Here is a diagram of the Graylog2 components (note that the messages are sent from your other servers):

Basic Graylog2 Setup

For a very basic setup, all of the components can be installed on the same server. For a larger, production setup, it would be wise to set up some high-availability features because if the server, Elasticsearch, or MongoDB components experiences an outage, Graylog2 will not gather the messages generated during the outage. Continue reading Install Graylog2 And Centralize Logs On Ubuntu 14.04 →

Linux

Install Bacula-web on Ubuntu 14.04

Leave a comment

How To Install Bacula-web on Ubuntu 14.04

Introduction

Bacula-web is a PHP web application that provides an easy way to view summaries and graphs of Bacula backup jobs that have already run. Although it doesn’t allow you to control Bacula in any way, Bacula-web provides a graphical alternative to viewing jobs from the console. Bacula-web is especially useful for users who are new to Bacula, as its reports make it easy to understand what Bacula has been operating.

In this tutorial, we will show you how to install Bacula-web on an Ubuntu 14.04 server that your Bacula server software is running on.

Prerequisites

To follow this tutorial, you must have the Bacula backup server software installed on an Ubuntu server. Instructions to install Bacula can be found here: How To Install Bacula Server on Ubuntu 14.04.

This tutorial assumes that your Bacula setup is using MySQL for the catalog. If you are using a different RDBMS, such as PostgreSQL, be sure to make the proper adjustments to this tutorial. You will need to install the appropriate PHP module(s) and make adjustments to the database connection information examples.

Let’s get started.

Install Nginx and PHP

Bacula-web is a PHP application, so we need to install PHP and a web server. We’ll use Nginx. If you want to learn more about this particular software setup, check out this LEMP tutorial.

Update your apt-get listings:

sudo apt-get update

Then, install Nginx, PHP-fpm, and a few other packages with apt-get:

sudo apt-get install nginx apache2-utils php5-fpm php5-mysql php5-gd

Now we are ready to configure PHP and Nginx.

Configure PHP-FPM

Open the PHP-FPM configuration file in your favorite text editor. We’ll use vi:

sudo vi /etc/php5/fpm/php.ini

Find the line that specifies cgi.fix_pathinfo, uncomment it, and replace its value with 0. It should look like this when you’re done.

cgi.fix_pathinfo=0

Now find the date.timezone setting, uncomment it, and replace its value with your time zone. We’re in New York, so that’s what we’re setting the value to: Continue reading Install Bacula-web on Ubuntu 14.04 →

Linux

Building for Production: Web Applications — Recovery Planning

Leave a comment

Building for Production: Web Applications — Recovery Planning

Introduction

Now that we have our example application setup, we should devise a recovery plan. A recovery plan is a set of documented procedures to recover from potential failures or administration errors within your server setup. Creating a recovery plan will also help you identify the essential components and data of your application server setup.

A very basic recovery plan for a server failure could consist of the list of steps that you took to perform your initial server deployment, with extra procedures for restoring application data from backups. A better recovery plan might, in addition to good documentation, leverage deployment scripts and configuration management tools, such as Ansible, Chef, or Puppet, to help automate and quicken the recovery process.

Recovery Plan Diagram

In this part of the tutorial, we will demonstrate how to create a basic recovery plan for the example WordPress application that we set up. Your needs will probably differ, but this should help you get started with devising a recovery plan of your own.

Recovery Plan Requirements

Our basic requirements are that we are able to recover from the loss of any server in the setup, and restore the application functionality and data (up to a reasonable point in time). To fulfill this goal, we will create an inventory of each server, determine which data needs to be backed up, and write a recovery plan based on our available assets. Of course, if any of these recovery plans are executed, the application should be tested to verify that it was restored properly. Continue reading Building for Production: Web Applications — Recovery Planning →

Linux

Building for Production: Web Applications — Deploying

Leave a comment

Building for Production: Web Applications — Deploying
Introduction

In this part of the tutorial, we will deploy our example PHP application, WordPress, and a private DNS:

DNS + Application Diagram

Your users will access your application over HTTPS via a domain name, e.g. “https://www.example.com”, that points to the load balancer. The load balancer will act as a reverse proxy to the application servers, which will connect to the database server. The private DNS will enable us to use names to refer to the private network addresses of our servers which ease the process of configuration of our servers.

We will set up the components that we just discussed on six servers, in this order:

  • Private DNS (ns1 and ns2)
  • Database Server (db1)
  • Application Servers (app1 and app2)
  • Load Balancer (lb1)

Let’s get started with the DNS setup.

Private DNS Servers

Using names for addresses helps with identifying the servers you are working with and becomes essential for the maintenance of a larger server setup, as you can replace a server by simply updating your DNS records (in a single place) instead of updating countless configuration files with IP addresses. In our setup, we will set up our DNS so we can reference the private network addresses of our servers by name instead of IP address.

We will refer to the private network address of each server by a hostname under the “nyc3.example.com” subdomain. For example, the database server’s private network address would be “db1.nyc3.example.com”, which resolves to it’s private IP address. Note that the example subdomain is almost completely arbitrary, and is usually chosen based on logical organization purposes; in our case, we “nyc3” indicates that the servers are in the NYC3 datacenter, and “example.com” is our application’s domain name.

Set this up by following this tutorial, and adding DNS records for each server in your setup:

After completing the DNS tutorial, you should have two BIND servers: ns1 and ns2. If you already know the private IP addresses of all of the servers in your setup, add them to your DNS now; otherwise, add the appropriate DNS records as you create your servers.

Now we’re ready to set up our database server.

Set Up Database Server

Because we want to load balance the our application servers, i.e. the ones running Apache and PHP, we need to decouple the database from the application servers by setting it up on a separate server. Decoupling the database from the application is an essential step before horizontally scaling many types of applications, as explained in this blog post: Horizontally Scaling PHP Applications: A Practical Overview.
Continue reading Building for Production: Web Applications — Deploying →

Linux

Building for Production: Web Applications — Overview

Leave a comment

Building for Production: Web Applications — Overview

Introduction

This 6-part tutorial will show you how to build out a multi-server production application setup from scratch. The final setup will be supported by backups, monitoring, and centralized logging systems, which will help you ensure that you will be able to detect problems and recover from them. The ultimate goal of this series is to build on standalone system administration concepts, and introduce you to some of the practical considerations of creating a production server setup.

If you are interested in reviewing some of the concepts that will be covered in this series, read these tutorials:

While the linked articles provide general guidelines of a production application setup, this series will demonstrate how to plan and set up a sample application from start to finish. Hopefully, this will help you plan and implement your own production server environment, even if you are running a different application on a completely different technology stack. Because this tutorial covers many different system administration topics, it will often defer the detailed explanation to external supporting articles that provide supplemental information.

Our Goal

By the end of this set of tutorials, we will have a production server setup for a PHP application, WordPress for demonstration purposes, that is accessible via https://www.example.com/. We will also include servers that will support the production application servers. The final setup will look something like this (private DNS and remote backups not pictured):

Production Setup

In this setup, the servers in the Application box are considered to be essential for the application run properly. Aside from the recovery plan and the remote backup server, the remaining components—backups, monitoring, and logging—will be added to support the production application setup. Each component will be installed on a separate Ubuntu 14.04 server within the same DigitalOcean region, NYC3 in our example, with Private Networking enabled.

Continue reading Building for Production: Web Applications — Overview →