Cloudlytics supports following types of log processing for System and Application log, which will help you to monitor your systems and other important metrics.
This document provides detailed information on how to configure your systems for each type of log in order to process your logs and monitor them through Cloudlytics.
It might be possible that you have to make some additional changes with respect to OS, but thankfully they are not so difficult.

  • System logs
    • General system logs
    • SSH logs
  • Application logs
    • Apache access logs
    • Apache error logs
    • NGINX access logs
    • NGINX error logs
  • Network Logs
    • Squid Proxy Logs
    • Tiny Proxy Logs

Following this document you should be able to successfully send data to Cloudlytics.

System Logs

General system logs

All the actions regarding your system are stored in /var/log/syslog ( Ubuntu ) by default . For example which cron was executed at what time etc.
These metrics are useful in order to see the state of your system.
Cloudlytics helps you to monitor your system using the syslog.

Next we will go through a few steps, to help you configureĀ System Logs of your system using Rsyslog.

Step 1: Create a System Log Stream in your Cloudlytics account

Following screen shots will help you in order to create a System Log Stream

  • Choose “+” to Add new Stream

  • Choose “System” option

  • Choose “System Logs”

  • Fill the following details

  • You will receive a token ( Stream Token ) once the Stream is successfully created. This token is very important since all the future configurations will be done using this token only.

Step 2: Configure Rsyslog on your system to forward your logs to Cloudlytics

The following configuration is required on each of the system you want to monitor.

Create an empty file “/etc/rsyslog.d/21-system-cloudlytics.conf” under “/etc/rsyslog.d” and add the details below, while replacing TOKEN_HERE with the token you received after creating a Stream. (Pleaser refer the above image )

Please pay attention towards the tag “Cron-Server” after %HOSTNAME% .
Tags are provided by you which will help you to identify your resources when you start monitoring your systems on Cloudlytics. You can provide as many as tags you want.
Just provide a “,” after each of tag. Example : %HOSTNAME% ,MyTag_1,MyTag_2,Mytag_3

What did we achieve in the above step ??
We told Rsyslog to forward all the System log data to the Stream which you configured in Cloudlytics.
Your data will be pushed to the Stream for which you have configured the TOKEN .

Please make sure that you provide the correct token for against the stream which you want to configure, else the data will pushed but will not be processed and no data will be available.

You have now successfully configured System log with Cloudlytics.

Step 3: Restart rsyslog

 

SSH Logs

SSH logs helps you get details about the login activities in your system. It is quite useful to monitor your SSH log in order get details about any attacks.
We will be using Rsyslog in order to send SSH SSH logs to the endpoints.
Next we will go through steps, to help you configure SSH log file of your system using Rsyslog.

Step 1: Create a SSH Log Stream in your Cloudlytics account
In order to create SSH Stream refer the following steps;

  • Choose “+” to Add new Stream
  • Choose “System” option
  • Choose “SSH”
  • Fill the necessary details and click “Create Stream”
  • You will receive a “Stream Token”. This token is very important since all the future configurations will be done using this token only.

Step 2: Configure Rsyslog on your system to forward your logs to Cloudlytics

The following configuration is required on each of the system you want to monitor.

Create an empty file “/etc/rsyslog.d/21-system-ssh-cloudlytics.conf” under “/etc/rsyslog.d” and add the details below, while replacing TOKEN_HERE with the token you received after creating a Stream.

Please pay attention towards the tag “Server-1-SSH” after %HOSTNAME% .
Tags are provided by you which will help you to identify your resources when you start monitoring your systems on Cloudlytics. You can provide as many as tags you want.
Just provide a “,” after each of tag. Example : %HOSTNAME% ,MyTag_1,MyTag_2,Mytag_3

What we did in the above step ??
We told Rsyslog to forward all the SSH log data to the Stream which you configured in Cloudlytics.
Your data will be pushed to the Stream for which you have configured the TOKEN . So please make sure that you provide correct token for appropriate log type which you want to configure, else the data will pushed but it will not be processed and you will not be able to make any sense out of your data.
The above configuration will work perfectly fine for any Ubuntu System on which you did not have made any changes and all the settings are default.

Change ??
Look at above configuration and pay attention to the line $InputFileName /var/log/auth.log.
This line describes the location of your SSH log file , the location where your OS is storing all your SSH activities , by default on Ubuntu systems all the SSH activities are logged into “/var/log/auth.log”.
Above configuration will work perfectly fine and all your logs will be pushed to your Stream without any error, if you have not made any changes to the default log location, but this will fail if you have changed your auth.log path
or you are working on any other OS, Example : Amazon Linux, CentOS etc.
Log file locations are specific to OS,so please make sure that you provide correct path to $InputFileName , if you have made any changes to the default log location or your OS stores SSH logs at a different location.
You have now successfully configured System log with Cloudlytics.

Step 3: Restart rsyslog

 

Application Logs

Apache Logs

Follow the steps guide to help you configure Apache Log file of your system using Rsyslog.
We will create two Streams for Apache Logs. One for the Access Logs and another for Error Logs.

Step 1: Create a Apache Access Stream andĀ Apache Error Stream in your Cloudlytics account
In order to create Streams refer the following steps;

Apache Access Stream

  • Choose “+” to Add new Stream
  • Choose “Application” option
  • Choose “Apache access”
  • Fill the necessary details and click “Create Stream”
  • You will receive a “Stream Token”. This token is very important since all the future configurations will be done using this token only.

Apache Error Stream

  • Choose “+” to Add new Stream
  • Choose “Application” option
  • Choose “Apache error”
  • Fill the necessary details and click “Create Stream”
  • You will receive a “Stream Token”. This token is very important since all the future configurations will be done using this token only.

Step 2: Configure Rsyslog on your system to forward your Apache logs to Cloudlytics

The following configuration is required on each of the system you want to monitor.

Create an empty file “/etc/rsyslog.d/21-apache-cloudlytics.conf” under “/etc/rsyslog.d” and add the details below, while replacing APACHE_ACCESS_LOG_TOKEN with the token you received after creating a Apache access log Stream and APACHE_ERROR_LOG_TOKEN with the token you received after creating a Apache error log Stream

Please pay attention towards the tag “Server-1-Access-Log” and “Server-1-Error-Log” after %HOSTNAME% .
Tags are provided by you which will help you to identify your resources when you start monitoring your systems on Cloudlytics. You can provide as many as tags you want.
Just provide a “,” after each of tag. Example : %HOSTNAME% ,MyTag_1,MyTag_2,Mytag_3


What we did in the above step ??
We told Rsyslog to forward all the Apache access log data and Apache error log data to the respective Streams which you configured in Cloudlytics.
Your data will be pushed to the Stream for which you have configured the TOKEN . So please make sure that you provide correct token for appropriate log type which you want to configure, else the data will pushed but it will not be processed and you will not be able to make any sense out of your data.

If you don’t want to configure both of the log types , please refer the following section which will show the configuration needed for Apache access log and Apache error log separately.
Please make necessary changes for token and log file path accordingly.

Apache Access Log

Apache Error Log

Step 3: Restart rsyslog

 

NGINX Logs

Follow the steps guide to help you configure NGINX Log file of your system using Rsyslog.
We will create two Streams for NGINX Logs. One for the Access Logs and another for Error Logs.

Step 1: Create a NGINX Access Stream and NGINX Error Stream in your Cloudlytics account
In order to create Streams refer the following steps;

NGINX Access Stream

  • Choose “+” to Add new Stream
  • Choose “Application” option
  • Choose “NGINX access”
  • Fill the necessary details and click “Create Stream”
  • You will receive a “Stream Token”. This token is very important since all the future configurations will be done using this token only.

NGINX Error Stream

  • Choose “+” to Add new Stream
  • Choose “Application” option
  • Choose “NGINX error”
  • Fill the necessary details and click “Create Stream”
  • You will receive a “Stream Token”. This token is very important since all the future configurations will be done using this token only.

Step 2: Configure Rsyslog on your system to forward your NGINX logs to Cloudlytics

The following configuration is required on each of the system you want to monitor.

Create an empty file “/etc/rsyslog.d/21-nginx-cloudlytics.conf” under “/etc/rsyslog.d” and add the details below, while replacing NGINX_ACCESS_LOG_TOKEN with the token you received after creating a NGINX access log Stream and NGINX_ERROR_LOG_TOKEN with the token you received after creating a NGINX error log Stream

Please pay attention towards the tag “Server-1-NGINX-Access-Log” and “Server-1-NGINX-Error-Log” after %HOSTNAME% .
Tags are provided by you which will help you to identify your resources when you start monitoring your systems on Cloudlytics. You can provide as many as tags you want.
Just provide a “,” after each of tag. Example : %HOSTNAME% ,MyTag_1,MyTag_2,Mytag_3


What we did in the above step ??
We told Rsyslog to forward all the NGINX access log data and NGINX error log data to the respective Streams which you configured in Cloudlytics.
Your data will be pushed to the Stream for which you have configured the TOKEN . So please make sure that you provide correct token for appropriate log type which you want to configure, else the data will pushed but it will not be processed and you will not be able to make any sense out of your data.

If you don’t want to configure both of the log types , please refer the following section which will show the configuration needed for NGINX access log and NGINX error log separately.
Please make necessary changes for token and log file path accordingly.

NGINX Access Log

NGINX Error Log

Step 3: Restart rsyslog

 

Send Logs securely using Rsyslog (System, Application)

Step 1: Install rsuslog-gnutls on the server (linux)

  • For Amazon Linux : yum -y install rsyslog-gnutls
  • For Ubuntu: apt-get install rsyslog-gnutils

Step 2: Download SSL certificate

Step 3: Add Configuration to encrypt log events in transit

  • Add following lines after the defining workDIrectory
    • #RsyslogGnuTLS
      $DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/data-cloudlytics.cert
      $ActionSendStreamDriver gtls
      $ActionSendStreamDriverMode 1
      $ActionSendStreamDriverAuthMode x509/name
      $ActionSendStreamDriverPermittedPeer data.cloudlytics.com

Step 4: Configure rsyslog to send log files to Cloudlytics

  • Configuration files can be downloaded from following links Apache, Nginx, System Log, SSH Log
  • Download configuration files from the links and replace “TOKEN_HERE” with the stream token from your Cloudlytics account.
  • Replace “TAGS_HERE” with the comma separated tags that you want to assign to events.
  • Restart the rsyslog service and check your Cloudlytics for processed events/logs.