xetusoss/forwards-logs repository overview

⁠ForwaRDS-logs

Filters and forwards RDS mysql general logs to other logging infrastructure, such as greylog or syslog.

⁠Running the app

Mount the directory containing the configuration files to /forwaRDS-logs/etc. For example, if you stored your configuration files in /path/to/forwards/config/

docker run -v /path/to/forwards/config:/forwaRDS-logs/etc xetus-oss/forwards-logs

⁠Setup and configuration

⁠forwardsconfig.groovy

main application configuration

The configuration file must be named forwardsconfig.groovy and uses the groovy language for configuration.

⁠The ac (amazon config) closure

the ac closure, which requires the following properties be set

• accessKey -> The access key used to authenticate with amazon web services api
• secretKey -> The secret key used to authenticate with amazon web services api
• instanceIdentifier -> The database instance identifier

Here is an example of what this would look like

ac {
  accessKey = 'MYACCESSKEY'
  secretKey = 'MYSECRETKEY'
  instanceIdentifier = 'db-id'
}

⁠The forwarders list of maps

a forwarder is a two part object, the first part is the logging level at which log entries that do not get filtered should be forwarded to. This should be specified in a map with level as the key.

The second part is the filters map, which is keyed by a FilterOnEntryProperty enumeration and has a value that represents a regular expression to execute on the FilterOnEntryProperty.

FilterOnEntryProperty enumeration: enumerates the fields of a GeneralLogEntry object, which are the following ->

• TIME -> timestamp for the log entry
• ID -> the id of the log entry
• COMMAND -> the command being logged
• ARGUMENT -> the argument for the command being logged

The forwarders configuration parameter is a list of maps, each forwarder is evaluated in the order specified in the forwardsconfig.groovy, the first forwarder that does not filter out a particular entry will forward the entry at the specified log level to all applicable log appenders.

Here's an example forwarders configuration

import ch.qos.logback.classic.Level
import com.forwards.FilterOnEntryProperty;

forwarders = [[
  level:Level.INFO, filters: [
    (FilterOnEntryProperty.COMMAND): ~/Connect/,
    (FilterOnEntryProperty.ARGUMENT): ~/^(?!webapp)[\s\S]*/]
]]

This will log entries at the info log level where the command is Connect, and where the argument does not begin with webapp

⁠logback.groovy

logging configuration

The logback configuration happens through groovy configuration, there are more detailed explanations on how to do that here⁠

IMPORTANT NOTE ON LOGGING APPENDERS Using the typical logback appenders, the timestamp would be set to the time the logging event occured, and not the time set for the database log entry. New appenders were created that extend the common logging appenders that allow setting the timestamp to the timestamp of the database log entry instead of when this applications log event happened. The only requirement is that the object array passed when logging through the application includes either the GeneralLogEntry object or a Date object with the timestamp.

The following appenders were created:

• TimestampModifyingConsoleAppender -> extends ConsoleAppender
• TimestampModifyingGELFAppender -> extends GELFAppender, for logging to greylog
• TimestampModifyingSyslogAppender -> extends SyslogAppender, for logging to syslog

If you use these appenders instead of the ConsoleAppender, GELFAppender or SyslogAppender where appropriate, you can have the timestamps match the timestamp from the general log.

Here is an example logback.groovy configuration:

import static ch.qos.logback.classic.Level.*
import ch.qos.logback.classic.encoder.PatternLayoutEncoder

import com.forwards.logging.TimestampModifyingConsoleAppender
import com.forwards.logging.TimestampModifyingGELFAppender
import com.forwards.logging.TimestampModifyingSyslogAppender
 
appender("STDOUT", TimestampModifyingConsoleAppender) {
  encoder(PatternLayoutEncoder) {
    pattern = "%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{0} - %msg%n"
  }
}

appender("GELF", TimestampModifyingGELFAppender) {
  graylog2ServerHost = "test.graylog.com"
  graylog2ServerPort = 12201
  useLoggerName = true
  graylog2ServerVersion = "0.9.6"
  chunkThreshold = 1000
  messagePattern = "%m%rEx" 
  shortMessagePattern = "%.-100(%m%rEx)"
  additionalFields = [application: "forwaRDS"]
  includeFullMDC = true
  hostName = 'RDS-general-log'
}

appender("SYSLOG", TimestampModifyingSyslogAppender) {
  syslogHost = "localhost"
  facility = "USER"
  suffixPattern = "[%thread] %logger %msg"
}

root(INFO, ["STDOUT"])
logger("com.forwards", TRACE, ["STDOUT", "GELF", SYSLOG], false)