brunswickheads/ade-s390x repository overview

⁠Anomaly Detection Engine

ADE detects anomalous time slices and messages in Linux logs (either RFC3164 or RFC5424 format) using statistical learning.

To predict anomalous behavior ADE processes the Linux logs to create a model of expected behavior and compares the expected behavior with the behavior of the time periods of interest. It does not require that either messages or time slices be labelled. ADE uses unsupervised statistical learning algorithms that depend on the behavior of enterprise IT solutions running on Linux being stable and predictable.

The ADE analysis results are written to files in XML format, which can be viewed using a web browser or used in other processing to support the enterprise Linux IT solution that is generating the logs.

For each time slice (interval), ADE measures how unusual the interval is by

• Calculating an anomaly score that describes how unusual the time slice is
• Determining the number of similar message strings within an interval
• Determining the number of new message strings within an interval

ADE creates a summary file with this information for all of the time slices (interval) within a day (period).

⁠To run

Start the MariaDB container:

docker run --name=mariadb -v /tmp/mariadb:/var/lib/mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=passw0rd brunswickheads/mariadb-5.5-s390x:latest mysqld_safe --connect-timeout=30

Note the --connect-timeout=30 was used on my system because it is running on a zPDT and would timeout more readily.

Start the ADE container:

docker run --rm -i -t --name=ade -p 8022:22 --link=mariadb -v /tmp/ade:/var/local/ade -e MARIADB_ROOT_PASSWORD=passw0rd -e MARIADB_ADE_PASSWORD=passw0rd brunswickheads/ade-s390x

To use your own setup.props file which, by default, looks like this:

# --------------------------------------------------------------------
# AdeExt properties
# --------------------------------------------------------------------
adeext.msgRateReportFreq=5
adeext.msgRateMsgToKeep=1000
adeext.parseErrorToKeep=100
adeext.parseErrorDaysTolerate=2
adeext.parseErrorTrackNullComponent=false
adeext.runtimeModelDataStoreAtSource=true

adeext.msgRate10MinSlotsToKeep=24
adeext.msgRate10MinSubIntervalList=1,2,3,6,12,24
adeext.msgRateMergeSource=true

# --------------------------------------------------------------------
# Paths
# --------------------------------------------------------------------

ade.flowLayoutFile=conf/xml/FlowLayout.xml
ade.outputPath=/var/local/ade/output/
ade.analysisOutputPath=/var/local/ade/output/continuous
ade.xml.xsltDir=conf/xml
ade.criticalWords.file=conf/criticalWords.txt
ade.analysisGroupToFlowNameMapperClass=org.openmainframe.ade.ext.os.LinuxAnalysisGroupToFlowNameConstantMapper
ade.outputFilenameGenerator=org.openmainframe.ade.ext.output.ExtOutputFilenameGenerator
ade.inputTimeZone=GMT+00:00
ade.outputTimeZone=GMT

# --------------------------------------------------------------------
# Database settings
# --------------------------------------------------------------------

ade.dataStoreType=SQL
ade.databaseUrl=jdbc:mysql://mariadb:3306/ade
ade.databaseDriver=org.mariadb.jdbc.Driver
ade.databaseUser=dbuser
ade.databasePassword=passw0rd
ade.database.keepOnlyAscii=true

# --------------------------------------------------------------------
# Interval/Period settings
# --------------------------------------------------------------------

# Available period modes: HOURLY, DAILY, WEEKLY, MONTHLY
ade.periodMode=DAILY

You can use -v <path-to-your-setup.props>:/opt/ade/conf/setup.props on the run command. You can then set your own password or change any of the other parameters. If you change the password remember to reflect this in the -e MARIADB_ADE_PASSWORD argument.