travisgroth/reddit-exporter repository overview
reddit-exporter
A prometheus compatible exporter for generating stats about the traffic on a given subreddit. It is meant to help capture traffic and utilization trends over time without crawling over historic data. The most likely use case is for monitoring a community related to a particular product or service, though more novel usage may arise.
A single exporter can handle multiple subreddits and breaks down all stats by subreddit in addition to other dimensions.
Internally, reddit-exporter runs a continous scan on new content to a subreddit to accumulate metrics on posts and comments. Any "cheap" point-in-time metrics are retrieved at scrape time.
Current Capabilities
- Post counter by type
- Comment counter by type (top level discussion or a reply)
- Configurable regex matching counters for both post and comment content
- Subscriber gauge
- Active user gauge
Maybe Capabilities
- Custom metrics via RPC to a downstream service
Install
Building reddit-exporter uses go modules and requires go 1.11+
Go Install
go install github.com/travisgroth/reddit-exporter/cmd/reddit-exporter
Compile
go build cmd/reddit-exporter/*.go
Docker
docker run -p 8000:8000 travisgroth/reddit-exporter -s askreddit
Helm
See chart source here
$ helm repo add reddit-exporter https://reddit-exporter-chart.storage.googleapis.com
"reddit-exporter" has been added to your repositories
helm upgrade --install reddit-exporter-test reddit-exporter/reddit-exporter --set subreddits="{wtf,askreddit}"
Usage
Command Line
Usage:
reddit-exporter [flags]
Flags:
--address string Metrics server bind address (default "0.0.0.0")
-h, --help help for reddit-exporter
-p, --port int Metrics server port (default 8000)
--regexfile strings File containing regex matches in format 'name;regex'
Can be specified multiple times or comma separated
-s, --subreddit strings Subreddit(s) to monitor (Required).
Can be specified multiple times or comma separated
-v, --verbose Turn on verbose mode
You may list multiple subreddits either comma separated or with multiple '-s' flags.
Verbose mode logs individual events and is good for debugging.
See next section for how to use regexfiles.
Typical command:
$ ./reddit-exporter -v -s askreddit,wtf
time="2019-02-23T15:52:36-05:00" level=info msg="Listening on 0.0.0.0:8000"
time="2019-02-23T15:52:36-05:00" level=info msg="Starting scan for subreddits: [askreddit wtf]"
time="2019-02-23T15:52:39-05:00" level=info msg="Scan started..."
time="2019-02-23T15:52:40-05:00" level=debug msg="Post from Nar40e03" author=Nar40e03 subreddit=AskReddit
Regex Configuration
You may configure reddit-exporter with multiple files containing labeled regexes. You may use json, yaml or properties file format for your regexfile. Matches to these regexes will be kept track of in a counter broken down by matchgroup and matchname.
Example yaml file:
MyMatchGroup:
MyFirstMatchName: ^first$
MySecondMatchName: ^second$
Equivilent properties file:
MyMatchGroup.MyFirstMatchName=^first$
MyMatchGroup.MySecondMatchName=^first$
Each regex has a matchgroup name as well as a match name.
-
Match Groups should be used to label related regexes for easier aggregation and querying later.
-
Match Names indicate the exact regex that matched
-
Both the match name and match group appear as labels on the metric created for the regex
Regexes are not wildcard padded so if you are looking for mid-string matching, be sure to include '.*' on both ends of your regex.
reddit-exporter uses viper under the hood, so as long as your chosen format presents key/value matches as expected, any supported file format should work.
Sample Metrics
Assuming a regexfile containing the following:
test.the=.* the .*
# HELP subreddit_active_users Current active users by sub
# TYPE subreddit_active_users gauge
subreddit_active_users{subreddit="askreddit"} 95117
# HELP subreddit_comments_total Comment counters by type and sub
# TYPE subreddit_comments_total counter
subreddit_comments_total{subreddit="askreddit",type="discussion"} 56
subreddit_comments_total{subreddit="askreddit",type="response"} 45
subreddit_comments_total{subreddit="wtf",type="discussion"} 3
# HELP subreddit_comment_matches_total Comment regex counters by matchgroup name, match name, and sub
# TYPE subreddit_comment_matches_total counter
subreddit_comment_matches_total{match="the",matchgroup="test",subreddit="askreddit"} 42
# HELP subreddit_posts_total Post count by sub, flair and type
# TYPE subreddit_posts_total counter
subreddit_posts_total{flair="",subreddit="askreddit",type="self"} 7
subreddit_posts_total{flair="",subreddit="wtf",type="link"} 1
# HELP subreddit_post_matches_total Post regex counters by matchgroup name, match name and sub
# TYPE subreddit_post_matches_total counter
subreddit_post_matches_total{match="the",matchgroup="test",subreddit="askreddit"} 3
# HELP subreddit_subscriber_users Current subscriber users by sub
# TYPE subreddit_subscriber_users gauge
subreddit_subscriber_users{subreddit="askreddit"} 2.1760036e+07
Tag summary
Content type
Image
Digest
Size
9.1 MB
Last updated
over 5 years ago
docker pull travisgroth/reddit-exporter