pmdastatsdStatsD protocol performance metric domain agent (PMDA)

Synopsis

$PCP_PMDAS_DIR/statsd/pmdastatsd [-Z maximum udp packet size] [-P port] [-v] [-g] [-o debug output filename] [-s] [-r parser type] [-a port] [-z maximum of unprocessed packets]

Description

StatsD is simple, text-based UDP protocol for receiving monitoring data of applications in architecture client-server. pmdastatsd is an Performance Metrics Domain Agent that collects StatsD data, aggregates them and makes them available to any Performance Co-Pilot client, which is ideal for easily tracking stats in your application.

The statsd PMDA supports Counter, Gauge and Duration (with instances for minimum, maximum, median, average, 90th percentile, 95th percentile, 99th percentile, count and standard deviation) metric types, offers multiple parsing options: Ragel or handwritten/custom parser, offers multiple aggregating options for duration metric type: basic histogram or HDR histogram, supports custom form of labels, logging, exports multiple metrics about itself and may be configured either with an ini file or command line options.

Configuration

A brief description of the pmdastatsd command line options follows:

-Z, --max-udp=<value>

Maximum allowed packet size, any larger then this will be thrown away. Default: 1472

-P, --port=<value>

Which port agent will listen to for incoming trafic. Default: 8125

-v, --verbose

Verbosity level. Prints info about agent execution into logfile. Valid values are 0-2. 0 = Default value, shows config information, read socket state, and  first 100 dropped messages. 1 = Shows PMNS and related information. 2 = Most detailed verbosity level, also shows dropped messages above 100. All levels include those belows.

-o, --debug-output-filename=<value>

You can send USR1 signal that 'asks' agent to output basic information about all aggregated metric into a $PCP_LOG_DIR/pmcd/statsd_{name} file. Default: 0

-s, --version

Flag controlling whether or not to log current agent version on start. Default: 0

-p, --parser-type=<value>

Flag specifying which parser to use for incoming packets, Basic = 0, Ragel = 1 . Default: 0

-a, --duration-aggregation-type=<value>

Flag specifying which aggregation scheme to use for duration metrics, basic histogram = 0, HDR histogram = 1 . Default: 1

-z, -max-unprocessed-packets=<value>

Maximum size of packet queue that the agent will save in memory. There are 2 queues: one for packets that are waiting to be parsed and one for parsed packets before they are aggregated. Default: 2048

The agent also looks for a

pmdastatsd.ini file in the $PCP_PMDAS_DIR/statsd directory. There, the same options may be specified, albeit with slightly different names as follows:

max_udp_packet_size=<value>

port=<value>

verbose=<value>

debug=<value>

debug_output_filename=<value>

version=<value>

parser_type=<value>

duration_aggregation_type=<value>

max_unprocessed_packets=<value>

Should an option be specified in both pmdastatsd.ini and command line, then the latter takes precedence. Most of the time you will want to configure the agent with an ini file, as the agent should never be executed directly.

Location of the log file. By default, a log file named statsd.log is written in the current directory of pmcd(1) when pmdastatsd is started, i.e. $PCP_LOG_DIR/pmcd. If the log file cannot be created or is not writable, output is written to standard error instead.

Installation

If you want to install the pmdastatsd, do the following as root:

# cd $PCP_PMDAS_DIR/statsd
# ./Install

To uninstall, do the following as root:

# cd $PCP_PMDAS_DIR/statsd
# ./Remove

pmdastatsd is launched by pmcd(1) and should never be executed directly. The Install and Remove scripts notify pmcd(1) when the agent is installed or removed.

Usage

Once started, pmdastatsd will listen on specified port for any content in a form of:

<metricname>:<value>|<type>

There may be multiple such messages in single datagram, split by a newline character, so this:

<metricname>:<value>|<type>\n<metricname>:<value>|<type>

is valid as well. Constraints for each of the above variables are:

<metricname> = [a-z][a-zA-Z0-9_.]*

<value> = described further in each metric type subsection

<type> = One of the following: c , g or ms

If verbose logging is turned on, agent will log every message parsed and related failures.

All recorded metrics will, if parsed and aggregated successfully, be made available under statsd.* namespace.

1 Counter metric

Stores metrics as simple counters, adding any incoming values to already existing ones.

<metricname>:<value>|c

Where value is positive number.

Example:

After aggregating following messages:

metric:20|c

metric:10|c

metric:3.3|c

Value available to PCP will be:

pminfo --fetch statsd.metric

-> inst [0 or '/'] value 33.3

2 Gauge metric

Stores metrics as modifiable values, with an option to either set, increment or decrement values.

<metricname>:<value>|g

Where value can be in a form of:

'-{value}', when negative value is supplied agent will substract value stored with the value passed

'+{value}', when positive value with a leading plus sign is supplied, the agent will add the passed value to the value stored

'{value}', when a value without any leading sign is supplied, the agent will set the metric to the passed value.

Initial value for metric of gauge type is 0.

Example:

After aggregating following messages:

metric:20|g

metric:+10|g

metric:-3.3|g

Value available to PCP will be:

pminfo --fetch statsd.metric

-> inst [0 or '/'] value 26.7

3 Duration metric

Aggregates values either via HDR histogram or simply stores all values and then calculates instances from all values received.

<metricname>:<value>|ms

Where value is a positive number.

Example:

With larger message count, the values may vary based on selected duration aggregation scheme.

After aggregating following messages:

metric:10|ms

metric:20|ms

Values available to PCP will be:

pminfo --fetch statsd.metric

-> inst[0 or '/min'] value 10

-> inst[1 or '/max'] value 20

-> inst[2 or '/median'] value 10

-> inst[3 or '/average'] value 15

-> inst[4 or '/percentile90'] value 20

-> inst[5 or '/percentile95'] value 20

-> inst[6 or '/percentile99'] value 20

-> inst[7 or '/count'] value 2

-> inst[8 or '/std_deviation'] value 5

Note:

Once you send given metricname with specified type, the agent will no longer aggregate any messages with same. metricname but different type and will throw them away.

4 Labels

StatsD datagrams may also contain key:value pairs separated by commas like so:

metric,tagX=X,tagW=W:5|c

OR

metric:5|ms|#tagX:X,tagW:W

Where tagX is a key, X is a value and tagW is a key, W is a value .

Both key and value of such pair are [a-ZA-Z0-9_.]{1,} .

Both formats are interchangeable and you may combine them together. When key is not unique, right most value takes precendence. This is valid:

metric,tagX=1:5|c|#tagX:2

Pair with key tagX will have value of 2.

You may use these labels to map specific values to some PCP instances. PCP labels are also assigned to these PCP instances. Pairs are ordered by key in resulting instance name and label descriptor.

Single label:

metric,tagX=X:5|c

Such payload would map to PCP as follows (non-related labels were ommited):

pminfo --fetch --labels statsd.metric

->

inst [0 or '/tagX=X'] value 5

inst [0 or '/tagX=X'] labels {'tagX':'X'}

As shown earlier you may also send payload with multiple labels. When multiple labels are supplied they are split in instance name by '::'. Example:

metric,tagX=X,tagW=W:5|c

This resolves to:

pminfo --fetch --labels statsd.metric

->

inst [0 or '/tagX=X::tagW=W'] value 5

inst [0 or '/tagX=X::tagW=W'] labels {'tagX':'X','tagW':'W'}

Note:

Be mindful of the fact that duration metric type already maps to instances even without any labels. Sending labeled value to a such metric creates another 9 (as there are that many hardcoded) instances.

Example:

metric:200|ms

metric:100|ms

metric,target=cpu2:10|ms

metric,target=cpu2:100|ms

metric,target=cpu2:1000|ms

Creates 18 instances. Duration data type and label name compose instance name in following manner:

pminfo --fetch --labels statsd.metric

...

inst [10 or '/max::target=cpu0'] value 1000

inst [10 or '/max::target=cpu0'] labels {'target':'cpu0'}

...

5 Hardcoded stats

Agent also exports metrics about itself:

statsd.pmda.received

Number of datagrams that the agent has received

statsd.pmda.parsed

Number of datagrams that were successfully parsed

statsd.pmda.dropped

Number of datagrams that were dropped

statsd.pmda.aggregated

Number of datagrams that were aggregated

statsd.pmda.metrics_tracked

This metric has 3 instances. counter - Number of tracked counter metrics. gauge - Number of tracked gauge metrics. duration - Number of tracked duration metrics. total - Number of tracked metrics total.

statsd.pmda.settings.max_udp_packet_size

Maximum UDP packet size

statsd.pmda.settings.max_unprocessed_packets

Maximum size of unprocessed packets Q

statsd.pmda.settings.verbose

Verbosity flag

statsd.pmda.settings.debug

Debug flag

statsd.pmda.settings.debug_output_filename

Debug output filename

statsd.pmda.settings.port

Port that is listened to

statsd.pmda.settings.parser_type

Used parser type

statsd.pmda.settings.duration_aggregation_type

Used duration aggregation type

These names are blacklisted for user usage. No messages with these names will processed. While not yet reserved, the whole statsd.pmda.* namespace is not recommended to use for user metrics.

Files

$PCP_PMCDCONF_PATH

command line options used to launch pmdastatsd

$PCP_PMDAS_DIR/statsd/Install

installation script for the pmdastatsd agent

$PCP_PMDAS_DIR/statsd/Remove

undo installation script for the pmdastatsd agent

$PCP_LOG_DIR/pmcd/statsd.log

default log file for error messages and other information from pmdastatsd

PCP Environment

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

See Also

PCPIntro(1), pmcd(1), pcp.conf(5) and pcp.env(5).

Info

PCP Performance Co-Pilot