All Statsd Metrics

account-auditor Metrics

Metric Name	Description
account-auditor.errors	Count of audit runs (across all account databases) which caught an Exception.
account-auditor.passes	Count of individual account databases which passed audit.
account-auditor.failures	Count of individual account databases which failed audit.
account-auditor.timing	Timing data for individual account database audits.

account-reaper Metrics

Metric Name	Description
account-reaper.errors	Count of devices failing the mount check.
account-reaper.timing	Timing data for each reap_account() call.
account-reaper.return_codes.X	Count of HTTP return codes from various operations (e.g. object listing, container deletion, etc.). The value for X is the first digit of the return code (2 for 201, 4 for 404, etc.).
account-reaper.containers_failures	Count of failures to delete a container.
account-reaper.containers_deleted	Count of containers successfully deleted.
account-reaper.containers_remaining	Count of containers which failed to delete with zero successes.
account-reaper.containers_possibly_remaining	Count of containers which failed to delete with at least one success.
account-reaper.objects_failures	Count of failures to delete an object.
account-reaper.objects_deleted	Count of objects successfully deleted.
account-reaper.objects_remaining	Count of objects which failed to delete with zero successes.
account-reaper.objects_possibly_remaining	Count of objects which failed to delete with at least one success.

account-server Metrics

..note::

“Not Found” is not considered an error and requests which increment errors are not included in the timing data.

Metric Name	Description
account-server.DELETE.errors.timing	Timing data for each DELETE request resulting in an error: bad request, not mounted, missing timestamp.
account-server.DELETE.timing	Timing data for each DELETE request not resulting in an error.
account-server.PUT.errors.timing	Timing data for each PUT request resulting in an error: bad request, not mounted, conflict, recently-deleted.
account-server.PUT.timing	Timing data for each PUT request not resulting in an error.
account-server.HEAD.errors.timing	Timing data for each HEAD request resulting in an error: bad request, not mounted.
account-server.HEAD.timing	Timing data for each HEAD request not resulting in an error.
account-server.GET.errors.timing	Timing data for each GET request resulting in an error: bad request, not mounted, bad delimiter, account listing limit too high, bad accept header.
account-server.GET.timing	Timing data for each GET request not resulting in an error.
account-server.REPLICATE.errors.timing	Timing data for each REPLICATE request resulting in an error: bad request, not mounted.
account-server.REPLICATE.timing	Timing data for each REPLICATE request not resulting in an error.
account-server.POST.errors.timing	Timing data for each POST request resulting in an error: bad request, bad or missing timestamp, not mounted.
account-server.POST.timing	Timing data for each POST request not resulting in an error.

account-replicator Metrics

Metric Name	Description
account-replicator.diffs	Count of syncs handled by sending differing rows.
account-replicator.diff_caps	Count of “diffs” operations which failed because “max_diffs” was hit.
account-replicator.no_changes	Count of accounts found to be in sync.
account-replicator.hashmatches	Count of accounts found to be in sync via hash comparison (broker.merge_syncs was called).
account-replicator.rsyncs	Count of completely missing accounts which were sent via rsync.
account-replicator.remote_merges	Count of syncs handled by sending entire database via rsync.
account-replicator.attempts	Count of database replication attempts.
account-replicator.failures	Count of database replication attempts which failed due to corruption (quarantined) or inability to read as well as attempts to individual nodes which failed.
account-replicator.removes.<device>	Count of databases on <device> deleted because the delete_timestamp was greater than the put_timestamp and the database had no rows or because it was successfully sync’ed to other locations and doesn’t belong here anymore.
account-replicator.successes	Count of replication attempts to an individual node which were successful.
account-replicator.timing	Timing data for each database replication attempt not resulting in a failure.

container-auditor Metrics

Metric Name	Description
container-auditor.errors	Incremented when an Exception is caught in an audit pass (only once per pass, max).
container-auditor.passes	Count of individual containers passing an audit.
container-auditor.failures	Count of individual containers failing an audit.
container-auditor.timing	Timing data for each container audit.

container-replicator Metrics

Metric Name	Description
container-replicator.diffs	Count of syncs handled by sending differing rows.
container-replicator.diff_caps	Count of “diffs” operations which failed because “max_diffs” was hit.
container-replicator.no_changes	Count of containers found to be in sync.
container-replicator.hashmatches	Count of containers found to be in sync via hash comparison (broker.merge_syncs was called).
container-replicator.rsyncs	Count of completely missing containers where were sent via rsync.
container-replicator.remote_merges	Count of syncs handled by sending entire database via rsync.
container-replicator.attempts	Count of database replication attempts.
container-replicator.failures	Count of database replication attempts which failed due to corruption (quarantined) or inability to read as well as attempts to individual nodes which failed.
container-replicator.removes.<device>	Count of databases deleted on <device> because the delete_timestamp was greater than the put_timestamp and the database had no rows or because it was successfully sync’ed to other locations and doesn’t belong here anymore.
container-replicator.successes	Count of replication attempts to an individual node which were successful.
container-replicator.timing	Timing data for each database replication attempt not resulting in a failure.

container-server Metrics

Note

“Not Found” is not considered an error and requests which increment errors are not included in the timing data.

Metric Name	Description
container-server.DELETE.errors.timing	Timing data for DELETE request errors: bad request, not mounted, missing timestamp, conflict.
container-server.DELETE.timing	Timing data for each DELETE request not resulting in an error.
container-server.PUT.errors.timing	Timing data for PUT request errors: bad request, missing timestamp, not mounted, conflict.
container-server.PUT.timing	Timing data for each PUT request not resulting in an error.
container-server.HEAD.errors.timing	Timing data for HEAD request errors: bad request, not mounted.
container-server.HEAD.timing	Timing data for each HEAD request not resulting in an error.
container-server.GET.errors.timing	Timing data for GET request errors: bad request, not mounted, parameters not utf8, bad accept header.
container-server.GET.timing	Timing data for each GET request not resulting in an error.
container-server.REPLICATE.errors.timing	Timing data for REPLICATE request errors: bad request, not mounted.
container-server.REPLICATE.timing	Timing data for each REPLICATE request not resulting in an error.
container-server.POST.errors.timing	Timing data for POST request errors: bad request, bad x-container-sync-to, not mounted.
container-server.POST.timing	Timing data for each POST request not resulting in an error.

container-sync Metrics

Metric Name	Description
container-sync.skips	Count of containers skipped because they don’t have sync’ing enabled.
container-sync.failures	Count of failures sync’ing of individual containers.
container-sync.syncs	Count of individual containers sync’ed successfully.
container-sync.deletes	Count of container database rows sync’ed by deletion.
container-sync.deletes.timing	Timing data for each container database row synchronization via deletion.
container-sync.puts	Count of container database rows sync’ed by Putting.
container-sync.puts.timing	Timing data for each container database row synchronization via Putting.

container-updater Metrics

Metric Name	Description
container-updater.successes	Count of containers which successfully updated their account.
container-updater.failures	Count of containers which failed to update their account.
container-updater.no_changes	Count of containers which didn’t need to update their account.
container-updater.timing	Timing data for processing a container; only includes timing for containers which needed to update their accounts (i.e. “successes” and “failures” but not “no_changes”).

object-auditor Metrics

Metric Name	Description
object-auditor.quarantines	Count of objects failing audit and quarantined.
object-auditor.errors	Count of errors encountered while auditing objects.
object-auditor.timing	Timing data for each object audit (does not include any rate-limiting sleep time for max_files_per_second, but does include rate-limiting sleep time for max_bytes_per_second).

object-expirer Metrics

Metric Name	Description
object-expirer.objects	Count of objects expired.
object-expirer.errors	Count of errors encountered while attempting to expire an object.
object-expirer.timing	Timing data for each object expiration attempt, including ones resulting in an error.

object-reconstructor Metrics

Metric Name	Description
object-reconstructor.partition.delete.count.<device>	A count of partitions on <device> which were reconstructed and synced to another node because they didn’t belong on this node. This metric is tracked per-device to allow for “quiescence detection” for object reconstruction activity on each device.
object-reconstructor.partition.delete.timing	Timing data for partitions reconstructed and synced to another node because they didn’t belong on this node. This metric is not tracked per device.
object-reconstructor.partition.update.count.<device>	A count of partitions on <device> which were reconstructed and synced to another node, but also belong on this node. As with delete.count, this metric is tracked per-device.
object-reconstructor.partition.update.timing	Timing data for partitions reconstructed which also belong on this node. This metric is not tracked per-device.
object-reconstructor.suffix.hashes	Count of suffix directories whose hash (of filenames) was recalculated.
object-reconstructor.suffix.syncs	Count of suffix directories reconstructed with ssync.

object-replicator Metrics

Metric Name	Description
object-replicator.partition.delete.count.<device>	A count of partitions on <device> which were replicated to another node because they didn’t belong on this node. This metric is tracked per-device to allow for “quiescence detection” for object replication activity on each device.
object-replicator.partition.delete.timing	Timing data for partitions replicated to another node because they didn’t belong on this node. This metric is not tracked per device.
object-replicator.partition.update.count.<device>	A count of partitions on <device> which were replicated to another node, but also belong on this node. As with delete.count, this metric is tracked per-device.
object-replicator.partition.update.timing	Timing data for partitions replicated which also belong on this node. This metric is not tracked per-device.
object-replicator.suffix.hashes	Count of suffix directories whose hash (of filenames) was recalculated.
object-replicator.suffix.syncs	Count of suffix directories replicated with rsync.

object-server Metrics

Metric Name	Description
object-server.quarantines	Count of objects (files) found bad and moved to quarantine.
object-server.async_pendings	Count of container updates saved as async_pendings (may result from PUT or DELETE requests).
object-server.POST.errors.timing	Timing data for POST request errors: bad request, missing timestamp, delete-at in past, not mounted.
object-server.POST.timing	Timing data for each POST request not resulting in an error.
object-server.PUT.errors.timing	Timing data for PUT request errors: bad request, not mounted, missing timestamp, object creation constraint violation, delete-at in past.
object-server.PUT.timeouts	Count of object PUTs which exceeded max_upload_time.
object-server.PUT.timing	Timing data for each PUT request not resulting in an error.
object-server.PUT.<device>.timing	Timing data per kB transferred (ms/kB) for each non-zero-byte PUT request on each device. Monitoring problematic devices, higher is bad.
object-server.GET.errors.timing	Timing data for GET request errors: bad request, not mounted, header timestamps before the epoch, precondition failed. File errors resulting in a quarantine are not counted here.
object-server.GET.timing	Timing data for each GET request not resulting in an error. Includes requests which couldn’t find the object (including disk errors resulting in file quarantine).
object-server.HEAD.errors.timing	Timing data for HEAD request errors: bad request, not mounted.
object-server.HEAD.timing	Timing data for each HEAD request not resulting in an error. Includes requests which couldn’t find the object (including disk errors resulting in file quarantine).
object-server.DELETE.errors.timing	Timing data for DELETE request errors: bad request, missing timestamp, not mounted, precondition failed. Includes requests which couldn’t find or match the object.
object-server.DELETE.timing	Timing data for each DELETE request not resulting in an error.
object-server.REPLICATE.errors.timing	Timing data for REPLICATE request errors: bad request, not mounted.
object-server.REPLICATE.timing	Timing data for each REPLICATE request not resulting in an error.

object-updater Metrics

Metric Name	Description
object-updater.errors	Count of drives not mounted or async_pending files with an unexpected name.
object-updater.timing	Timing data for object sweeps to flush async_pending container updates. Does not include object sweeps which did not find an existing async_pending storage directory.
object-updater.quarantines	Count of async_pending container updates which were corrupted and moved to quarantine.
object-updater.successes	Count of successful container updates.
object-updater.failures	Count of failed container updates.
object-updater.unlinks	Count of async_pending files unlinked. An async_pending file is unlinked either when it is successfully processed or when the replicator sees that there is a newer async_pending file for the same object.

proxy-server Metrics

In the table, <type> is the proxy-server controller responsible for the request and will be one of account, container, or object.

Metric Name	Description
proxy-server.errors	Count of errors encountered while serving requests before the controller type is determined. Includes invalid Content-Length, errors finding the internal controller to handle the request, invalid utf8, and bad URLs.
proxy-server.<type>.handoff_count	Count of node hand-offs; only tracked if log_handoffs is set in the proxy-server config.
proxy-server.<type>.handoff_all_count	Count of times only hand-off locations were utilized; only tracked if log_handoffs is set in the proxy-server config.
proxy-server.<type>.client_timeouts	Count of client timeouts (client did not read within client_timeout seconds during a GET or did not supply data within client_timeout seconds during a PUT).
proxy-server.<type>.client_disconnects	Count of detected client disconnects during PUT operations (does NOT include caught Exceptions in the proxy-server which caused a client disconnect).

Additionally, middleware often emit their own metrics

proxy-logging Middleware

In the table, <type> is either the proxy-server controller responsible for the request: account, container, object, or the string SOS if the request came from the Swift Origin Server middleware. The <verb> portion will be one of GET, HEAD, POST, PUT, DELETE, COPY, OPTIONS, or BAD_METHOD. The list of valid HTTP methods is configurable via the log_statsd_valid_http_methods config variable and the default setting yields the above behavior.

Metric Name	Description
proxy-server.<type>.<verb>.<status>.timing	Timing data for requests, start to finish. The <status> portion is the numeric HTTP status code for the request (e.g. “200” or “404”).
proxy-server.<type>.GET.<status>.first-byte.timing	Timing data up to completion of sending the response headers (only for GET requests). <status> and <type> are as for the main timing metric.
proxy-server.<type>.<verb>.<status>.xfer	This counter metric is the sum of bytes transferred in (from clients) and out (to clients) for requests. The <type>, <verb>, and <status> portions of the metric are just like the main timing metric.

The proxy-logging middleware also groups these metrics by policy. The <policy-index> portion represents a policy index:

Metric Name	Description
proxy-server.object.policy.<policy-index>.<verb>.<status>.timing	Timing data for requests, aggregated by policy index.
proxy-server.object.policy.<policy-index>.GET.<status>.first-byte.timing	Timing data up to completion of sending the response headers, aggregated by policy index.
proxy-server.object.policy.<policy-index>.<verb>.<status>.xfer	Sum of bytes transferred in and out, aggregated by policy index.

tempauth Middleware

In the table, <reseller_prefix> represents the actual configured reseller_prefix or NONE if the reseller_prefix is the empty string:

Metric Name	Description
tempauth.<reseller_prefix>.unauthorized	Count of regular requests which were denied with HTTPUnauthorized.
tempauth.<reseller_prefix>.forbidden	Count of regular requests which were denied with HTTPForbidden.
tempauth.<reseller_prefix>.token_denied	Count of token requests which were denied.
tempauth.<reseller_prefix>.errors	Count of errors.

tempurl Middleware

Metric Name	Description
proxy-server.tempurl.digests.<digest>	Count of requests authorized using the specified <digest>; may be one of sha1, sha256, or sha512.

formpost Middleware

Metric Name	Description
proxy-server.formpost.digests.<digest>	Count of requests authorized using the specified <digest>; may be one of sha1, sha256, or sha512.