jennifer-metrics
Read metric time series of a JENNIFER domain's instances from the console's daily system statistics (the per instance section), as long rows: one row per bucket, instance and metric. Use it to detect anomalies: compute a baseline per metric and instance with stats avg/stddev over a day and compare the recent buckets (z-score), or compare instances with each other at the same time. Then narrow down with jennifer-problems, jennifer-app-summary and jennifer-errors.
- Run jennifer-instances first for domain_id and, if you need only some, instance_ids.
- interval=5m gives 5 minute buckets, interval=1h hourly ones. Counts (service_count, count) are per bucket, max are maxima, and rates, times and resources are averages in the bucket.
- Only buckets that have ended by
to(and now) are returned. The console stores 0 for a bucket an instance did not report in, e.g. before a restarted pod's new id existed, so a run of exact zeros usually means absence rather than idleness. - error_count is the count of the errors jennifer-errors breaks down by type, agent stops and restarts included.
- Periods up to 31 days; each server day is one console call.
Metrics (name, unit):
- Transactions: service_count (count), service_rate (TPS), service_time (avg response ms), service_err_count, service_slow_count, service_cpu, service_sql_time, service_externalcall_time, service_fetch_time, service_method_time, max_tps, active_service, max_active_service, bad_response_active_service.
- SQL and calls: sql_count, sql_time, externalcall_count, externalcall_time, fetch_count, fetch_time, average_db_pool_active_count, average_db_pool_idle_count, average_db_pool_configured_count, max_db_pool_active_count.
- System and runtime: sys_cpu (%), max_sys_cpu, proc_cpu (%), max_proc_cpu, sys_mem, sys_mem_rate (%), max_sys_mem_rate, proc_mem, heap_used (MB), heap_committed, heap_usage (%), nonheap_used, gc_time, gc_count, gc_time_usage, finalize_pending, thread_current, thread_daemon, thread_started, collection_count, file_count, socket_count, alive_agent.
- Users and events: active_user, concurrent_user, visit_hour, visit_day, user_req_interval, event_count, event_normal_count, event_warning_count, event_fatal_count, error_count, frontend_measure_count, average_frontend_time, average_frontend_network_time.
jennifer-metrics [profile=VALUE] domain_id=VALUE [instance_ids=VALUE] [metrics=VALUE] [interval=VALUE] [duration=VALUE] [from=VALUE] [to=VALUE] [limit=VALUE]
- profile=VALUE
- Optional. Name of the JENNIFER console connect profile (type jennifer-apm-console). Every such profile you are granted is used when omitted. With domain_id, which is an id of one console, exactly one profile must be used: name it when you are granted several.
- domain_id=VALUE
- Required. e.g. 1000. jennifer-instances lists them.
- instance_ids=VALUE
- Optional. e.g. 75553,75554. Comma separated instance ids (OID) from jennifer-instances. Every instance that reported in the period is read when omitted.
- metrics=VALUE
- Optional. e.g. service_time,service_rate,sys_cpu. Comma separated metric names from the list in the description. Default: service_rate,service_time,service_count,service_err_count,active_service,sys_cpu,proc_cpu,heap_usage,gc_time.
- interval=VALUE
- Optional. 5m (default) or 1h. The bucket size: 5 minute averages or hourly aggregates.
- duration=VALUE
- Optional. e.g. 10m for the last 10 minutes. Recent period up to now, in s(second), m(minute), h(hour), d(day) units. The last hour when no period is given. At most 31 days.
- from=VALUE
- Optional. Start of the period, yyyyMMddHHmmss. Omitted digits are padded with zero. Overrides duration.
- to=VALUE
- Optional. End of the period, yyyyMMddHHmmss. Omitted digits are padded with zero. Now when omitted.
- limit=VALUE
- Optional. e.g. 20000. Maximum rows, 100000 by default. One row is one bucket of one metric of one instance, so narrow metrics or instance_ids rather than raise it.
Output fields
| Field | Type | Name | Description |
|---|---|---|---|
| _time | Date | Time | e.g. 2026-10-27 10:05:00. Start of the bucket (5 minutes or an hour). |
| profile | String | Connect profile | e.g. jennifer-prod |
| domain_id | Integer | Domain ID | e.g. 1000 |
| instance_id | Long | Instance ID | e.g. 75553 |
| instance_name | String | Instance name | e.g. siem-mon-console-1. Not unique over time; group by instance_id as well. |
| metric | String | Metric | e.g. service_time |
| value | Double | Value | e.g. 132.5. The metric's value in the bucket, in the unit given in the description. |
Usage
Response time per instance over 6 hours:
jennifer-metrics domain_id=1000 duration=6h metrics=service_time | timechart span=5m avg(value) by instance_name
Metrics whose last 30 minutes deviate from the day's baseline (z-score):
jennifer-metrics domain_id=1000 duration=1d metrics=service_time,service_rate,service_err_count,sys_cpu,heap_usage,active_service | eval recent = if(_time >= dateadd(now(), "min", -30), value, null) | stats avg(value) as base_mean, stddev(value) as base_sd, avg(recent) as recent_avg by metric, instance_id, instance_name | eval z = if(base_sd > 0, round((recent_avg - base_mean) / base_sd, 2), 0) | search abs(z) >= 3 | eval abs_z = abs(z) | sort limit=20 -abs_z
Buckets far above the instance's own baseline (when did it start):
jennifer-metrics domain_id=1000 duration=1d metrics=service_time | join metric, instance_id [ jennifer-metrics domain_id=1000 duration=1d metrics=service_time | stats avg(value) as base_mean, stddev(value) as base_sd by metric, instance_id ] | eval z = if(base_sd > 0, round((value - base_mean) / base_sd, 2), 0) | search z >= 3 | sort limit=50 -z | fields _time, instance_name, metric, value, base_mean, z
Compare instances over the last hour (an outlier instance):
jennifer-metrics domain_id=1000 duration=1h metrics=service_time,sys_cpu | stats avg(value) as avg_value by metric, instance_id, instance_name | sort metric, -avg_value
Hourly totals over a week: