JENNIFER APM

The screens are good for seeing what is happening now, but comparing a day of metrics against a baseline, or putting the applications of an anomalous window next to those of a normal one, is better done by query. This app adds nine query commands that read JENNIFER data.

• Seven console commands - read the same data as the console screens through a JENNIFER Console connect profile (a console account). They are built to be used in the order you narrow down the cause of a performance anomaly, and AI Agent Root Cause Analysis uses them too. They come with app 1.1.2610.0 or later
• Two token commands - read transactions and running services through a JENNIFER APM connect profile (an Open API token)

Every command's options and output fields are listed in full on the store's Query Commands tab. This page is about the order in which to chain them.

Command overview

The token commands cannot read the statistics, problem verdicts, application summaries and transaction profiles the console commands read. Use the console commands for root cause analysis.

The root cause flow

The console commands chain in the order targets → detect → verdict → scope → evidence → cause. Each step's result sets the period, instances and applications of the next. Replace domain_id=1000, the times and the txids in the examples with values from your environment.

1. Targets - jennifer-instances

Every console command takes a domain_id. Start by finding the domain and instance IDs.

jennifer-instances

A restarted instance comes back with the same name and a new instance ID. When analyzing the past, give a period to find the old IDs that have gone away.

jennifer-instances domain_id=1000 duration=1d
| search registered == false
| fields instance_id, instance_name, status

2. Detect - jennifer-metrics

From a day of metrics, compute a baseline (mean and standard deviation) per metric and instance, and see with a z-score how far the last 30 minutes strayed from it. Treat |z| of 3 or more as anomalous.

jennifer-metrics domain_id=1000 duration=1d metrics=service_time,service_rate,service_err_count,sys_cpu,heap_usage,active_service
| eval recent = if(_time >= dateadd(now(), "min", -30), value, null)
| stats avg(value) as base_mean, stddev(value) as base_sd, avg(recent) as recent_avg by metric, instance_id, instance_name
| eval z = if(base_sd > 0, round((recent_avg - base_mean) / base_sd, 2), 0)
| search abs(z) >= 3
| eval abs_z = abs(z)
| sort limit=20 -abs_z

To see when the anomaly started, attach a z-score to every bucket.

jennifer-metrics domain_id=1000 duration=1d metrics=service_time
| join metric, instance_id [
    jennifer-metrics domain_id=1000 duration=1d metrics=service_time
    | stats avg(value) as base_mean, stddev(value) as base_sd by metric, instance_id
  ]
| eval z = if(base_sd > 0, round((value - base_mean) / base_sd, 2), 0)
| search z >= 3
| sort limit=50 -z
| fields _time, instance_name, metric, value, base_mean, z

3. Verdict - jennifer-problems

Check what JENNIFER judged to be a problem, and on which instance, at the time you found. An instance stopping, restarting or reconnecting explains a gap or a jump in that instance's metrics.

jennifer-problems domain_id=1000 from=20261026045000 to=20261026051000
| sort _time

Zero rows is not a failure: it means JENNIFER found no problem in that period.

4. Scope - jennifer-app-summary, jennifer-errors

Compare the applications of the anomalous window with the window just before it, and find the ones whose calls, errors or response time changed.

jennifer-app-summary domain_id=1000 from=20261027020000 to=20261027040000 span=1h
| eval period = if(_time >= date("20261027030000", "yyyyMMddHHmmss"), "incident", "baseline")
| stats sum(calls) as calls, sum(errors) as errors, sum(total_response_time) as total by application, period
| eval avg_rt = round(total / calls, 1)

If errors went up, compare with the same hour the day before to find the error types that grew.

jennifer-errors domain_id=1000 from=20261027090000 to=20261027100000
| stats sum(count) as count by instance_name, error_type_name
| join type=left instance_name, error_type_name [
    jennifer-errors domain_id=1000 from=20261026090000 to=20261026100000
    | stats sum(count) as before_count by instance_name, error_type_name
  ]
| eval before_count = nvl(before_count, 0), increase = count - before_count
| sort limit=20 -increase

5. Evidence - jennifer-xview

List the slow or failed transactions of the narrowed window and see what share of the response time went to SQL, external calls and CPU. Keep the window to a few minutes around the problem.

jennifer-xview domain_id=1000 duration=10m min_response_time=3000
| sort limit=20 -response_time
| fields _time, instance_name, application, response_time, sql_time, external_call_time, cpu_time, error_name, txid

jennifer-xview domain_id=1000 from=20261027030000 to=20261027031000 min_response_time=1000
| stats count, avg(response_time) as rt, avg(sql_time) as sql, avg(fetch_time) as fetch, avg(external_call_time) as ext, avg(cpu_time) as cpu by application
| sort limit=10 -rt

6. Cause - jennifer-profile

Pick a representative transaction and take its row's domain_id, txid, collection_time and response_time.

jennifer-xview domain_id=1000 duration=10m
| sort limit=1 -response_time
| fields domain_id, txid, collection_time, response_time, application

Pass them to jennifer-profile and find the slowest SQL and external calls.

jennifer-profile domain_id=1000 txid=-2980609390376924461 collection_time=1792972233338 response_time=482
| search type == "SQL" or type == "TX_CALL"
| sort limit=10 -elapsed_ms
| fields no, type, name, start_ms, elapsed_ms, share, parameters

An N+1 pattern - the same SQL over and over - shows when you group by SQL.

jennifer-profile domain_id=1000 txid=-2980609390376924461 collection_time=1792972233338
| search type == "SQL"
| stats count, sum(elapsed_ms) as total_ms, max(elapsed_ms) as max_ms by name
| sort -total_ms

elapsed_ms includes nested calls, so an outer method covers the time of the calls inside it. Look for the cause in the deepest rows or in SQL and TX_CALL rows rather than in the outermost method. A large GAP is time spent in code that is not profiled, in locks or in waiting.

Token commands

The token commands read through the same Open API as the collector. Use them to see the same transactions the collector stores without collecting them, or to see the services running right now.

jennifer-apm-transaction-time domain_id=1000 duration=10m

jennifer-apm-active-service-list domain_id=1000

Common options

Error codes

When a console call fails, a console command ends the query with an error, so zero rows really means no data. The token commands, by contrast, only write call errors to the server log; when their result is empty, check the connect profile and the network as well.

Things to know