Web Monitoring
Web Monitoring shows recent web requests collected with the web log schema on three tabs: Bots, Errors and Crawling. Queries run asynchronously on the server and are cancelled automatically when you leave the screen.
Common conditions
Set the conditions at the top right of the screen.
- Tables: Pick one or more tables collected with the web log schema. Each table shows its logger names. Without a pick, All tables are read.
- Range: The last 1 minute, 5 minutes, 10 minutes or 1 hour. The range you picked last is kept for your next visit.
- Refresh: Runs the queries again with the same conditions.
- Cancel queries: Shows with N running while queries run, and cancels them all.
- You move to another tab or menu.
- A new query starts because the conditions changed.
- The browser window is closed, or the result is not fetched for over 30 seconds.
- A query runs for over 10 minutes.
A user can run up to 30 queries at the same time.
Drill-down and filters
- Clicking a bar or a slice narrows the other charts and the list below to that value. Clicking the same item again clears it.
- Clicking an IP in a list also narrows to that IP. On the Errors tab, tables and status codes are clickable too.
- Active filters are listed on the Filters bar above the tab. Remove them one by one, or all at once with Clear all or Ctrl+Q.
- Tabs, filters and pages are kept in the address. The browser's Back button returns to the previous state, and a shared address opens the same view.
Common widget features
- IP addresses: A country flag shows on the left. Hover to see the country and the ASN.
- View query: Click the gear at the top right of a widget and choose View query to see the query it ran, highlighted. Copy to clipboard to take it to the query screen for your own analysis.
- Lists: Up to the latest 1,000 rows are fetched and paged. Beyond 1,000, narrow with a filter or a range.
Bots tab
Shows requests whose User-Agent is not a browser. Bot types are as follows.
| Bot type | Description |
|---|---|
ai_agent | Crawlers and fetchers of LLM services, such as GPTBot and ClaudeBot |
search_engine | Search engine crawlers |
seo | SEO tool crawlers |
social | Social media link previews |
scanner | Vulnerability scanners |
asm | Attack surface management tools |
go, java, python | HTTP client libraries |
shell | curl, wget |
unknown | No User-Agent, or an unknown client |
- Top bot IPs: The 20 bot IPs with the most requests, with their bot types
- Top paths bots reach: The 20 paths bots requested most. Grouped by URL template when one matches, otherwise by path shape (numbers, UUIDs and so on replaced by tokens).
- Recent requests: The latest requests of the picked IP or path. Without a pick, those of the busiest bot IP.
The two charts drill down into each other. Picking an IP shows the paths it reached; picking a path shows the IPs that reached it.
Errors tab
Shows requests with a response status code of 400 and above.
- Tables with the most errors, IPs with the most errors: The top 20
- Status codes: The share of each status code. Click a slice to narrow to that status code.
- Status code trend: Stacked bars of status codes over time. Drag across the chart to redraw it within that period; the other charts and the error log narrow to it too. You can drag again within the narrowed period.
- Error log: Error requests matching the conditions. Click an IP, a table or a status code to narrow to it.
401, 403 and 404 from one IP across many paths can signal authorization probing or path scanning.
Crawling tab
Shows object ID requests: requests carrying a parameter set to Object ID on its URL template. Without object ID parameters, this tab is empty.
- IPs with the most object ID requests: The top 20, with the number of distinct objects each touched
- Most accessed URL templates: The top 20
- Object ID requests: Each request with its object ID (
parameter=value). A request with several object IDs shows one row per ID.
The two charts drill down into each other. An IP that touched many objects in a short time may be enumerating objects. BOLA Detection judges this automatically.