URL Normalization
URL Normalization manages how request paths are grouped into URL templates and how parameters are extracted. It has three tabs: URL Templates, Sync Status and Test. Adding, editing and deleting templates requires an administrator.
URL Templates tab
The list of registered templates. Narrow it by host and by keywords (template, description), and click a row to open the edit panel on the right. Check several rows to delete them at once.
Adding and editing a template
Click Add template or a row in the list to open the panel on the right.
- Host: The table name that stores the logs. Leave empty to apply to any host.
- Method: The HTTP method. Any applies to every method.
- Path template: The pattern of the request path, in the syntax below.
- Path parameters: Created automatically for each placeholder of the path template. Set the type and the length range.
- Query parameters: Keys of the query string. Register a name and a type with Add query parameter.
- Description: What the template is, for example "Order list of a user".
A saved template reaches every node within seconds.
Path template syntax
Each segment is one of the following.
| Form | Meaning | Example |
|---|---|---|
| Fixed text | A segment equal to that text | orders |
:name | One segment. Its value is extracted as the parameter of that name | :order_id |
:name.ext | One segment ending in a fixed extension. The value without the extension is extracted | :image_id.png |
* | The rest of the path. Last segment only | /static/* |
- A template starts with
/and has no query string. - It is lowercased when saved, with duplicate and trailing slashes removed. Matching ignores case.
- Name placeholders in lowercase letters, digits and underscores, after what they point to. For example
:user_id,:order_id.
When several templates fit a request, the most specific one is used.
- An exact host comes before any host.
- Within a host, a fixed segment comes before
:name.ext, then:name, then*. - At the same position, an exact method comes before Any.
For example, with both /api/users/me and /api/users/:user_id, a request to /api/users/me matches the fixed template.
Parameter types
| Type | Meaning |
|---|---|
| Object ID | Points to an object someone owns. BOLA detection and the Crawling tab look only at this type. For example an order number, an account ID, a per-user document name |
| Public name | The name of a public page anyone may read. For example a blog post slug, a product page name |
| Number | A number that does not point to an object, such as a quantity, a page or a size |
| Enum | A parameter with a few fixed values. For example sort, lang |
| Text | Free text such as a search term |
| URL | A URL or a redirect target. For example return, next |
| Not set | Not decided yet |
Setting Object ID without evidence brings legitimate crawling and public page reads up as BOLA candidates. Set Object ID only on parameters where one subject handles few values and a request for someone else's value gets 401, 403 or 404.
Parameters other than numbers can have a Min length and a Max length. Values outside the range are flagged as parameter anomalies (too_short, too_long). Set the range most requests fall in, not the rare long value.
Sync Status tab
Templates are stored on the analysis server and then distributed to every node through the behavior profile websec-path-templates. This tab shows the distribution status.
| Status | Meaning |
|---|---|
| In sync | Every template has reached the nodes |
| Building | The profile is being rebuilt |
| Pending | Saved templates have not reached the profile yet |
| No profile | The profile was removed. Rebuild creates it again |
Saving or deleting a template rebuilds the profile automatically. If the status does not change for a long time, click Rebuild.
Test tab
Normalizes one request the way the nodes do. Enter the method, host and request path, and click Test to see the following.
- The matched template
- The name, type and value of each parameter
- The path shape (the path with numbers, UUIDs and so on replaced by tokens). Use it to decide a new template when nothing matches.
If the templates have not reached the nodes yet, the result says so.
The tested request stays in the address, so a shared address shows the same result.