Install Guide
Overview
This app collects the JSON log files exported by Somansa Privacy-i into Logpresso Sonar. Sonar connects over SFTP to the host where the log files accumulate and reads them. Collecting documents directly from the Elasticsearch index that Privacy-i writes to is also supported.
The parser the app provides determines the log type, normalizes code values into their Korean display labels, and renames fields to match the schema. The log types the parser handles and the schema each one is normalized to are listed below.
| Field prefix | Log type | Log schema |
|---|---|---|
action_ | Personal data file action result | privacy-i-action |
dlog_ | Discover personal data scan result | privacy-i-discover-log |
dfile_ | Discover personal data detected file | privacy-i-discover-file |
elog_ | Endpoint policy violation incident | privacy-i-endpoint-log |
efile_ | Endpoint incident file detail | privacy-i-endpoint-file |
eevt_ | Endpoint user activity event | privacy-i-endpoint-event |
Scope
Requirements
| Item | Requirement | Note |
|---|---|---|
| File access permission | Read permission on the log file directory and its subdirectories | The SFTP account for remote collection, the Sonar collection node account for local collection |
| Sonar permission | System administrator permission | Access to Collect > Loggers and System > Connect Profiles |
| Network allowance | TCP/22 (Sonar collection node → log file host) | Inbound allowed on the log file host, outbound allowed on the Sonar collection node. Use the actual SSH port if it was changed |
| Elasticsearch app (alternative) | Required only when collecting directly from an Elasticsearch index | See Collect directly from an Elasticsearch index |
Logpresso collection configuration
Check the directory path where the log files accumulate and the file naming rule with your Privacy-i contact. The path depends on how log forwarding is configured, and you use it for the directory path and filename pattern inputs in the steps below.
Step 1: Add a connect profile
The SFTP loggers connect to the log file host with an SSH type connect profile. Go to System > Connect Profiles, click Add, and enter the following values.
| Type | Item | Setting |
|---|---|---|
| Required | Name | A unique name that identifies the connect profile |
| Required | Identifier | A unique identifier for the connect profile, used in Logpresso queries |
| Required | Type | Select SSH |
| Required | Host | IP address or hostname of the log file host (e.g. 192.0.2.1) |
| Required | Port | SSH port. Defaults to 22 |
| Required | Account | An SSH account with read permission on the log files (e.g. test_user) |
| Required | Authentication type | Select Auth key or Password. Defaults to Auth key |
| Optional | Password | Enter only when the authentication type is Password |
| Required | Timeout | Connection timeout in seconds. Defaults to 30 |
| Optional | Keep-Alive Interval | In milliseconds. 60000 when not specified |
When you choose Auth key as the authentication type, open the Set public key dialog next to the authentication type field, copy the Sonar SSH public key, and register it in the authorized_keys file of that account on the log file host. The connection test fails until the key is registered.
Once the values are entered, run the connection test and confirm that the result is a success.
Step 2: Add a logger
Go to Collect > Loggers, click Add, and enter the following values. The settings below the logger model depend on the type of the logger model.
| Type | Item | Setting |
|---|---|---|
| Required | Name | A unique name that identifies the logger |
| Required | Logger model | Select Privacy-i |
| Required | Storage location/Collection location | Select a node appropriate for your Logpresso platform configuration |
| Required | SSH profile | Select the connect profile registered in step 1 |
| Required | Directory path | Absolute path of the directory that holds the log files |
| Required | Filename pattern | A regular expression matching the names of the files to collect |
| Optional | Charset | utf-8 when not specified |
| Optional | Filename Tag | Field name to record the name of the collected file |
| Optional | Path Date Format | Format used to extract a date from the file path. Used with Scan Days |
| Optional | Scan Days | Collect only files whose extracted date falls within the given days |
| Required | Table | Enter a table name starting with PRIVACY_I (e.g. PRIVACY_I) |
- This logger model collects only the files directly under the directory path and does not descend into subdirectories. Entering a parent path or a wildcard in the directory path does not make it pick up files in subdirectories. If the files accumulate across subdirectories in your environment, see "(Reference) When the log files are stored differently" below.
(Reference) When the log files are stored differently
The app provides a single logger model, of the SFTP Directory Watcher type. When the storage layout differs as shown below, add a logger model of the matching type yourself under Collect > Logger Models, select Privacy-i as its parser, and create the logger from that model. The logger settings vary with the type you choose.
| Log file storage layout | Logger model type | Note |
|---|---|---|
| Collected in one directory on a remote host | SFTP Directory Watcher | Provided by the app. Proceed with step 2 as written |
| Split across subdirectories, by date or a similar key | SFTP Daily Rolling Directory Watcher | Point the directory path at the parent of the subdirectories. Adds the monitoring period and old log scan date settings |
| On the local file system of the Sonar collection node | Recursive Directory Watcher | The connect profile in step 1 is not needed. Set Recursive to true |
(Alternative) Collect directly from an Elasticsearch index
In environments where Privacy-i stores its logs in Elasticsearch, you can fetch the documents straight from the index instead of going through files. This method requires the Elasticsearch app to be installed as well.
Install the Elasticsearch app, then register the following in order. For the individual settings, refer to the Elasticsearch app manual.
- Under System > Connect Profiles, add a connect profile of the
Elasticsearchtype and enter the endpoint and the credentials. - Under Collect > Logger Models, add a logger model with the type set to
Elasticsearchand the parser set toPrivacy-i. - Under Collect > Loggers, select that logger model and connect profile, then enter the index name to collect from and the sort field. Use a table name starting with
PRIVACY_I.
With this method the parser determines the log type from the _type value of the document and stores it in the doc_type field.
Post-installation verification
Run the query below to confirm that collection is actually working.
- Normal: the number of records collected within 10 minutes is 1 or more.
- Abnormal: if it is 0, recheck the file read permission of the SFTP account, the directory path, the filename pattern, and the connection test result of the connect profile. Also check whether the files are accumulating in subdirectories.
Run the query below to confirm that log types are being determined correctly. For file collection the values start with action, dlog, dfile, elog, efile, or eevt; for Elasticsearch collection they start with pi_. Either means normalization was applied.