This guide is intended for Loom administrators and users who are responsible for the data ingestion pipeline, i.e. the process of structuring, mapping and manipulating the events prior to the anomaly-detection phase.
Overview Sophie’s data structuring process consist of the following steps:
The pre-processing step is used for:
- Modifying events before they start processing (for example to mask sensitive data).
- splitting events into multiple smaller events.
The pre-processing is executed by the Data-Input, and can be configured by going to the Data-Inputs screen, then selecting “pre-processor”:
In the pre-processing screen, add a code that manipulates the event.
The method accepts 'sample' and the 'metadata' as the input and returns a JSON with either the modified input or an array of events.
As you type, you will be able to see on the left the expected results of your function:
The Mapping step matches every event with a Source-Type (responsible for structuring the event) and a Source (organizational grouping).
There are several methods to map the data:
The agent shipping the events can specify in the designated Source-Type and Source in event metadata.
For example, see the option to specify an Application in the Rsyslog wizard:
When configuring a data-input, you have the option to choose between single and multi-format setting. When selecting “single” you will be required to specify the Source-Type events are mapped to:
Writing a “Mapping function” for the Data-Input is offers the greatest flexibility.
Write a function that gets a sample event and its metadata, and returns the Source-Type, Application, and Service. On the left side, you can see the “before and after” of the mapping step. Use the “Test” button to see the results of running your script on hundreds of events.
Assigning in the Data-Input Structure page
This method should be used when the Source-Type already extracts properties that can be used to determine the Application and Service.
In the Data-Input Structure page, you can drag the “Application” and “Service” labels and drop them over the desired properties:
Each event is first mapped to a Source-Type, which is an object containing “parsing instructions” for a specific event-structure.
Each event must contain a Timestamp representing the time at which the event took place. By default, Sophie discards events which don’t contain such a timestamp.
When Auto-Structuring is used, Sophie will try to guess the correct Timestamp to use, preferring earlier timestamps and more-accurate ones.
When events get discarded due to failure to extract a timestamp, a notification banner will show:
Additionally, an email will be sent to addresses subscribed to receive TIMESTAMP_PARSING_ERROR notifications.
Troubleshooting timestamp errors
Sophie will alert with the following error whenever there are timestamp issues:
- "Events are being dropped due to timestamp parsing failure. Please refer to the Data Structuring guide or contact email@example.com"
To troubleshoot if the extraction fails on some events, navigate to the Structure page and click “Test”.
If any of the example events failed to extract a timestamp, it will appear under the “Timestamp Missing” - and clicking the red information tooltip will show the failing example:
If the timestamp seems to be extracted properly, it might be that the format isn’t supported.
Go to Settings→Administration→Timestamp Formats
Paste the timestamp extracted from the event and see if it matches any of the default formats.
If not, an error will be displayed:
You now have the option to add a new Timestamp format, following the Java 8 standard.
Whenever a new Source-Type is created, Sophie first attempts to “learn” its structure and parse its events automatically. The Auto-Structuring can be toggled for a specific Source-Type, in its Structure page:
Sophie will alert with the following error whenever Auto-structuring is slowing down Sophie:
- "Auto-structuring is slowing down Sophie. Please refer to the Data Structuring guide or contact firstname.lastname@example.org"
It might happen that the Auto-Structuring of some Source-Type becomes slow. When this happens, a warning will be presented in the web-app:
The offending Source-Type can be located in the logs.
This typically happens when multiple source-types are mapped to the same Source-Type (i.e. when a single Source-Type is responsible for more than a single structure).
To troubleshoot, first check if this is the case, and if so, split the events into two or more Source-Types, each handling a single event-structure.
If a slow Source-Type is responsible for a single structure, troubleshoot it by first restarting the “learning” (locate the “Restart training” button in the structure page):
If the problem persists, disable auto extraction and write a script parsing your events.
Sophie will alert with the following error whenever there are issues with the rate of scripts:
- "The rate of script errors is above normal. Please refer to the Data Structuring guide or contact email@example.com"
The different scripts might fail due to various reasons – script errors, change in the data structure or just some edge case that wasn’t considered.
When the script fails, events are discarded, and a message is printed to the log.
If the rate of failures is significant, a banner will be displayed in the web-application:
Additionally, an email will be sent to addresses subscribed to receive JS_ERROR notifications.
To troubleshoot, locate the problematic Source-Type and events in the logs, then head to the corresponding Structure page, paste the problematic event and click “Go”.
Fix the script to handle the event correctly, then save it.
Troubleshooting New-Metrics rate
Sophie will alert with the following error whenever there are issues with new metrics:
- "Too many metrics are being generated. The problematic property was automatically classified as invalid. Please refer to the Data Structuring guide or contact firstname.lastname@example.org"
A problematic Structuring script or classification could cause Sophie to track and analyze senseless data.
Often, this will be random data (e.g. Thread ID) classified as Meter or a valid property that gets assigned semi-random values (e.g. HTTP Status that is extracted incorrectly).
Sophie will identify such scenarios and act by invalidating the offending property.
When this happens, a notification will be presented in the web-app:
Additionally, an email will be sent to addresses subscribed to receive TOO_MANY_RAW_METRICS notifications.
To troubleshoot, search the logs for the warning specifying which property was invalidated.
Open the Source-Type Structure page for the relevant Source-Type, then fix the script and reclassify the property.