Install the Loom Systems Fluentd plugin

To use the Fluentd agent with Sophie, you will need to install and configure the Loom open-source output plugin.
The plugin formats the events in JSON and sends them over a TCP (encrypted by default) socket.
To install the plugin run the following command:

gem install fluent-plugin-loomsystems

Configure the plugin

To match events and direct them to the loomsystems output, add the following clause to your Fluentd configuration file:

<match **>
  @type loomsystems
  host <your-subdomain>.loomsystems.com
</match>

Remember to replace <your-subdomain>  with your data-domain.
Next, add the loomsystems  tag to every source  you would like to ship.

Example of match (output) with event tag: 

<source>
  @type dummy
  dummy {"hello":"loomsystems"}        
  tag loomsystems
</source>  

<match loomsystems.**>
  @type loomsystems
  host <your-subdomain>.loomsystems.com
</match>

Restart the Fluentd agent for the configuration to take place.
See more advanced plugin configuration options in the plugin repository

Shipping files

When shipping files, it is recommended to add several properties to the source clause to make it easier to map and process by Sophie.
Here is an example of a recommended configuration:

<source>
  @type tail
  path /path/to/app1/*.log
  pos_file /tmp/fluentd.app1.pos
  path_key tailed_path
  tag loomsystems.app1
  format none
  #encoding UTF-8
</source>

<source>
  @type tail
  path /path/to/app2/*.log
  pos_file /tmp/fluentd.app2.pos
  path_key tailed_path
  tag loomsystems.app2
  format none
</source>

<filter loomsystems.app1>
  @type record_transformer
  <record>
    loomApplication "app1name"
  </record>
</filter>

<filter loomsystems.app2>
  @type record_transformer
  <record>
    loomApplication "app2name"
    loomService "app2service"
    loomSourceType "app2souceType"
    prop1 "val1"
  </record>
</filter>

<match loomsystems.**>
  @type loomsystems
  host example-data.loomsystems.com
</match>

path_key  - this adds the file name to every sent entry
filter  - this clause is used to add additional attributes to the events
loomApplication, loomService  and loomSourceType  are special entries that serve as hints to Sophie, helping in mapping and processing the data.

Ship Multi-line events

The td-agent provides a regex-based Multiline Parser Plugin, allowing you to merge multiple log lines and ship them as unified events. The full guide can be found here.

In the relevant source section of your configuration file,
add the following parameters: 

format multiline
format_firstline /^<ReplaceWithTheFirstLine'sRegex>/
format1 /<?message>(.*)/

for example:

<source>
  @type tail
  path /path/to/app2/*.log
  pos_file /tmp/fluentd.app2.pos
  path_key tailed_path
  tag loomsystems.app2
  format multiline
  format_firstline /^\d{4}-\d{2}-\d{2}\s/
  format1 /<?message>(.*)/
</source>

Notice that after the first 2 parameters (format multiline, and format_firstline), you can add formatN (when N is a number) for the number of formats you'd like your multiline part to catch (so only log lines matching one of these formats will be merged). And, you need to define a title for them, using the:

/?<SomeTitle>(SomeCapturingGroup)/

before every capturing group.

Did this answer your question?