Install the Loom Systems Fluentd plugin

To use the Fluentd agent with Sophie, you will need to install and configure the Loom open-source output plugin.
The plugin formats the events in JSON and sends them over a TCP (encrypted by default) socket.
To install the plugin run the following command:

gem install fluent-plugin-loomsystems

Configure the plugin

To match events and direct them to the loomsystems output, add the following clause to your Fluentd configuration file:

<match **>
  @type loomsystems
  host <your-subdomain>.loomsystems.com
</match>

Remember to replace <your-subdomain>  with your data-domain.
Next, add the loomsystems  tag to every source  you would like to ship.

Example of match (output) with event tag: 

<source>
  @type dummy
  dummy {"hello":"loomsystems"}        
  tag loomsystems
</source>  

<match loomsystems.**>
  @type loomsystems
  host <your-subdomain>.loomsystems.com
</match>

Restart the Fluentd agent for the configuration to take place.
See more advanced plugin configuration options in the plugin repository

Shipping files

When shipping files, it is recommended to add several properties to the source clause to make it easier to map and process by Sophie.
Here is an example of a recommended configuration:

<source>
  @type tail
  path /path/to/app1/*.log
  pos_file /tmp/fluentd.app1.pos
  path_key tailed_path
  tag loomsystems.app1
  format none
  #encoding UTF-8
</source>

<source>
  @type tail
  path /path/to/app2/*.log
  pos_file /tmp/fluentd.app2.pos
  path_key tailed_path
  tag loomsystems.app2
  format none
</source>

<filter loomsystems.app1>
  @type record_transformer
  <record>
    loomApplication "app1name"
  </record>
</filter>

<filter loomsystems.app2>
  @type record_transformer
  <record>
    loomApplication "app2name"
    loomService "app2service"
    loomSourceType "app2souceType"
    prop1 "val1"
  </record>
</filter>

<match loomsystems.**>
  @type loomsystems
  host example-data.loomsystems.com
</match>

path_key  - this adds the file name to every sent entry
filter  - this clause is used to add additional attributes to the events
loomApplication, loomService  and loomSourceType  are special entries that serve as hints to Sophie, helping in mapping and processing the data.

Did this answer your question?