This isn't as common as it used to be, but still today you might see log shippers failing because of encoding issues.
We primarily encounter this today with Microsoft EventLog and SQL Server error logs, which use the UTF-16LE encoding.

This guide will show examples of how to convert encoding when using two popular open-source log-shippers - NxLog and LogStash.

Converting encoding in NxLog

NxLog offers and extension for these cases. Add the following clause to your configuration file:

<Extension charconv>
    Module xm_charconv
</Extension>

Then, in your input , add the following Exec  command:

Exec $raw_event = convert($raw_event,"<source-encoding>","UTF-8");

Converting encoding in LogStash

 LogStash uses codecs to decode and encode your data. The example shows the default codec which is plain:

input {
    codec => plain {
        charset => "<source-encoding>"
    }
    ...
}

Provide feedback!

Suggestions for improvement? we'd love to hear them!

Did this answer your question?