When writing high-volume of logs network might become a bottleneck. Here are some guidelines to reduce or limit the network bandwidth your log-shipper produces.

Option 1 - encode

All modern log shippers (we know of) support efficient encoding (examples are Lumberjack, Protocol-buffers, Message-pack). Encoding the data efficiently, v.s. sending using e.g. JSON, are significant. Loom supports many modern encodings - let us know if you'd like to use one!

Option 2 - compress

Compression can be CPU-costly but by changing the compression-level you can get to your sweet point for this trade-off.
Rsyslog, for example, supports 9 levels of compression.

Option 3 - filter what you send

 Configure your agent to discard trace-level logs or message-patterns that you just don't need. Here's how you do this with Rsyslog, and here's the same for Filebeat.

Option 4 - Place a cap on bandwidth usage

If you're concerned about sudden peaks in network usage by the log shipper that might impact other systems, we recommend using IPTABLES (available in all modern Linux distributions) to cap the bandwidth the process is allowed to use. Here's a good StackExchange article on how to do this. 


That's it! If you have any additional ideas we'd love to hear from you so that we can expand this article.

Did this answer your question?