- Issue created by @murz
Some log consumers and scrappers have a limit for the max length of a single log line.
And the long lines exceeding the limit got truncated or split into several lines, producing the invalid JSON strings (with no closing curly bracket).
For example, for stdout and Docker the limit is 16kb by default, see the discussion here https://github.com/moby/moby/issues/33788
Some other log scrappers have 4kb or 64kb limits.
These limits look pretty enough for regular log messages, but when we have an exception, the log length can easily exceed this limit, producing problems with parsing logs.
But we can't simply just cut the end of the log, because it is an object, not a string. And can't just remove all keys after the long line.
Seems the best way could be like this:
1. Introduce a new setting with the maximum length of the log line.
2. Generate the JSON line, and check if it fits the limits.
3. If doesn't fit, go through all the keys and search for the longest value in them.
4. Cut the longest value to fit the limits, and recreate the JSON.
5. If not enough, get the next longest value and cut it too.
This will work for most cases, but if we have several long lines, it will cut the most longest line to zero, instead of proportionally cutting each line.
So, would be good to invent something to make this cut smarter and keep the start of each line, at least.
Active
1.0
Code