Logstash – “load error: psych”

Just a quick note today on a issue we’ve been diagnosing with a client whilst upgrading their ELK cluster from a v2.x stack to a v6.x stack on Ubuntu 18.04.

Problem

After upgrading Logstash, we noticed that Logstash was failing to start properly and we were getting the following errors in the logs:

#journalctl -e -u logstash.service

Mar 13 09:41:10 logstash[20574]: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59:in `require':
Mar 13 09:41:10 logstash[20574]: It seems your ruby installation is missing psych (for YAML output).
Mar 13 09:41:10 logstash[20574]: To eliminate this warning, please install libyaml and reinstall your ruby.
Mar 13 09:41:11 logstash[20574]: [ERROR] 2019-03-13 09:41:11.007 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemspecError) There was a LoadError while loading logstash-core.gemspec:
Mar 13 09:41:11 logstash[20574]: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
Mar 13 09:41:11 logstash[20574]: Does it try to require a relative path? That's been removed in Ruby 1.9.
Mar 13 09:41:11 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 13 09:41:11 systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 13 09:41:11 systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Mar 13 09:41:11 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 95.
Mar 13 09:41:11 systemd[1]: Stopped logstash.

The key lines here are:

Mar 13 09:41:11 logstash[20574]: [ERROR] 2019-03-13 09:41:11.007 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemspecError) There was a LoadError while loading logstash-core.gemspec:

Mar 13 09:41:11 logstash[20574]: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory

Following some investigation, we found that the following setting had been changed in the /etc/defaults/logstash file:

LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"

So what’s going on here and why is this a problem?

Well, according to this post on GitHub…

“certain versions of the JRuby runtime and libraries in certain plugins (e.g., the Netty network library in the TCP input) copy executable files to the temp directory which causes subsequent failures when /tmp is mounted noexec”

@danhermann

The TCP plugin was being used in this configuration, which explains why we were seeing this problem.

Fix

The fix that worked for us was to remove the Java option completely from our configuration. This will default to using the /tmp directory, which doesn’t have ‘noexec’ set in our environment. This allows the libraries to be copied to /tmp and executed as required.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.