My pipeline build is failing on jenkins2. Guess why

Pipeline builds have the annoying habit of sandboxing themselves, which means that they watch for "dangerous" commands. If your pipeline groovy script has a dangerous command (which can be a whole hell of a lot of things), it will not let you execute the command and fail the script.

If your sandbox is enabled, you need to approve of all dangerous executions, on the page found under:  Manage Jenkins > In-process Script Approval

Or you could turn off the sandbox. Which is a funky and annoying guardian, that makes testing a script a devil of a thing.

Jenkins and script approval. One word. Nazis.

Maybe I'm the only one who's been having problems with the Jenkins Pipeline Plugin. Maybe I'm the only one who pounded away at my keyboard for three hours trying to find out why I couldn't get the name of the user who initiate the pipeline.

This piece of code. This one terrible piece of code

By default, Groovy sandboxing is turned on when you work on a pipeline job and when that happens, the compiler only lets through code that is either safe (which translates to Does Nothing) or else is approved by an administrator.

That all sounds good. That all sounds like a great way to control these jobs, and ensure good quality in the jobs.

But the approval process is a mess if you don't turn off Groovy sandboxing. Because Jenkins won't allow you to run the script with unapproved code, and Jenkins won't allow you to approve code without running the script first.

It gets better: Jenkins runs a job until it finds an unapproved action, after which it stops, and demands that you approve the action first.

In the example above, accessing the rawBuild variable is considered unsafe, so you have to approve it first. But the method getCauses() is also unsafe, but it won't bother to tell you that it's unsafe. Which means that you spend your morning doing things like this:
  • Write the code
  • Run the jenkins - it fails
  • Go to the Script Approval page and approve the use of rawBuild
  • Run the jenkins - it fails
  • Go to the Script Approval page and approve the use of getCauses()
This is how revolutions begin. Little things like that make us take up arms against our oppressor. Just saying.

Jenkins2 errors: "No valid crumb was included in request "

After installing my Jenkins2 on Ubuntu 14.04, I got everything up and running, only to find jenkins.log filling up with error messages, to the tune of 60 per minute, 70,000 messages per day: 

Sep 21, 2017 9:27:21 AM doFilter
WARNING: Found invalid crumb ad97b73a2a6de7a04b11f45e3fe9bf8f. Will check remaining parameters for a valid one...
Sep 21, 2017 9:27:21 AM doFilter
WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.

After a bit of Googling, the consensus was that this was connected to the Build Monitor Plugin, which I promptly uninstalled, but to no avail.

Digging a bit more, it looks like this is connected to the CSRF protection implemented in Jenkins2, but not found in Jenkins1. Once I disabled this, the logging stopped.

I'm not saying that this is the solution to the problem. I'm just saying that if you have this issue, can't fix it, then disabling CSRF protection will work.

To disable:

  1. Log in to Jenkins as an Administrator
  2. GOTO: Jenkins > Manage Jenkins > Configure Global Security and enable Prevent Cross Site Request Forgery exploits
  3. Uncheck this option

This is NOT a long term solution. This just stops the logging, until you can dig deeper and find the real source of the problem.

Elasticsearch plugin troubleshooting

Problem: Elasticsearch won't restart

After trying to start Elasticsearch, using
sudo service elasticsearch start
, it confirmed that it had started elasticsearch, but when I ran
sudo service elasticsearch status
all I got in response was:

* elasticsearch is not running 

Nothing more than that, which was goddammed annoying because I was busy trying to do real work (Reddit).

I found evidence of the cause in /var/log/elasticsearch/elasticsearch.log:

So it was a plugin. But what plugin?

I ran these commands to try and reinstall the licenses:

cd /usr/share/elasticsearch/bin
sudo ./plugin remove license
sudo ./plugin install license

After running that, I got a better error message:

ERROR: Plugin [watcher] is incompatible with Elasticsearch [2.4.6]. Was designed for version [2.4.4]

So it was the watcher plugin causing problems. Fixing it was easy enough though:

sudo ./plugin install watcher
sudo ./plugin install license

That's all it took. I restarted elasticsearch and I was on my way.

Jenkins2 on AWS - make a redirect from http to https

You think it would be simple to set up a http redirect on an AWS Marketplace setup for Jenkins2, but it isn't. As it turns out, the instructions from AWS itself are wrong, and so are the instructions from Bitnami!

Here is the redirect that will work. This should be valid for most apache setups on AWS, and then I will show you how to apply it to a Jenkins2 server.

Add this to your apache conf file:

RewriteEngine on
RewriteCond %{HTTP:X-Forwarded-Proto} ^http$
RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Then restart apache and you should have no problem after that.

To do this for Bitnami's implementation of Jenkins2 on AWS (this is the AWS Marketplace image), do the following:

  1. Add the above change to the top of: /opt/bitnami/apps/jenkins/conf/httpd-prefix.conf
  2. Restart apache:
    sudo /opt/bitnami/ restart apache