Posted at: 2018-02-27 @ 09:11:52
Recently one of our elastic beanstalk deployments suddenly failed out of the blue with the following error in the EB console.
Command failed on instance. Return code: 1 Output: invalid byte sequence in UTF-8. For more detail, check /var/log/eb-activity.log using console or EB CLI.
Looking in said eb-activity.log we have the following line:
in the eb-activity.log
/AppDeployStage0/AppDeployPreHook/50npm.sh] : Activity has unexpected exception, because: invalid byte sequence in UTF-8 (ArgumentError)
This stack is a node application stack and we didn't write this script. So next step was a P1 AWS support ticket, thankfully we have a level of support that results in fast answers.
AWS confirmed it was an issue with ebnode.py file that 50npm.sh calls.
"Elastic Beanstalk passes STDOUT and STDERR of npm install to Python and Ruby based logger scripts. These loggers expect NPM modules to conform to standards and use UTF-8 characters."
The root cause here was that some dependency we had in our application contains an NPM module that is using some non UTF-8 charaters and that causes EB to break on a deployment of the new version.
The AWS provided solution
The AWS provided workaround is to modify the ebnode.py script to redirect NPM install output to /dev/null
Specifically look for a lines like
check_call([npm_path, '--production', 'install'], cwd=app_path, env=npm_env)
and modify to:
with open(os.devnull, 'w') as devnull:
check_call([npm_path, '--production', 'install'], cwd=app_path, env=npm_env, stdout=devnull)
Then the updated ebnode.py would need to be uploaded into S3 (the elasticbeanstalk-$REGION-$ACCOUNTID bucket probably makes the most sense) and using .ebextensions to pull the new ebnode.py file and overwrite the default one.
If your thinking messy and hard, yes and yes.
How i fixed it
I took our whole application and moved it to a docker container. Then I deployed it to AWS Fargate which is the new "run a container without worrying about hosts"
I've been informed that AWS is planning on fixing this, however, sounds like it won't be fixed until they bump the node EB environment again.
So if your running a node application in elastic beanstalk be warned, this could happen to you. In our case, it wasn't even a dependency we updated! It was a dependency of a dependency.