I recently had an issue with the
LC_CTYPE locale variable being set to an invalid value.
I searched the complete machine for where it could be set, but was unable to find the configuration location.
Further investigation led to the revelation that my local OSX machine config was causing the error on the remote host.
The initial error was from a Python3 program that I was trying to get to run on a remote linux host. The helpful but unexpected error was as follows:
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps. This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables: export LC_ALL=C.UTF-8 export LANG=C.UTF-8 Click discovered that you exported a UTF-8 locale but the locale system could not pick up from it because it does not exist. The exported locale is 'en_US.UTF-8' but it is not supported
The most obvious workaround was to simply follow what the error recommended and perform:
export LC_ALL=C.UTF-8 export LANG=C.UTF-8
This causes the locale to be set to predictable values, and allows you to get on. However, I still wanted to understand what was causing the incorrect locale to be configured in the first place.
There were a few odd things I noticed in this case.
Once I logged into the box, becoming
root or any other user fixed this problem.
This was useful information for debugging, later on.
locale printed along with the output, the following errors:
locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE=UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=
apt-get command, also printed this error from
perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = "UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
I also noticed in the output of
LC_ALL was not set.
This meant that the issue had to lie with the remaining
As mentioned before,
root was correctly configured and working fine.
So I decided to compare the environment values for the two users:
$ echo $LC_CTYPE UTF-8 $ locale | grep CTYPE LC_CTYPE=UTF-8 # echo $LC_CTYPE` # locale | grep CTYPE LC_CTYPE="en_US.UTF-8"
In the above output, I saw that
LC_CTYPE was unset for
was being defaulted to a valid value of
However, for the
ssh user, it was set to the invalid value
locale to complain when it was run.
- This issue was a combination of using OSX, a terminal emulator and SSHing into a remote Linux host.
- Unlike Linux, OSX allows non-locale strings as a valid value for the
LC_CTYPEand other locale environment variables.
- By default, most terminal emulators including iTerm2 and Terminal.app will set this value to
- These variables are then forwarded to the remote host by
sshwhenever a new connection is established.
- You can verify this by running
localeon your local and remote machines. In my specific case,
LC_CTYPE=UTF-8was the invalid value and causing the above errors.
- This also explained why changing the user to become
rootor some other user worked, as that did not carry-forward the environment and its associated variables.
The best solution is to disable forwarding of these variables on the server itself. To do so, you must have access to change the
sshd config file at
Change or remove the
AcceptEnv directive which is set to the below value, by default:
# remove/change/comment this line: AcceptEnv LANG LC_*
You will of course, need to reload the
sshd daemon for the changes to take effect.
This is the ideal solution, as it allows you complete control over what default locales are used by your users and is also isolated from the the behaviour of individual ssh clients.
If for some reason, you’re unable to change the setting on the server, you can instead fix this for yourself, on the client side.
The first approach in this case is to configure your terminal emulator to not set the locale values on startup. This should make all these values default to
C which stands for “computer” and is valid across both OSX and Linux.
For iTerm2, you can uncheck “Set locale variables automatically” by going to “Preferences > Profiles > Terminal > Environment”.
For Terminal.app you need to uncheck “Set locale environment variable on startup”, by going to “Preferences > Profiles > Advanced > International”:
If you want do want your terminal to set locale variables, you can instead disable forwarding these in your ssh client configuration.
~/.ssh/ssh_config file to comment out the
SendEnv option as follows:
Host * ... # Comment/Remove this line. SendEnv LANG LC_* ...
Ideally, you should never ignore warnings/messages related to the locale, unless you know what you’re doing. Often these are indicative of other issues in the system that may not be immediately apparent.
Like most interesting bugs, the cause and effect in this case were so far apart that it took more than a few hops before we got to the root cause.
Usually, when facing an error in a remote machine, one would never suspect one’s local machine to be the problem.
I still look at the
LC_CTYPE=UTF-8 environment variable on my machine and wonder at how harmless it appears.