Terminal stops responding when encounter non-utf8 characters

[dong-zeyu]

The terminal will totally stop responding when some specific non-utf8 characters accidently sent from the bash (or other process).

An easy way to trigger this problem: run echo -n -e "\xe4\xba" in jupyter terminal.

Then jupyter server will report an exception, and the terminal will totally stop responding, and the server will report exception on every hit. The only way is to restart the terminal.

The quick fix for this is to replace the default decoder mode from "strict" to "replace" / "ignore" / "backslashreplace":

diff --git a/terminado/management.py b/terminado/management.py
index c62be2c..29190ea 100644
--- a/terminado/management.py
+++ b/terminado/management.py
@@ -18,6 +18,7 @@ import itertools
 import logging
 import os
 import signal
+import codecs

 try:
     from ptyprocess import PtyProcessUnicode
@@ -50,6 +51,7 @@ class PtyWithClients(object):
         if preexec_fn is not None:
             kwargs["preexec_fn"] = preexec_fn
         self.ptyproc = PtyProcessUnicode.spawn(**kwargs)
+        self.ptyproc.decoder = codecs.getincrementaldecoder('utf-8')(errors='backslashreplace')

     def resize_to_smallest(self):
         """Set the terminal size to that of the smallest client dimensions.

This issue will happens frequently when accidently input some non ascii characters, like 了(b'\xe4\xba\x86' in utf-8), but use only one backspace to remove it (it will actually become some invisible chars b'\xe4\xba'), and press enter. Then the terminal will get stuck when scrolling back to bash history (same as echo -n -e "\xe4\xba").

[blink1073]

HI @dizzam, your diff looks good to me, would you like to submit a PR?

[dong-zeyu]

Do you think I should leave "replace" / "ignore" / "backslashreplace"? Typically, I think, we should use "ignore" so that the decoder will replace any unknown char be replaced by "�". However, "backslashreplace" will replace it by string "\x**".

Also, this patch currently works on both win and linux, but it touches the inner data of PtyProcessUnicode and I'm not sure if it is safe in the future.

I've submitted a PR to pexpect/ptyprocess#63 to allow spawn with the argument codec_errors="ignore", but it might be slow to wait this patch goes into a new release. Also, there is something to do with windows, which uses winpty instead.

[blink1073]

Typically, I think, we should use "ignore" so that the decoder will replace any unknown char be replaced by "�".

That sounds good to me